Matching engineers to projects with Jev vs. LLMs and embeddings
【Opus 級？】Jev でエンジニアとプロジェクトをマッチング ― LLM・埋め込みモデルと比較
B · Useful, limited Sapeet algorithm engineer runs a carefully reported benchmark with confidence intervals and stated limits, but on 40 synthetic pairs with a model-generated answer key and no fairness test.
Summary based on the full text·Original: about 10 minutes to read
In short
A Qiita post (22 October 2026) by Akimasa Nishida, algorithm engineer at Sapeet. He tests TypeSafe AI's Jev, a model that returns probabilities for typed questions instead of generating text, on scoring how well an engineer's skill sheet fits a project brief. The answer key is Claude Opus 5.5's averaged judgement. Jev ranked candidates at least as well as Claude Haiku, much faster and cheaper, and its confidence score flagged the pairs most worth a human check. The differences in ranking accuracy are within the error margin.
Key numbers
- 0.87 / 0.78 / 0.47Kendall tau-b against Opus 5.5 labels for Jev, Haiku 4.5 and the ruri embedding model
- $0.093 vs $4.64cost per 1,000 pairs, Jev versus Haiku 4.5 (temperature 0)
- 182 msJev median latency per pair, against 1,472 ms for Haiku 5.5
- 19 of 22clearly mismatched pairs Jev excluded without dropping a valid pair (Haiku 5.5: 17)
- ~3xlarger error versus the answer key for Jev's lowest-confidence 20% of pairs
As reported by the source.
Japanese·Web page·opens qiita.com
What it reports
- Data: five fictional projects and eight fictional engineers (40 pairs), drafted by Claude and reviewed by a person, including hard cases such as an overqualified tech lead and a vague self-assessor.
- Four criteria (stack match, domain experience, level for difficulty, recent hands-on use) weighted by project type into a 0-1 match score.
- Ranking accuracy (Kendall tau-b, mean of five projects): Jev 0.87, Haiku 4.5 at temperature 0 0.78, Haiku 5.5 0.61, ruri embeddings 0.47. Jev versus Haiku 4.5 is not significant (p = 0.46). Opus 5.5 agreed with itself at 0.95.
- Jev put the same person first as the key in all five projects. Haiku rated the 18-year tech lead as more than experienced enough for a data platform role despite no dbt or BigQuery experience; Jev and the key did not.
- Speed and cost: Jev 182 ms and $0.093 per 1,000 pairs; Haiku 5.5 1,472 ms and $0.50; Haiku 4.5 1,740 ms and $4.64. Run-to-run standard deviation: Jev 0.003, Haiku 5.5 0.031.
- Screening: Jev excluded 19 of 22 clearly mismatched pairs without error, Haiku 5.5 17, ruri 3.
- Jev's lowest-confidence 8 pairs had mean error 0.082 against 0.026 for the rest; they contained 5 of the 8 largest errors (Haiku's self-reported confidence caught 4).
- Re-scoring low-confidence pairs with Haiku 5.5 did not improve accuracy and doubled cost.
- Scaled to 10,000 pairs: Jev about $0.9, Haiku 5.5 about $5, Opus 5.5 about $210.
How they got it
Each method scored all 40 pairs five times on 15 October 2026; Claude models ran via Amazon Bedrock from Tokyo, ruri locally. Accuracy is rank correlation with the Opus 5.5 key, with 95% confidence intervals. Cost is token count times list price.
Limits
- The answer key is a model, not people, and Claude drafted the data, which may favour Haiku.
- 40 pairs; accuracy differences fall within the error range.
- Latency comparisons cross different network paths.
- Score confidence can be overconfident on out-of-distribution tasks, per a cited report.
- No fairness testing; the author says AI use in assignment or hiring needs fairness checks and a human final decision.
Why it matters for companies
For HR and staffing teams the pattern is first-pass screening: score every pairing cheaply, deprioritise clear mismatches, and route low-confidence cases to a person. At about 1/220 of the cost of scoring with the reference model, full-matrix screening becomes affordable. The sample is far too small, and too synthetic, to support use in real assignment or hiring decisions without bias testing on real data.
Key passages in translation
Japanese original
Jev が最も高い値でしたが，Haiku 4.5 との差は統計的に有意ではありませんでした（p = 0.46）．
Jev had the highest value, but the difference from Haiku 4.5 was not statistically significant (p = 0.46).
Japanese original
Jev は候補が多いときに人が確認する順番を決める一次スクリーニングに向いている
Jev suits first-pass screening that sets the order in which people review candidates when there are many.
Japanese original
アサインや採用への AI の利用には，公平性の検証と人による最終判断が欠かせない．本実験は公平性を検証したものではない
Using AI for assignment or hiring requires fairness testing and a final human decision. This experiment did not test fairness.
Summary based on the full text. Read the original for the full argument and data.Read the original on qiita.com
- Tags
- Model cost