Research Nov 20, 2026 · Originally published Nov 19

AWS details how Amazon Connect Talent's structured AI interviews were evaluated: scores matched independent experts 89.3% of the time, and first-answer-only scores ran up to 1.14 points lower

The question every AI interview product faces is why anyone should trust the score. AWS's design principles and independent evaluation data, for teams building interview or assessment products.

Image: AWS Machine Learning Blog
Primary source AWS Machine Learning Blog · aws.amazon.com Read the original ↗

// Key points

  • The AI interviewer gathers evidence but does not score; a separate evaluation step scores with a fixed formula afterward, flags a competency instead of scoring it when evidence is thin, and does not count missing evidence against the candidate.
  • Scoring uses only transcribed responses, not video, voice tone or appearance, excludes candidate identity information, and evaluates each competency separately.
  • An independent I-O psychology firm found 91% of generated competencies traceable to the job description and 98% of questions targeting the intended competencies; scoring matched expert decisions 89.3% of the time, and 97.6% when candidates gave clear evidence.
  • In a preliminary evaluation, scores based on the first response alone were up to 1.14 points lower on a five-point scale than scores based on the full conversation with follow-ups.

Builder's takeThis one bears directly on my AI Interview product. What I agree with most is splitting the roles, an interviewer that doesn't score and a scorer that doesn't talk, and flagging rather than scoring when evidence is thin; both go straight at the problem of a candidate's score varying run to run, which I also wrote about in my post on why interview scores vary run to run. My advice for interview products: add a first-answer versus full-conversation comparison to your evals, and let your own data say how much follow-ups are worth.

Every Monday, the week in AI in one email

Every Monday morning: the 6 AI stories that mattered most in the past seven days, with my take, in your inbox.

Your email is only used to send the weekly and is never shared. Every email has a one-click unsubscribe link. See the latest issue →