You wouldn't let a new hiring manager loose on candidates without watching a few of their interviews first. An AI interviewer deserves the same scrutiny, and probably more. An AI hiring bias audit, done before go-live, is how you find out whether the tool scores like your best interviewers, treats equivalent answers equally, and doesn't quietly disadvantage any group.
This is a practical method you can run with a spreadsheet, a few hiring managers and an afternoon or two. It isn't a legal audit, and it doesn't replace one where the law requires it. It's the homework that makes a legal audit go smoothly.
#Why audit an AI interviewer before it meets a candidate?
Three reasons.
Fairness. Canadian human rights laws prohibit discrimination in employment on protected grounds, whether a person or a piece of software does the screening. "The vendor said it was fair" isn't a defence.
Accuracy. A tool that's perfectly unbiased but scores randomly is still useless. You need to know it agrees with the people whose judgement you trust.
Regulation is arriving. New York City has required bias audits for automated employment decision tools since July 2023. Colorado's AI Act has been delayed to 1 January 2027. If you hire across borders, you'll meet these rules sooner or later, and having your own evidence ready is much easier than scrambling.
#Step 1: Build a golden set of scored transcripts
A golden set is a collection of real or realistic interview transcripts, scored by trained humans, that becomes your answer key.
- Gather 40–60 transcripts per role family. Use past interviews (with appropriate consent and privacy review) or have staff role-play answers across the full range, from weak to excellent.
- Strip identifying details. Names, schools, locations, anything that could nudge a scorer.
- Have at least two trained humans score each one independently against your rubric (our guide to writing a structured interview scoring rubric covers anchors). No conferring.
- Resolve disagreements by discussion to produce a consensus score per dimension. Record the original independent scores too: you'll need them.
#Step 2: Measure AI-versus-human agreement
Now let the AI score the same transcripts and compare. Three numbers tell most of the story:
| Measure | What it tells you |
|---|---|
| Exact agreement | % of dimension scores where AI and consensus match exactly |
| Within-one agreement | % where they're no more than one point apart on a 1–5 scale |
| Correlation | Whether the AI ranks candidates in the same order as humans |
The crucial comparison isn't AI versus perfection. It's AI-versus-human agreement compared with human-versus-human agreement. If your two trained managers agree with each other within one point 85% of the time, and the AI agrees with the consensus within one point 83% of the time, the AI is performing like another trained interviewer. If the AI is at 60%, something's wrong.
Illustrative example (hypothetical numbers): a Calgary retailer scores 50 golden transcripts. Human-to-human within-one agreement is 88%. The AI's within-one agreement with consensus is 86% on Communication but 64% on Problem solving. That gap doesn't mean "reject the tool". It means read the Problem solving disagreements. Often the anchor is vague, and both humans and the AI are guessing.
Also look at direction. Is the AI systematically harsher or more lenient than humans on one dimension? A consistent half-point lean is easy to spot in a simple average and easy to miss in a correlation.
#Step 3: Run paired-answer tests
This is the test that catches the bias you'd never see in aggregate numbers. Take one answer and present it several ways that shouldn't change the score:
- Accent and dialect. Have different speakers record the same answer script: for example, speakers with Caribbean, South Asian, Québécois, Newfoundland and Nigerian English accents. In voice mode, transcription quality can differ by accent, and that can leak into scores.
- Voice versus typed. Submit the identical answer once spoken and once typed, as a candidate with a typed-answer accommodation would.
- Grammar and fluency. Rewrite an answer in plainer English, or with common second-language grammar patterns, keeping the substance identical.
- Names and pronouns. Swap names, pronouns and cultural references in the content.
Score every version. On a well-behaved system, the scores should be the same or within normal noise. Any consistent gap is a finding worth investigating before you go live.
#Step 4: Check adverse impact (and respect its limits)
Once you have real or pilot data, compare outcomes across groups. The classic tool is the four-fifths rule from the US Uniform Guidelines on Employee Selection Procedures:
- For each group, calculate the selection rate: the share who pass the stage (for example, scored Hire or better).
- Divide each group's rate by the rate of the highest-scoring group. That's the impact ratio.
- A ratio below 0.80 is generally treated as evidence of adverse impact.
Illustrative example (hypothetical numbers):
| Group | Interviewed | Scored Hire or better | Selection rate | Impact ratio |
|---|---|---|---|---|
| Group A | 200 | 90 | 45% | 1.00 |
| Group B | 150 | 63 | 42% | 0.93 |
| Group C | 120 | 42 | 35% | 0.78 |
Group C is below 0.80, so you'd dig in: which questions drive the gap, and do the transcripts show a real difference in answer quality or a scoring problem?
Now the important caveats. The Guidelines themselves say smaller differences can still be adverse impact when they're significant in statistical and practical terms, and that larger differences based on small numbers that aren't statistically significant may not be. In practice:
- Groups under about 30 people produce unstable ratios. One extra hire can move a ratio across the line in either direction. Report them with a warning, suppress them, or pool over a longer period.
- 0.80 isn't a Canadian legal standard. Canadian human rights law doesn't set a numeric threshold. Use the ratio as a smoke alarm, not a pass mark.
- Demographic data must be voluntary. Collect it separately from evaluations, with clear consent, and never let it reach scorers.
MSZ Hire's Analytics & fairness page follows these rules: it shows impact ratios by group against a dashed 0.80 line, suppresses groups under 30 people, keeps self-reported demographic data separate from evaluations, and charts score parity by interview mode so you can see whether candidates who type their answers as an approved accommodation are treated the same as those who speak. The Export audit pack button produces a PDF and CSV for your independent auditor.
#What does NYC Local Law 144 require?
If you use an automated employment decision tool for candidates or employees in New York City, Local Law 144 (enforced since 5 July 2023) requires:
- A bias audit by an independent auditor within one year before you use the tool.
- A published summary of the audit results.
- Notice to candidates at least 10 business days before use.
Even if you never hire in New York, LL144 is a useful template: independent, recent, public and with advance notice. Those four words make a good internal standard.
#Your vendor-evidence checklist
Ask every AI interview vendor, including us, for these. "We take fairness seriously" isn't an answer.
| Ask for | What good looks like |
|---|---|
| AI-vs-human agreement data | Method, sample size, and comparison with human-human agreement |
| Paired-answer or counterfactual testing | Which variations were tested (accent, mode, names) and the results |
| Adverse-impact results | Ratios by group with sample sizes; small groups flagged or suppressed |
| What the model looks at | A clear list of inputs. Ideally no facial, emotion or voice-tone analysis |
| Evidence for each score | Quotes or timestamps showing why each rating was given |
| Insufficient-evidence handling | Missing answers withheld, not scored as zero |
| Version control | Which model and rubric version scored each candidate |
| Regression testing | Golden-set re-runs after model or rubric changes |
| Human override and audit trail | Recorded human decisions, with reasons, kept alongside the AI output |
| Independent audit | Who did it, when, and where the summary is published |
| Security and compliance status | Certifications achieved versus in progress, stated plainly |
On that last line, in the spirit of practising what we preach: MSZ Hire's SOC 2 programme is in progress, not complete. Each scorecard shows the model and rubric version used, every rating links to quoted evidence, and recorded human decisions sit alongside the unchanged AI recommendation in the audit log.
#How do you plan an AI hiring bias audit before launch?
A realistic pre-launch plan for one role family:
- Week 1: build and score the golden set with two trained humans.
- Week 2: run the AI on the golden set and the paired-answer variants; review disagreements and fix anchors.
- Weeks 3–6: pilot with real candidates, humans reviewing every result.
- Week 7: review adverse-impact ratios (with small-sample caution) and mode parity. Decide whether to scale.
Want to see what candidates experience during the pilot? Try the AI interview as a candidate. And if your pilot is in Ontario, read up on writing your AI disclosure for job postings before the first posting goes live.