- Write the scorecard before scheduling, because a coding exercise alone cannot tell you whether someone can own a deployment.
- Score four or five observable areas: technical delivery, problem framing, working with users, ownership and judgement.
- Ask every candidate the same core questions, then follow up on their own contribution and what happened after launch.
- Build the work sample from the real job, and keep it bounded — unpaid work resembling a customer deliverable is a poor substitute for an interview.
- Separate "we did not test this" from "the candidate did not meet the bar" in every written note.
Turn the first year's outcomes into observable areas
Describe the outcomes for the first six to twelve months: an AI workflow in production, a customer's systems integrated, a deployment process improved. Avoid outcomes that depend on a future product decision the candidate cannot control. Turn those into observable areas:
- Technical delivery: APIs, data, failure modes, security, testing and operations at the level the role requires.
- Problem framing: separating the stated request from the operating problem and defining a useful first version.
- Working with users and partners: precise questions, honest trade-offs, and changing the plan when evidence changes.
- Ownership: carrying work from discovery through rollout, monitoring, documentation and handoff.
- Judgement: what must be solved now, what can wait, and when to involve product, sales, security or another engineer.
"Insufficient evidence", "meets the role" and "strong evidence for this role" are easier to apply than a number whose meaning changes between interviewers. A candidate can be technically capable and still lack evidence for the customer-facing or production responsibility this role carries.
Ask the same five questions for evidence
- Tell me about a project where the request was incomplete or changed during delivery. What did you learn first, what did you build, and what changed after users tried it?
- Tell me about a deployment that failed or needed a rollback. How did you find the cause, communicate it, and prevent a repeat?
- Describe explaining a technical limitation to a customer. What decision did the explanation support?
- Which part of the system did you own after it went live, and how did you decide whether it needed more work?
- Tell me about a decision you escalated. What did you bring, and what would have happened if you had acted alone?
Ask for artifacts the candidate can share safely: an architecture sketch, a runbook, an incident summary, an anonymised implementation plan. Do not ask for a former employer's confidential information. Familiarity with interview language is not a proxy for delivery skill.
Use a work sample that matches the job
For a production AI role, give an unreliable model output and a request to ship quickly. For a data and integration role, give conflicting schemas, an unavailable API and a customer deadline. State what is available, what is missing, how long it should take, and whether they can ask questions.
Score the reasoning, not a hidden correct architecture. Look for whether the candidate asks about users, constraints, data quality, access and failure impact, proposes a small first release that produces evidence, names privacy, security and rollback where they apply, explains how they would measure whether it works, and makes ownership and handoff clear.
Make the decision reviewable
Have interviewers submit notes independently before the debrief, each linking a conclusion to an observed answer or exercise decision. If the role requires travel, customer contact, on-call work, a clearance or a particular location, assess that requirement directly.
At the debrief, compare evidence against the scorecard. Do not average unrelated opinions into a number. If the panel disagrees, name the missing evidence and decide whether one more focused interview resolves it. The decision answers one question: can this person do the defined work, under the stated conditions, with a reasonable plan for the gaps?