Technical Interviews: A Rehearsal, Not a Quiz

From the field: rejected by the wrong question

I felt good leaving that interview: we had asked a hard question, the candidate could not answer it, and we thought we had protected the bar. Six months later, while integrating with the service they wrote, I understood that we had not protected the bar. We had just held it in the wrong place.

The question I asked was not a bad question. The problem was that it had nothing to do with the work we did. We were writing payment flows; nobody had balanced a tree in the previous year. What I measured was not the candidate’s engineering, it was how much interview practice they had done that week.

Worse: that question filtered some people out systematically. A senior who had spent two years shipping real product, with no time to practise interviews, would get stuck; a recent graduate with it fresh in mind would sail through. We were listening loudly to the wrong signal.

One question and its three parts

The question the interview has to answer: what would it be like to work with this person? Underneath it are three sub-questions, and all three are observable:

1. Can they make progress in a real codebase? Can they find their way inside unfamiliar code, or do they only work on a blank page?
2. What do they do when they meet something they do not know? This is the most valuable signal. Can they say “I do not know this, let me look it up”, or do they try to hide it?
3. How do they take and give feedback? Offer a suggestion during the task. Do they get defensive, do they think it over, and if they disagree can they say why?

I watch the second one especially, because it is the most common situation on the job. Nobody spends their day doing what they already know; everyone spends half of it working out something they do not.

The task: your own code, 90 minutes

We dropped the whiteboard and moved to this: a small, standalone repository cut and simplified from our own code. It contains a shrunken version of real work.

READY UPFRONT : repo cloned, dependencies installed, tests running
                (an interview that burns 40 minutes of the candidate's
                 time on setup measures nothing)

TASK          : "In this service the same notification sometimes goes
                 out twice. Find the cause, fix it, then write a test
                 that shows it will not happen again."
                 (cut from an incident we actually had)

TIME          : 90 minutes, together, shared screen

ALLOWED       : the internet, documentation, libraries, AI tools,
                asking us questions (that is how we work too)

WHAT WE DO    : watch, give a hint when they are stuck,
                and at some point deliberately make a suggestion
                -> to see how they take feedback

FINISHING IS NOT REQUIRED : how they move matters more than whether
                            they finish

The “finishing is not required” rule changes the assessment completely. Some candidates find the cause in 40 minutes and leave the fix half done; others move carefully down a path that would take two hours. Both are acceptable, because that is what the job looks like.

A fixed question set, so you can compare

The same task and the same four closing questions for every candidate. It feels dull, but it is the only way to get comparability — the thing I learned during 52 interviews while diagnosing an organisation applies here too: different questions give you different answers and none of them line up.

Assessment: write first, talk second

For a long time we did it this way: after the interview, three people went into a room, talked, and decided. Then I noticed that the decision was almost always set by whoever spoke first. The second person softened it and the third agreed.

We changed the rule: everyone writes their own note, alone, within 30 minutes. It can be one line, but it must contain two things: yes or no, and a one-sentence reason. The meeting only starts once the notes are written.

The difference: now there are different opinions on the table at the start of the meeting, and the discussion happens on those differences. This is the hiring version of the “who speaks first” problem from the psychological safety post.

After AI: the question changed

Banning tools during an interview no longer makes sense, because the person will use them from day one. The real question has changed: not “can they write code” but “can they review generated code”.

So we added a step to the task: we hand over a ready-made solution and ask “would you ship this?” The solution has two deliberate problems in it — one visible, one not. What we are looking for is not whether they find both; it is how they look. That is exactly the behaviour we want in code review, and it is now the most valuable part of the job.

The rejection email

You reject most candidates, which means that for most people who touch your hiring process, the experience is a rejection experience. We do two things: we reply within 48 hours, and we write one sentence of real reasoning.

We do this because it comes back, not because it is polite. One candidate we rejected applied again a year later and said in the interview: “I worked on the thing you wrote to me last time.” We hired them. A silent rejection never brings that person back.

What to track

What did not work for me

• Take-home assignments. We said “do this over the weekend”. Two problems appeared: candidates with jobs and families dropped out, and we could not tell who had written the solution. A 90-minute session together gives the same information more fairly.
• A five-round process. It looked rigorous. In practice good candidates got another offer by round three and withdrew. We cut it to three: an intro conversation, the technical session, and a chat with the team.
• A “culture fit” round. Nobody could define what it measured, and in practice it became “are they like us”. We replaced it with something concrete: we describe a real disagreement the team had and ask what they would do.

Checklist

Conclusion

When we rejected that candidate, the process worked perfectly: a prepared question, two interviewers, a joint decision. The only flawed thing was the question itself, and we did not notice for six months — because a hiring process has no feedback loop. You never find out what happened to the people you turned down.

That is why an interview should not be built as a tool for picking the right person, but as a rehearsal of working together. The closer the rehearsal is to the real thing, the more accurate your decision.

The test is simple: will the person you hire do what you asked them in the interview, in their first week? If not, what you measured was not the job.