How to assess DevOps engineers without a question bank

Search "DevOps interview questions" and you will find lists of fifty. What is Kubernetes. Explain CI/CD. Describe blue-green deployment. The lists are long because they are easy to write, and they persist because they feel rigorous: fifty questions must surely be more thorough than five.

Here is the problem, and every experienced infrastructure hiring manager already knows it: the candidates who answer definition questions best are the ones who studied the list. The correlation between reciting what a liveness probe is and knowing what to do at 2am when the probe is killing healthy pods is weak. Sometimes it is negative, because the person who memorised the glossary never had to live with the consequences.

This post is the framework we would use instead: what each assessment stage can honestly tell you, how to build the one stage that predicts performance, and where the traps are.

Start from what the job actually is

DevOps and SRE work, reduced to its operating core, is three loops:

1. Diagnose: something is wrong; find out what is true before deciding what is broken.
2. Change safely: make a modification to a live system in a way you can watch and undo.
3. Automate: notice a repeated action and make it a machine's job.

A good assessment measures how a candidate runs the loops that matter in the role. Start with two or three situations the person will genuinely own, such as diagnosing a failed deployment, containing a production incident, or removing a repeated manual step. Build the scenario and rubric from those situations. The environment need not reproduce your exact stack, but the decisions, risks, and level of ownership should match the job. Everything else - tool trivia, YAML syntax, which flags kubectl takes - is lookupable. Nobody who is any good works with the documentation closed, so testing whether they can is testing the wrong thing.

What each stage can honestly tell you

The CV tells you what a candidate claims. Nothing more. Titles are inflated or deflated by company norms; "managed Kubernetes infrastructure" spans everything from ran-a-tutorial to owned-a-40-cluster-platform. Use it to decide who to talk to, never to decide who can do the work.

A screening call tells you whether they have lived near production. Twenty minutes of "tell me about an incident you were part of" separates people who have been paged from people who have read about being paged. Listen for specifics: real error messages, real dead ends, what they checked first. Fabricated incident stories are conspicuously tidy.

A quiz or coding-puzzle screen tells you who prepared for quizzes. It has a legitimate use - high-volume entry-level funnels where you need a cheap coarse filter - and a real cost: experienced candidates decline them, and the ones who remain are selected for test-taking. We wrote about that trade-off separately in the hidden cost of HackerRank-style assessments.

A scenario in a real environment tells you how they operate. This is the stage that predicts performance, and it is the one most teams skip because it is the most work to build. The rest of this post is about building it.

Building a scenario that predicts

The shape: give the candidate access to a real, broken environment - a cluster with a failing deployment, a pipeline that stopped shipping, a service that degrades under load - and watch them work for thirty to forty-five minutes. Not a whiteboard description of the problem. The actual problem, with a shell.

What makes a scenario good:

• The failure is real, not staged trivia. Break it the way production breaks: a resource limit somebody lowered, a selector typo, a certificate that expired. If your own engineers would find the scenario plausible, it is a good scenario.
• There is more than one road in. Rigid single-path puzzles measure whether the candidate guesses your intended path. Real diagnosis has several valid entry points; score the reasoning, not the route.
• The fix is not the point. A candidate who fixes it in eight minutes by lucky pattern-match has shown you less than one who takes thirty, narrows methodically, states hypotheses, and recovers when the first one dies. Time-to-green is the weakest signal in the room.
• Let them use the docs. You are not hiring someone to work without the internet. Watching how someone reads documentation under pressure is itself signal.

What to listen for while they work:

• Do they look before they touch? The candidates who start with reading state (get, describe, logs, events) rather than restarting things are telling you how they will treat your production.
• Do they narrate hypotheses? "The endpoints are empty, so either the selector is wrong or the pods are not ready; checking readiness first" is the sound of a diagnostic loop running. Allow candidates who find continuous narration difficult to explain their reasoning at natural checkpoints instead.
• What do they do when they are wrong? The first theory failing is the most informative moment of the whole session. Look for the update, not the ego.
• Do they know what they would do next? Ending with "it works now" is junior. Ending with "it works, and I would add an alert on X so this pages before users notice" is the job.

Score each candidate against the same four dimensions: evidence gathering, hypothesis quality, change safety, and recovery after a wrong turn. Define observable anchors before interviewing. For example, a weak change-safety score means making an unexplained change without checking impact or rollback; a strong score means stating the expected effect, checking the blast radius, and identifying how to reverse it. Record evidence from the session before discussing an overall hire decision.

Nobody who is any good works with the documentation closed, so testing whether they can is testing the wrong thing.

The follow-up conversation

After the scenario, spend fifteen minutes on it. Ask why they checked what they checked. Ask what they would monitor to catch this earlier. Ask what they did not investigate and why. This conversation is fraud-resistant in a way nothing else is: someone who watched a walkthrough video can reproduce commands, but they cannot defend decisions they did not make. If the answers and the session do not match, believe the mismatch.

The traps

• Over-indexing on your own stack. Someone deep in ECS who has not run Kubernetes can be three weeks from productive; someone who memorised your stack's vocabulary can be permanently mediocre. Assess the loops, not the logos.
• The senior-pedigree shortcut. "Ex-FAANG" tells you they passed an interview at a company with different problems from yours. It is the same claims-versus-evidence gap wearing a nicer jacket.
• Unstructured "culture" conversations doing silent screening. If a stage has no rubric, it is measuring confidence and similarity, and it is where your pipeline quietly filters out exactly the self-taught candidates who would have aced the scenario.
• Building it once and never calibrating. Run your own engineers through the scenario first. If your strongest people struggle, the scenario is miscalibrated, and every candidate score it produces is noise.

The cost, honestly

A real scenario assessment costs real effort: environment setup, isolation so candidates cannot hurt anything, calibration, and an engineer's time observing each session. That cost is why most teams fall back to question lists, and it is a rational objection. Two honest answers to it: first, the cost of a mis-hire in an infrastructure role - measured in incidents, rebuilds, and the team's time - dwarfs the cost of assessing properly. Second, this is becoming infrastructure you can use rather than build.

Where we fit in

SkillBricks is that infrastructure. Every candidate on the platform has already worked scenarios exactly like the ones described here: real broken environments, real shells, an observer recording the full diagnosis path, with the said-versus-did follow-up built in. Search is by verified skill and tier, candidates are anonymous until you commit to contact, and pricing only applies at that point - browsing the evidence is free.

If you would rather build the capability in-house, everything above is the blueprint, and it works. If you would rather start from candidates whose diagnostic loop is already on record, that is what the platform is for.