If you work in a technical or specialist field you've probably experienced one of the two scenarios:

You sit down for a technical interview in which you are asked a bunch of questions. There's not much room for explanation either you know what they are asking for or you don't. You aren't able to find an opportunity to showcase the skills and breadth of experience you have. A hour after the interview you receive an email telling you that you will not be proceeding to the next round.
After many months of begging, management has finally provided you with a new senior hire. They inform you of how great this person is and how well they'll fit into the company. Within three days it becomes clear that this new "senior hire" does not possess the skills to complete the most basic of tasks that they were hired for.

As you may have guessed from the title of this post, both have to do with poor technical interview practices. In many cases it's not entirely anyone's fault that the process is flawed. Technical people are asked to create the interviews without any training on how to create them, or similarly they have been promoted to a management position and again never given proper management training.

I'm not the first and I'm sure I won't be the last to write about technical interviews, but having found success based on how I design my technical interviews I'm sharing my approach.

I'll note that when I say technical interviews I mean those that are focussed on assessing the candidates ability to do the job, opposed to those that assess other things such as "cultural fit".

Recruitment and Selection

In management terms the hiring process is made up of two parts:
Recruitment and Selection.

Recruitment is the activities you conduct to influence potential candidates to apply to roles at your organisation. This influence may be positive, such as attracting graduates from a nearby university through sponsorship and events, or negative such as having difficult to obtain certifications on the job description to cause candidate to not apply.

With recruitment the goal is try ensure that the people applying to your company are highly likely to make it through the selection process. Poor recruitment activities cause you to not have access to pools of good candidates, or to be overwhelmed by applications by candidates that will never make it past your selection process.

Selection is process by which you decide which candidates are offered a role at your company. A good selection process is one that is both efficient in determining which candidates should be offered a role; and accurate in ensuring that the candidates hired are actually correct for the role. Broadly speaking organisations use activities such as the following to make their decisions:

screening applications
technical interviews
cultural fit interviews
aptitude tests
personality tests
background checks
medicals

Designing the Interview

Generally speaking we want our interview process to be:

Efficient: time is usually the most valuable resource for both interviewees and interviewers, and in many cases is literally costing money. Apart from the lost time and money, inefficient interview processes can lead to losing candidates because they received an offer before the company made up it's mind. In extreme cases candidates may simply give up on the process.
Accurate: passing the interview should mean that the candidate is able to do the role they are hired for, hopefully excelling in it. An inaccurate interviewing process may lead to the company terminating the successful candidate and searching for a new one, or the successful candidate sticking around draining resources and causing problems.
Repeatable: There are two major reasons you'd want a repeatable interview process: the first is a repeatable process is Scalable, allowing you to have many interviewers conduct the process, including those that do not have the ability to design the actual process. The second is that it Reduces Bias leading to better outcomes for candidates and the company. Note that in this context the bias we are accounting for is unconscious cognitive biases opposed to conscious discriminatory biases. A good candidate should be marked as a good candidate no matter who interviews them or if the same interview is conducted before or after lunch.

Given these requirements here's how we can aim to achieve them:

Efficient: We should design our technical interview to only cover topics relevant to the role at hand such that we can make an evaluation after a single interview. Depending on the exact role I'd expect a technical interview to last 1-2 hours.
Unless you can rigorously show that an having something in the interview that isn't directly related to the role is directly related to the success of candidates in the role, you should remove it.
Accurate: The technique I apply here is ensuring that the topics covered during a technical interview directly relate to the day to day activities of the role. Note that if you wish to rigorously improve your accuracy you will need access to a large pool of candidates and roles over time, in order to measure what techniques work best for your organisation.
Repeatable: The easiest way to have a repeatable process is to have a documented process. This should not only include how the interviews will be conducted, but also how the candidates should be evaluated.

Evaluation Criteria

Standardising how candidates are evaluated is one of the most important parts of having a repeatable and accurate process. To do this involves establishing a criteria in which all candidates will be evaluated against. For technical interviews I create the criteria from the following:

Knowledge: Information and understanding of a topic.
Be warned: this category is the most likely to contain irrelevant bloat, or over-specific and useless items.
Skills: The ability to carry out a specific type of task.
Attributes: Character traits that are important. In some cases you might have "failure" attributes which if exhibited count negatively towards the evaluation.

Just like the overall process, our evaluation criteria should be efficient and accurate, and thus extremely relevant to the role being hired for. My recommendation is to have at least 5 but no more than 10 criteria that a candidate is evaluated on.

When developing your criteria you should obviously cover what is required for a role, but you may also want to cover other desirable areas. This is particularly useful when hiring in generalist roles where it is likely that each candidate will have a difference mix of relevant capabilities.

Interview Questions

Now that we have our evaluation criteria we can start putting together our actual interview questions. At this point, picking questions will depend a lot on the exact technical area being examined, the level of the role, and your evaluation criteria.

The following might not apply to all questions or your specific use-case, but these are the things I keep in mind when generating and picking questions:

If you are expecting to re-use the same interview for multiple roles, it may help to generate a pool of questions. You can then select questions for the interview based on the specific evaluation criteria for the role.
A criterion should be able to be evaluated from a single question. One broad question is better than 10 narrow questions.
Questions should cover multiple criteria where possible. This is particularly true for criteria that do not have direct questions such as attributes.
Questions should allow candidates to showcase their relevant areas of expertise - especially if hiring in a generalist role as described above. This generally means broad or open ended questions.
Where possible base questions on real tasks from the role. This could be from already completed tasks by someone in the same role, or from the expected deliverables from a new role.
Can I extend this question after the candidate answers to check additional criteria?

When generating a question I find it useful to document the following:

The criteria being checked by the question.
The expected correct answers.
Extension questions that the interview may/must ask.
Red flags - known incorrect answers, negative attributes.

Interview Format

Before we start actually conducting interviews, we should give some thought to exactly how we are going to run them. Here's a few formats that you might pick from:

Standard Question / Answer: This is probably the format you're most familiar with where the interviewer will ask a question and the candidate answers. This continues back and forth until there are no more questions.
Whiteboarding: In this format the candidate is asked to present their answer on a whiteboard (or suitable substitute), rather than just verbally. This is especially effective when asking candidates about processes and designs.
Exam Style: In this format the candidate is given questions they need to answer and are left for a period of time to answer them. The interviewer then reviews the answers with them. Although termed exam, you probably do not need to treat it exactly like an exam, for example, don't ask for written essays of answers.

Personally I use a combination of styles within the same interview. I start with an exam style giving the candidate 10-15 minutes to go through the questions. I make it clear that I'm not expecting written answers but they are welcome to make notes for themselves. I find this beneficial as:

It gives candidates time to organise their thoughts without the pressure of being watched. This makes the process fairer for those who may struggle in a normal interview processes such as those with high anxiety or neurodiverse folks.
I generally want to avoid top of the head answers. The best candidate may need a little time to come up with their best answer.
For some questions such as software system architecture, allowing the candidate time to come up with the solution better matches how this work would be conducted in a real work situation.

I still utilise standard question / answer approaches when asking the extension questions, or finding more details about an answer. In fact I specifically use this approach in some questions to drive a candidate to the point where they do not know something so that I can see how they react when they don't know the answer to check some attributes.

I find whiteboarding system architectures and designs useful for my particular field as it more closely matches how that work is done and I can evaluate the candidate on how they would tackle these tasks in the real job.

Evaluating the Candidate

At this point evaluating the candidate should be straight-forward using the criteria that you established prior to conducting the interviews. I recommend both assigning numerical score and taking notes for each criteria as you assess them.

For the numerical score I usually use a fairly narrow range of 0-5 to represent how well they fit the criteria where a 3 is passable, 4 fully meeting expectations, and a 5 is above expectations. Keeping the range low helps me avoid over-thinking the score on what is a fairly subjective exercise in the first place.

I strongly recommend taking some notes as well as this will help remind you of why you assessed the candidate in a particular way in the future. These can be very useful when trying to work out which candidate you will hire as they might have different expertise in different areas.

Reducing Cognitive Biases

As we've already mentioned, in order to have a repeatable process it's important that we reduce our cognitive biases. It is during the evaluation step that this is particularly important. There are a few things that we can do to reduce the impact of biases:

Ensure that you always have at least 2 people doing evaluations during the interview. I generally aim to involve people from different areas of the business as they can bring different perspectives in their evaluation.
Note: the people doing evaluations don't have to be involved in conducting the interview, the might just be silent observers.
Do not discuss your thoughts on a candidate with other interviewers until you've written your own evaluation. Doing so negates the positive effects of having multiple interviewers.
Using both numerical scores and writing notes help solidify your thinking and reduces biases.

For more information on cognitive biases and reducing their impact in decision making I recommend reading "Thinking Fast and Slow" and "Noise" by Daniel Kahneman.

Wrapping Up

This post is already very long so I'll wrap things up here. In a future post I'll run through actual examples.

In the mean time I hope that this post has provided useful guidance that will help you in designing your own technical interview processes.