- a better way to interview

Summary

Recruitment is expensive and time-consuming. It risks inducing interview fatigue in both senior staff and potential new employees. Current ways of interviewing could be more interesting all around and better at identifying talent, if we shift our mindset from looking weaknesses to exploring strength, potential and team fit.

One way to give both interviewers and candidates a better understanding of each other is to replace the standard coding task with a take-home assignment, to be discussed in-depth in the face-2-face technical interview.

In this article, I'm exploring my experiences from both sides of the interview table, before suggesting a better way of conducting tech interviews.

Intro

Over the years, I've interviewed countless software developers, tech leads and engineering managers. I started out interviewing when I was a developer, with no clue what was going on, and, with time, as a hiring manager - hopefully with more of a clue. I've sat in on interviews to learn from others, been mock interviewed and conducted mock interviews, and mentored developers new to interviewing.

I've even had the opportunity to put together and lead a team to design a recruitment program from scratch. In short, I thought I was pretty good at telling what good interview questions are and how much time to allocate for a tech interview.

And then - when I recently started looking for a new role, I realised how different it is to sit on the other side of the table. I’d forgotten how nerve-racking it is, and how the nerves affect my ability to think. It’s embarrassing to perform less than perfect in front of an audience. As the adrenaline rises and the brain freezes, I can no longer program myself out of a paper bag - let alone solve an entry-level problem.

Sometimes, the easier the question, the harder it is to do well. I don’t mind (much) if I miss an interview when I’ve been able to showcase my abilities and thinking. If I miss an interview because I cannot for the life of me remember an algorithm I learned at uni decades ago - let's say, create all possible combinations of one letter from each item in an ordered array of strings without using built-in functions - it’s annoying and down-putting.

What’s worse, it sets me up for failure in the next interview. I start to anticipate the stress days in advance. As soon as I join the meeting, the adrenaline starts pumping... Hello brain-freeze my old friend, goodbye logical thinking. When an interview doesn’t go well, I blame myself. If only I had prepared properly, I’m out of practise, I should have revised more… it’s all my fault.

- K?
- Yes, Tiny Assistant?
- Why do you think it’s all your fault? You know you know your stuff. You’ve got this.
- Thank you, Tiny Assistant. I’m glad you believe in me. It’s called Imposter Syndrome. Don’t look now, but they’re the one who sits on my other shoulder.
- … Oh. That one. I don’t like them.
- I did say don’t look. I don’t like them either.
- I don’t want to think about it. Let’s eat cheese instead.
- Good choice, Tiny Assistant. If in doubt, always eat cheese.

… or is it?

I’m very fortunate to have a down-to-earth mentor. I casually mentioned in a chat how stressful I find interviewing - and they replied…

“I know what you mean. I hate coding in front of people. I just can’t do it.”

It made me think. If one of the cleverest people I know hates interviews, because they feel like a failure and can’t think straight when someone is staring at them when coding… then maybe the problem isn’t just me. Maybe it’s a systemic issue with how we conduct interviews. So I started asking round. And guess what.

Everyone agrees (yep, you read that right. I literally talked to everyone, and they all agreed!). I even got the same feedback from a hiring manager during a recent interview, when I made a casual remark about my nerves:

“Yes, we all know it’s not great, but it’s the only tool we’ve got.”

Fortunately, I don’t think it’s the only tool we’ve got. Let’s explore what a good interview experience can look like.

- K?
- Yes, Tiny Assistant. I think I know what you’re going to say.
- You didn’t literally talk to everyone, did you.
- No, Tiny Assistant, you're right, I didn’t. It’s called hyperbole for comedic effect. But I did talk to a handful of very clever people. And that's virtually the same as literally talking to everyone.
- You think you’re funny, don’t you.
- Here. Have some cheese.

Good interviews, decent interviews, and some with room for improvement

I’ve had a range of varying interview experiences, on both sides of the table. Some were good, and I enjoyed the experience, even learned something new. I’d happily interview for them again if a new role comes up in the future.

Others were not so great. I left feeling unfulfilled, for various reasons. Sometimes I let myself down (interview fatigue is a real thing). But more often, I felt unsatisfied because the interview format wasn’t ideal for the role.

It's ok to not pass an interview when the role and my skills aren’t a good fit - the sooner we can figure it out, the better. I do mind not progressing to the next stage when the interview format doesn’t give me a chance to showcase my ability and potential.

Likewise, as an interviewer, it is unsatisfying to leave an interview without knowing whether the candidate genuinely isn't suitable, with a sneaking suspicion that they were too nervous to do a good job, or a feeling that the tasks we used didn't really explore relevant knowledge and give them a chance to shine.

As an interviewer, it’s hard to focus on yet another interview using the same tired task that you have used in twelfty other interviews this week. Each candidate deserves the same commitment and courtesy, no matter the time of day, or how stressed I am about running behind in my day job. Unfortunately, we’re only human, and interview fatigue is a real thing for interviewers too.

- K? Do interviewers have to be hooman? Could you not use… let’s say, an AI-generated rat who works for cheese?
- Great idea, Tiny Assistant! If you keep reading, you’ll see that I suggest using online interviewing sites for technical pre-screening. They take the slog out of interviewing, and when used well help make the face-2-face more interesting.
- Oh... so no space for a cheese-motivated interviewer then.
- Don’t worry, you’re already doing a great job as proofreader. More cheese?
- Don’t mind if I do. Thank you.

Good interviews share some common traits:

Great communication throughout the interviewing process.
Explain each step and when to expect feedback. Give preparation advice and set clear expectations. It enables candidates to prepare and they know what to expect when they come to the interview. It also empowers them to make an informed decision to withdraw early if they feel the fit isn’t right.
There are several interview steps, each with a single, well-specified focus and enough time allocated.
When the interviewer is given just under an hour to introduce themselves and the company, ask questions about experience and leadership skills, present a coding challenge, watch the interviewee code it up, have an in-depth code conversation and leave enough time for the candidate to ask meaningful questions… chances are neither they nor the candidate have a chance to shine.
The interviewers are well-prepared, engaged and interested.
They help the candidate feel safe, and they participate in the discussion. They are interested and curious, and want to get to know the candidate. They have prepared well, and have time and energy to spare. It’s clearly not their fifth consecutive interview that day.
The interviewers focus on the candidate's strengths and potential.
It's far more telling to talk about an area where the candidate is strong, than chasing their weakest points. Give the candidate the opportunity to prepare, and they will both represent themselves better and get a better impression of you and your team.
The working culture is demonstrated first hand.
As a candidate, it’s awesome to get the chance to ask any questions about the company and the interviewer’s experience working there. However - an opportunity to experience the business domain and the kind of features they work on IRL is unbeatable. A good interview gives a flavour of the real-life problems the company works on, and how you solve them.
The questions and discussion points are relevant for the job
You'd think that's a given, right? It's not. Surprisingly many companies haven't tailored their tech interview stage sufficiently, especially for senior hires. Spend the extra time up front to figure out tasks that demonstrate the skills that are truly important for the role you're hiring for. Is it truly important for an engineering manager to remember a search algorithm they learned a decade ago? Or can we use the time better to discuss more advanced topics?

A better tech interview

Many standard technical interviews are designed to catch people out, instead of allowing them to show off their worth. The tech interviews I like best, both as an interviewer and an interviewee, focus on strengths and potential, not weaknesses. They allow both the company and candidate to understand if they are a good fit for each other.

So how do we both check that a candidate can code for toffee, and focus on the interesting bits, such as thought patterns, problem-solving approaches, growth potential and general fit with the team?

The answer is a substantial take-home assignment, submitted either through github or an online interview assessment website, followed by a face-2-face technical interview. It has several benefits:

it provides a less stressful experience,
the candidate can use tools they are familiar with,
they get a flavour of real-life problems that the team faces,
it sets the face-2-face up to enable a meaningful discussion about a well-prepared piece of code that both the interviewer and the candidate know and understand.

It’s as close as we can get in external recruitment to seeing real code that they have delivered to production.

Preparation, information, and follow up

Give candidates a week to prepare.
Explain the process in detail in writing. If you use an online coding interview website for assignments, link to it so that candidates can familiarise themselves with it. Give an example assignment to practise on.
Follow up with candidates, before and after the assignment.
Make sure they understand the process, and check that they’re not experiencing technical difficulties. Can they commit the necessary time? Do they have any additional accessibility needs?

The follow-up after the assignment is especially important for candidates who score low or drop out. I’ve seen a fair share of candidates drop out at the first hurdle because they couldn’t get the online test environment to work, without raising it with the recruiter. I don’t want to miss out on a clever grad with little interview experience who has been disheartened by technical issues.

But time!, I hear you say.
It's not as time-consuming as it sounds. It’s nice if a recruiter can take the time to call up and check in, but automated emails will do well too. In a recent job, we’d get hundreds of applications for our graduate intake. It quickly became overwhelming to manage and keep track of. We set up a simple recruitment database to keep track of applicants, and sent out automated e-mails at different stages of the recruitment workflow. The e-mails were tailored to the outcome of the interview stage. Once set up, recruiters and interviewers could get an overview, leave feedback, synchronise, and progress candidates with minimal manual effort.

Good communication and timely follow-up give your candidates a positive experience of your company, whether they were successful or not. Chances are they'll give you positive feedback and spread the word.

Pre-screening using an online coding interview website (optional)

Face-2-face interview time is valuable, both to applicants and interviewers. We want to use that time wisely, to understand the candidate's thought processes and mindset, and figure out if they're a good fit with our team.

Unfortunately, a lot of the tech interview is often spent solving a trivial technical problem. It kind of makes sense when interviews are conducted in the office - greet the candidate, set them up, then go and work for an hour or so while they complete a task. It doesn't make sense when tech interviews are conducted in a conference call. So let's just get the mechanical part of the interview out of the way before we even get to the face-2-face stage.

Use a simple pre-screening coding assignment to test for basic coding proficiency: use of loops, arrays, hash tables, searches and filtering. You’ll be surprised how many applicants can’t write a for-loop.
Remember to tailor the tasks for different roles and experience levels.
Make sure the tasks and complexity are relevant and suitable for the role you're interviewing for.
Ideally create your own tasks, don’t use the pre-defined ones.
Many of them rely on the coder having seen a similar problem before.
Interview technique training sites are incredibly useful - but it means many candidates are code kata pros. They are great at solving clever puzzles under time pressure, but less great at writing production-ready, maintainable, testable code that solves issues that real users will appreciate. Also, it's likely there's an (AI-generated) solution out there to copy and paste.
Keep the tasks straight-forward and provide test data.
Give good explanations of the expected behaviour, with plenty of well-designed test data. Don’t be tempted to add complexity that requires the candidate to spot a twist, or think of a clever algorithm. If they fail, it just means they didn't think of the clever trick. If they pass, it’s likely they’ve encountered it before. Ideally, we want to hire great developers, not people who have perfected their interviewing skills.

- K? What if someone uses AI to solve the pre-screening?
- Yes, that's an issue, isn't it. It essentially falls in the same category as code kata gurus - how do we filter out AI submissions and people who smash interviews, but can't code?
- Can't the interview site use ML to find commonalities?
- Yes, they do, and it's helpful to an extent. This is the main reason why I'm inclined to skip the pre-screening and go straight for a bigger take-home assignment instead. With a take-home assignment, we can discuss the submitted solution in the face-2-face. It becomes pretty obvious pretty quickly when someone has submitted code that they don't understand.
- But why would you set a task that can be solved by AI? You could just hire me instead!
- Exactly. Better to go straight for tasks that aren't too mechanical - that said, there are situations when pre-screenings are useful, e.g. when you're hiring a lot of grads at the same time.

The fun part: the take-home assignment!

The take-home assignment is fun and interesting for everyone involved. It's designed to give a flavour of the kind of problems we face at work, and be the starting point for interesting discussion in the face-2-face tech interview. We give the applicant enough time and tools to solve it without stress. It isn't a slog, and it combats interview fatigue. If any of those statements aren't met - back to the drawing board we go.

When the candidate is ready for their tech interview, we ask them to solve a domain-specific problem that is representative of our daily work, both in terms of our business domain, and common technical issues that we face.

The take-home assignment should be substantial, but achievable in 2 - 3 hours - we will assume that applicants have lives outside the interview process. We're asking them to dedicate a fair chunk of time to the task, so it needs to be rewarding for them as well.

Case study: Health tech

A health tech company designed a take-home assignment with two small features to find patients whose symptoms and medication aligned with the requirements for a drugs trial. The data was provided in comma separated strings, and the data formats were well-defined in the feature description. Test data examples were also given, some open, some hidden, to test happy paths and edge cases.

The task was straight forward but needed knowledge of e.g. loops, maps and filters. Candidates were given 2 1/2 hours to complete the task, and could choose to work in an online IDE, or submit via github. In the latter case, there was an additional task to complete.

Since the candidate can complete the task in their own time, without an assessor watching, the environment is less stressful, and they are likely to perform better. By investing their time, they show commitment and that they are serious about their application.

In the following face-2-face tech interview, we want to use the submission as a starting point for discussion. We figure out if they genuinely understand the solution, can reason about it and change it according to new feature requests. At that point in the interview, the candidate is hopefully relaxed and understands both the problem and the code well enough to make a good job of it.

Case study: Health tech

The patient data input was essentially just a huge string, giving plenty of scope to talk about how the candidate would redesign the code in different scenarios, depending e.g. on performance, data size, etc.

Would you use a database, and which kind? How do you design a DAL? What if we want to run this as a Cloud-deployed microservice? How can you test it? Etc etc.

It provided plenty of scope for discussion, depending on the level and experience of the applicant. It is also easy to design new requirements and changes, based on the submission.

They may have discussed the solution with a friend, and looked up information in the process - but let’s face it - isn’t it a good thing for a developer to use all available resources, rather than reinvent the wheel? For me, it’s a more realistic set up, that, when executed well, tells me more about a candidate than a coding challenge task under time pressure.

There are a few nitty-gritty details to decide on:

Agree on a clear deadline, with clear instructions how to act if things don't go to plan.
After you send out the assignment, follow up with the applicant that they understand the assignment and can dedicate the time. Get them to commit to a deadline, usually 4-7 days away. Be clear that you expect submission by the deadline, and what they should do if they encounter any issues.
This is an important measure of how they work against deadlines, and whether they communicate potential blockers clearly. Decide in advance how generous you want to be. If they have good reason why they can’t deliver, I’m ususally happy to agree on a new deadline and give out a different task. Be clear that if they don’t deliver and don't communicate, it will affect their chances to progress their application.
Do you insist on an online IDE?
Some companies use the same online interviewing sites as for screenings, with limited access to documentation, debugging tools and e.g. refactoring support. It has its uses - it's easy, the same for everyone, and you get support checking for duplication.
A nice feature is that you can roll back and watch the code evolve over time - sometimes that is nice when you try to understand when and how things went awry.
A big con for me is the crude debugging tools - many sites rely on printouts to stdout to debug and test during development. For a TDD addict, it's like being transported to one of Dante's special circles.
... or can they use their own IDE?
Personally, I prefer to let candidates have the choice of using their own IDE and submit their git repo, including tests. It’s a more natural working environment that allows them to focus on the problem and not the tooling. It gives a nice insight into their coding style and organisation. Do they provide tests or is it all println-debugging? (No tests - no pass... ;-)
Allow use of tools and documentation.
I strongly encourage that you don't try to police tooling and use of documentation. It's not possible anyway, and it puts people at a disadvantage. With access to my usual tools, I can use my brain cycles for problem solving and code design, instead of trying to remember the exact order of parameters of some nifty library function.
Make sure the tasks are sufficiently interesting and complex so that cheating won't help.
If a candidate asks StackOverflow, AI, or their mate the superhero programmer to submit a solution for them, but they don't understand it, they'll not be able to reason around the code. When you design the tasks, remember that the end goal isn't to receive a correct solution to a coding task, it's to ensure a meaningful conversation in the tech interview. Design for discussion.
Have a set of different tasks that you evolve over time.
Let's face it: if you did a good job coming up with tasks, by interview number N (where N is << than infinity) you'll start seeing better and better submissions. Subtly change the tasks, and have a few different ones to switch between. It keeps it interesting, and your team will love to help come up with ideas if you let them.
Allocate sufficient time.
I usually stumble and fall on this hurdle when I design tasks.
How hard can it be, right? If I solved it into 20 minutes, then surely 40 minutes is sufficient? Think again.
Let t(I) = T (minutes) be the time you need to code up a solution. Allocate t = 3.14 * T, and ask a colleague to solve it. Discover that t(colleague) = x * t(I), where x >> T. Double the time again and give it to another colleague, only to realise that they need twice that time again...
Be sensible about time. This isn't a time trial. The main reason we set a time limit isn't to test the candidate's ability to code under time pressure - it's to free up our interviewers, and to not pull interviewees out of their day job for hours at a time. With a week-long take-home assignment, this concern is less relevant. Most candidates should be able to dedicate 2 - 3 hours at a time of their choosing.
Validate the tasks and interview workflow
Arrange mock interviews with colleagues who are at the level you are hiring for. It’s good practice for them, and you’ll get good feedback on the assignment:
- did they enjoy it?
- is the time allocation right?
- does it test the right skills?
- does it allow for meaningful in-depth questions around the solution?

Tailor the assessments to the role

Make sure the tasks and complexity are relevant and suitable for the role you're interviewing for. This is the first real opportunity for your potential new colleague to get an impression of your team - represent yourself well.

Are you hiring grad students going into their first developer role?
Choose something relatively simple, that gives them a chance to demonstrate basic programming skills, perhaps a design pattern, and their thought patterns.
Looking for a new architect?
I liked as assignment to design a new feature for an existing cloud-based music streaming platform. You can choose to talk about API’s, microservices, data access, databases, latency, design patterns, security…
It even works when interviewing Product Managers.
Ask them to create user stories from a feature description. In the interview, role-play a sprint planning session with the developers. See how they work and communicate.

Congratulations! You are all set up for a fun, meaningful interview!

You're now set up for a face-2-face technical interview that is similar to interviewing an internal candidate who can showcase production code. You have a piece of work to discuss that both you and the applicant understand, and that you can discuss in-depth.

Use it to its full potential. Drill into the solution. It provides a window into the quality of code (and tests) the candidate provides when they can work on a problem over time and without anyone breathing down their neck. You can see their style, how they think, where they have put emphasis and effort in, and where they haven’t.

Go in-depth. Ask questions about the design and the solution. Are they happy with it? Ask them for a code review of their own code - what things would they change and why? Drill into the bits that aren’t quite 100% - or that are a bit too good to be true. Talk about their testing strategy, and the complexity of their algorithm. How do they handle edge cases and errors? This is a small code example - how would they handle big data sets?

Follow up with a feature request, asking the candidate to add/tweak a feature or improve their solution. This is an important step, since it verifies if they truly understand the code, or copied it off someone. Sure, nerves can and will play a role here, but hopefully less so when they are already familiar with the code they’re working on.

I can guarantee that you’ll have an interesting discussion, and you stand a good chance to figure out if the person has skills, values and potential that fit in with your team.

From the candidate’s point of view, they get a flavour of the domain, the kind of problems you solve and how you work. It won’t be perfect - but it gives a better idea than, say, writing a function to print out the first n fibonacci numbers (unless of course you are hiring teaching assistants for a university course in computer science - if you are, then go ahead! Fibonacci ftw!)

Case study: hiring engineering managers

When we interviewed for engineering managers a few years back, it was important to us to hire talented coaches and mentors, who could also roll their sleeves up and practically guide new developers on how to work professionally in a team. We specifically didn't want post-technical managers with little practical experience and know how.

It was important to us to understand:

how our future colleagues think about design and code quality
how they deliver feedback to a junior developer
that they can write decent code

We designed an interview task focused on code review and design, which also demonstrated ability to code. We set up a repo with small, overly complicated and difficult-to-read solution, which was more or less impossible to change without breaking the functionality. We provided some scant tests that only just tested the happy path.

There were no wrong or right answers. The task was tailored to allow for discussion and exploration, rather than expect a specific answer.

- Surely there was a wrong answer? There's always a wrong answer.
- As always, you are right, Tiny Assistant. There was a wrong answer in the interview:

"This code is fine, and I wouldn't change anything."

... you'd be surprised how many submissions we got to that account, from experienced team leads and engineering managers. The code really wasn't fine. And the lack of sensible tests made my skin itch.

The candidates had a week to submit a code review for a fictitious junior developer in git. We also asked them to clone the code and improve it based on their own review.

Before the face-2-face, we reviewed the comments and looked at the suggested improvements. Did they catch the most glaring design problems? How do they communicate the feedback? Did they deliver a decent code improvement?

In the interview, we talked through their feedback and solution, and their approach to testing. We asked them to add some new functionality. We discussed leadership, and how to manage performance.

This process had varying success. Some candidates clearly didn’t pay any attention to the task until the night before and came unprepared to the interview. Some didn’t see any issues with the code and only gave feedback on e.g. capitalisation of class names. Those candidates generally didn’t do so well when asked to modify the code, and often gave the feedback afterwards that they hadn’t realised that it was a hands-on role.

Some thrived on the challenge, clearly enjoyed the conversation, and went on to become highly rated colleagues.

In light of this, we iterated over the assessment, invented different tasks, and improved the instructions to have clearer expectations.

The interviewer team solved the tasks in different languages to make sure it was achievable, and we gave each other mock interviews. I, for one (and hopefully my peers too) enjoyed both the process and the interviews. It was interesting and developing to see the solutions of my colleagues, and discuss their approaches. Hopefully I became a better interviewer as well.

Conclusion

By introducing a take-home technical assignment, we can create a more relaxed interview environment. It makes life easier and more interesting for interviewers and interviewees alike, minimising the influence of nerves on the interview process. It gives us a chance to have a deeper conversation and assess not only on technical skills but the overall skill set of a candidate for the job, as well as giving them an insight into our business domain and team culture.

I hope you've enjoyed this article. Hopefully you've gained some new ideas that you can use in your daily life. Find me on LinkedIn to give feedback and keep the discussion going! I'd love to hear about your experience and ideas for better interview practices.

Kicki Frisch

Tech interviews from both sides of the table

- a better way to interview

Summary