Founder's Guide to Interviewing Engineers

FundersClub reviews thousands of technology startups every year, and as a VC, has backed a global portfolio of top startup founders. Our insights come from our network of top startup founders and startup investors, and from our own experiences. Christopher Steiner was a co-founder at Aisle50, YC S2011, acquired by Groupon in early 2015.

A succinct rundown of our findings:

• Run as consistent a process as possible

• Talking about past projects often isn't valuable

• Don't put too much weight on pedigree and school degrees

• Ask more simple programming questions in the interview rather than complex ones

• Don't draw too much meaning from past hires

• Hone in on the skills that are extremely important to your company

• Use pair programming to determine not only skills, but compatibility

• When interviewing, go to topics beyond the workplace

For founders and early VP Engineers of growing tech companies, maintaining a pipeline of technical talent can be the most important job they have. But recruiting good engineers can be difficult. The current job market has a deep demand for developers, one that has spawned an $8.1 billion recruiting industry all built around finding and placing engineers into jobs.

Finding good applicants can be hard enough—a fact evinced by the massive payments companies make to headhunters—but vetting engineers for competency and compatibility proves more of a challenge, even for larger companies who have hired hundreds or even thousands of engineers across years of experience.

Making the wrong technical hire, especially for an early startup, can be devastating. It's the early hires that founders, even technical ones, depend on to help steer the company, build out keystone features and make critical decisions about how the product will evolve. A mistake can be expensive, so founders tend to pore over resumes, looking for the perfect mix of pedigree, skills and experience.

From there, startups and larger companies employ all manners of strategies when trying to parse one candidate from another. They give timed programming tests, they ask philosophical computer science questions, they give candidates take-home work, they do pair programming on-site, and they like to talk to interviewees about their past projects, looking for similarities to the company's current or future work.

This whole time, of course, every person in the company who meets a candidate is judging their character, gauging their personality, searching for flaws and looking for fit. It's difficult to build a dependable process around hiring for anybody, let alone technical talent, where skills sometimes aren't as discernable or apparent as other traits.

No startup has a perfect process for hiring engineers. Even those with good ones can be improved.

With that in mind, we've tapped our own experience, and talked to dozens of founders who have been working on their own hiring processes for years. We've distilled the best feedback and tips into this guide.

We received extra help from Ammon Bartram, a co-founder at Triplebyte, which has built a hiring process around evaluating engineers' skills for its clients, which include Dropbox, Airbnb, Uber, Stripe and Twitch. To do this well, Triplebyte has honed its process across 1,000 developer interviews, developing a proven set of best practices, some of which Bartram has shared with us for this guide.

Bartram and his team are consistently re-reviewing their assumptions and their system. Triplebyte has five full-time interviewers who require a full month to learn its system of evaluation. Each interview that Triplebyte does is recorded, which allows the company to ensure that each interviewer is employing the same strategies.

Triplebyte strives to create an evaluation that is repeatable, and thus a fair judgement of an engineer's skill. That fact forms our first tenet:

Consistency, above all else, is key.

Running a consistent process, one that's repeatable, is most crucial thing when doing interviews for technical hires, Bartram says. "If you don't have consistency, it's very hard to get meaningful results."

There tends to be a huge amount of variance in how companies interview people. It will sometimes depend on who is in the office, how busy it is, what projects are currently a focus for the team, and by whom the candidate engineer may have been referred.

Using a consistent scoring system that runs across both skill competency and cultural compatibility has helped the process run by Alex Solomon, co-founder and CTO at PagerDuty, an enterprise incident resolution platform. Everybody who interviews a candidate at PagerDuty uses this internal scale to provide structured feedback. In addition to technical skill, PagerDuty scores its candidates on five traits it considers key: creativity, collaboration, humility, curiosity and transparency.

Bartram recommends that candidates interview with at least four different people at the company, which can control for one bad experience or one anomalously great experience. There should be a plan for what each person will talk to the candidate about. The same ground shouldn't be covered over and over again. This gives a company a larger picture of the candidate, and saves her from having to repeat herself over and over again.

Talking about past projects often isn't valuable.

Most interviewers like to dig into candidates' past projects, and have them walk through how they did something, how they solved a particular problem. This strategy on its face seems to make sense, and people gravitate toward it, Bartram says, because it's easy and it's far less stressful for both the interviewer and the interviewee than answering quiz-like questions or going through some kind of whiteboard exercise.

The problem, however, is that talking to people about past programming doesn't tell you how they will solve problems in the future. The interviewer doesn't know how much of the described solution and process for which the candidate may have been responsible, and these conversations tend to skate above the kind of gritty details that would make them useful.

Conversations about past work that don't include a deep dive into the associated code can bias the process toward people who are good talkers, and puts people who may be less chatty but good engineers at a disadvantage. Many of the best developers undersell themselves, Bartram notes, so trying to prompt them with these kinds of free-flowing conversations doesn't always work.

So unless a interviewer has a way to take a hard look at the code of a past project with a candidate—and if it was work for an employer, doing that is usually unethical—it's not worth spending much time or placing emphasis on talking about yesterday's projects.

Don't put too much weight on pedigree and school degrees.

Degrees from good schools offer some indicator of intelligence, but by relying on those, employers miss lots of great candidates who may not have such credentials, a fact that greatly increases the number of false negatives a process will produce.

Triplebyte uses a blind process that attempts to judge engineers on nothing but their skills and readiness to contribute. That means interviewers don't see a candidate's resume. They'll ask technical questions and dive into code, but they won't know if a candidate majored in computer science, civil engineering, or English. Or if they majored in anything at all, or even graduated.

Triplebyte has found highly qualified developers who were working unlikely jobs:

• Minimum wage line cook at a fast food restaurant
• Sales at a furniture store
• Grunt labor at a warehouse
• Driving an ambulance

In some cases, these people were self-taught and never acquired the intellectual paper that most companies require to attain an engineering job. "It's incredible where some of these people are," says Ammon. "But these jobs can change peoples' lives, which is pretty fun."

Bryan Smith, CTO of CyberGRX, which helps companies manage third party cyber risk, tends to agree with this approach. "In my opinion, most of the KSAs I look for are not obtained during pursuit of an undergraduate or master's degree," he says "Rather they come out of experience and learning from past mistakes."

You can make a take-home project an option for candidates, but don't require it.

Most of the best engineers with impressive resumes won't do a take-home engineering test. They don't have to. So by requiring that candidates do one, companies quickly eliminate many very qualified applicants.

Some engineers may prefer to do one of these, however, as they may not work well under pressure with somebody standing right next to them. Going back and doing a deep dive on the code a candidate wrote for your project is a great way to get a feeling on how this person solves problems and puts together solutions. One out of 5 engineers that Triplebyte interviews opts to do the take home test rather than answering more technical problems in person. Bartram considers the results from both processes to be useful.

Ask more simple programming questions in the interview, rather than complex ones.

Even if the question is one that 90% of developers will get correct, the way a candidate answers it offers a better, more dependable signal for aptitude than asking everybody a hard question, which many candidates may not solve at all, says Bartram.

An engineer who answers the question quickly, without probing his mind too deeply, is likely more practiced and skilled than an engineer who has to think about the problem for five minutes and scrape her way to the correct answer. Both engineers, however, could have been stumped by a difficult question, which wouldn't have offered the same quality of feedback for interviewers.

It's also helpful to pose questions and problems with multiple parts. Almost all candidates get stuck at some point in a technical interview, so it's helpful for all parties if there is a second part and even a third part of the question that, if the first solution is given to the candidate, they can work on find the next one. This lessens the pressure on the candidate, and gives interviewers a more complete picture of the candidate's skills.

Don't draw too much meaning from past hires.

Lots of startups that may have made five to 10 engineering hires will use their experience as a guide to future hires. While this tact seems perfectly reasonable, Bartram says that the data don't support it. Eight or so hires simply isn't enough data; it's not a significant sample size. So even though a founder may have found two good engineers from a particular school, it isn't significant enough to treat a degree from that school as a true signal, just as having a bad experience with a single engineer from a school also doesn't indicate anything significant.

"Companies will often develop biases via chance or via insufficient data," says Bartram.

Similar effects to hiring decisions can creep in passively, too. Engineers, just like anybody else, tend to favor people who possess experiences and skills similar to their own. This bias will make its way into evaluations whether a company realizes it or not, so it's something that has to be controlled for, or else much of the meaning in an evaluation has been compromised.

Hone in on the skills that are extremely important to your company.

Interviewers need to acknowledge that everybody is bad at something. Asking questions to gauge an engineer's average aptitude across a number of subjects sets the candidate up for failure, and it doesn't do much good for the interviewing company, either. The point is to hire people who are good at something your company cares about.

Triplebyte's data show that a candidate's highest rating in a single area is a better overall indicator of their worth as an engineer than their average overall score. If you ask somebody about back-end architecture and that's their strongpoint, they should rock the question. They might do very poorly on a question about front-end javascript frameworks, but that's okay—and it shouldn't place them behind a candidate with middling scores on both questions.

Use Pair Programming to Determine Not Only Skills, But Compatibility.

Chris Gervais, co-founder and VP of engineering at cloud security startup Threat Stack, likes to use pair-programming exercises throughout much of the interview process. And while the quality of the code a candidate produces during these sessions is important, he says, it's not the most important signal he tries to glean from this. He's trying to get an inkling as to how the candidate works with other engineers, regardless of skill level.

"We want team members who are excited about collaborating with their coworkers to solve problems, not a lone wolf who can’t be bothered," Gervais says. "Are they open to varying opinions about how to solve a particular problem and can they incorporate that feedback as they progress through?"

Doing pair programming gives Gervais a clear view to a candidate's adaptability, whether they ask for clarification or help quickly, and what they do when they get stuck. "We don't want people banging their heads against the desk when a five-second call for help on Slack would remove a blocker," he says.

Candidates who impress the team at Threat Stack are those who buy into a supportive and collaborative environment. The only way to figure that out, other than through reference checks—which don't always tell the whole story—is by working on a problem side by side

Go beyond the workplace.

Some teams find it helpful to include within their inputs a candidate's outside interests. Those with deep passions for things artistic and creative, says Rich Green, the chief product officer at SugarCRM, often have intangible skills that correlate well to a software team. "It's hardly a requirement," Green says. "But it's an example of an interesting factor in the evaluation process."

Alex Kubicek, CEO of Understory, a weather analytics startup, likes to ask developer candidates about their personal engineering projects, ones they've done outside of work. "The best developers tend to be the ones who are really passionate about what they do—and they often have stuff they've been working on in their spare time," he says.

These side projects are usually hosted on Github or another repository that Kubicek can look through with the developer without breaking any rules regarding other companies' IP and code. It's a good work-around for teams who like to talk about candidates' past work without making the mistake of not diving deep enough on it.

Larry Gadea, founder and CEO of Envoy, which buildsiPad-based visitor registration software, also looks for engineers who tinker outside of work. "If they code on the side, we want to talk to them. If not, we pass," he says.  "We also appreciate eagerness. If they come to the interview having built a plugin or something to demonstrate they're willing to go above and beyond for us, before even getting a job, that's awesome."

In summary:

1. Vetting great engineers from an applicant pool requires diligence and work. It’s a job. It’s best completed with a consistent process that involves a full team of people from the company. Interviewers should focus on skills and compatibility, and try to ignore resumes as much as possible.
2. Collaboration is an important skill for engineers, one that can be tested for through pair-programming. Try to focus on candidates’ methods and thought process, and pay greater attention to technical areas where candidates’ aptitudes are highest and on subjects that are most important to your company. Past results mean little to a new process. Good engineers come from all kinds of backgrounds, with just as many learning and work experiences.