Screening Engineers: You’re Doing It Wrong

If you’re screwing up any part of your engineer interview and hiring process, it’s probably the screen.

A screening interview is the first meeting with a prospective employee — the initial hurdle before proceeding to a full on-site hiring assessment. Small startups can run with a more informal process, but a good screen is essential if you’re attempting to hire at scale.

Screening interviews are no fun. Candidate quality is all over the map, so expectations are low.

You have far less data than you would in a full on-site assessment. You have maybe an hour; you touch on a fraction of the topics; the phone impedes communication and technical skill assessment; a single interviewer’s blind spots are a single point of failure; and at this stage the candidate may not be trying very hard because they may not care about you yet.

How is a rational interviewer supposed to render a hiring opinion? Trained to target “only the best” talent, operating on uncertain data, facing censure from teammates if the candidate performs poorly on-site — the easy decision is “no.”

But check that impulse. A screen is a productivity optimization, nothing more. Its purpose is to help both candidate and employer decide whether a full interview is a good use of time. There is no danger of making a bad hire from any choice you make at the screening stage.

On the contrary, the big risk at the screening stage is rejecting someone good. The error bars on a phone screen are huge. Unless a candidate is extremely amazing or extremely terrible, you are not going to make a good hiring decision after one hour.

If one hour isn’t enough, why not spend more time? This was Facebook’s approach. Instead of a single forty-five minute phone call, we did two of them. Three if the two answers disagreed!

I’m a little embarrassed to tell you the result: we were spending more than four extra screens to avoid one extra on-site interview loop — so the screens cost more time than they saved. (We cut back to one screen as the default.)

If spending extra interviewer time defeats the purpose of a screen, could we get a more predictive screen without it?

That’s the premise behind the take home programming challenge. I love puzzles — they provide a more comprehensive idea of a candidate’s technical skills, cost less interviewer time (only grading), and eliminate “noise” from bad communication or interviewer bias. They are totally worth the effort to construct and maintain. They have just one flaw: a candidate has to commit four to eight hours of time.

If a prospect is dying to work for your company and a test is the only way she can get her foot in the door, then sure, she might spend that time. But if she is happily employed or has lots of prospects — why would she commit a day to your interview process at the screening stage when a match is still a long shot? The screening interview helps you decide to spend more time with the candidate, but it’s a two way street — it also helps establish that your interview process will be a good use of her time.

I recommend a programming challenge as one way for candidates to get in the door (Facebook hired some amazing off-the-boards engineers through its recruiting puzzle.) But if a programming test is the only way to apply, you’ll miss out on the most sought after talent.

So what’s the secret to success if we’re constrained to an hour?

There are lots of novel ideas for screens kicking around, and I encourage you to experiment. I may be old fashioned, but I favor a high quality structured screening interview with the hiring manager, founder, or tech lead. Whatever assessment mechanism you pick, I recommend one key to maximize your results: default to “yes.”

In a hiring decision, the scariest mistake we can make is hiring the wrong person. In a screening decision, the scariest mistake we can make is passing up a potential hire. With hiring decisions, when we are in doubt, we default to “no” to avoid risk. When we’re screening, and we’re on the fence, the less risky answer is “yes.”

Defaulting to “yes” doesn’t mean bringing everyone on-site; it means an emphasis on filtering out candidates who have no shot rather than filtering inonly candidates who’ll be hired. Downstream offer rates should be your guide. If your on-site results in offers for 25%-40% of candidates, you’re in the zone. At the 40% end of the spectrum, your screen is a little tight and you might be rejecting some candidates you would have hired. 20% is a lower bound at which on-site interviewers consider open revolt. If defaulting to “yes” drives you below 25%, debug the quality of your phone screen.

Why not aim higher than 40%? Let’s do some math.

Say your onsite interview process takes 10 hours of engineer interviewer time per candidate. At a 40% success rate, that’s 25 hours per offer. At 25%, 40 hours each. At an 80% success rate we spend only 12.5 hours per offer. 80% would save dozens of engineer hours, shouldn’t we want that?

Think again. That 80% success rate is penny wise and pound foolish. 20–30 hours of engineer time to hire a new engineer is a tiny investment; you’d spend way more than that on on-boarding (I hope). The interview and on-boarding time are more than repaid by the contribution made by the new engineer.

If you tell screeners to select only candidates sure to be hired, you will miss out on most great candidates, especially non-pedigree talent or anyone that seems at all risky (oh hi, minority candidates — didn’t see you there…)

A tight screen misses good hires, and doing that costs you time. You could easily add a month to the time it takes to find one good engineer. Assuming a conservative 40 hour work week, delaying an engineer’s start date by 4 weeks costs you 160 hours of engineer time… in pursuit of saving… ten or twenty hours?

High onsite to offer ratios don’t maximize engineering productivity, they maximize interviewer morale. Bringing someone onsite who fails is embarrassing, and mediocre interviews feel like a waste of time no matter what the math says. But avoiding them is an awfully expensive perk.

Better to cultivate an interviewing team that understands the odds, believes it’s worth working hard to get high quality teammates, and doesn’t beat up screeners over the misfires. Then you gain a real hiring competitive advantage — in speed, throughput, and quality.

The mistake we make at the screening stage is to act as though we are making final hiring decisions. Done right, a screening interview is an opportunity to build a strong relationship with candidates, improve the productivity of your interview team, and discover diamonds in the rough. Why do it any other way?

Related Reading:

• Step 0: What to Look For
• Step 1: Sourcing
• Step 2: Filtering
• Step 3: Screening (this post!)
• Step 4: Assessing & Deciding
• Step 5: Closing

This post originally appeared on Medium.