I’ve been hiring folks in software development for about 20 years now, 15 of them remotely, so it’s no surprise I’ve accumulated some opinions over time.

I wanted to share a bit what we’ve done at ShipHero over the last 3 years, where we’ve experimented often with different approaches that have succeeded and failed in different ways. The more time goes by the clearer it becomes to me that there’s no one-size-fits-all approach to it, mostly because companies, which are just groups of humans, work in surprisingly different ways.

I do strongly believe that there’s one thing that will generally work better than average in healthy environments: testing for collaboration & communication skills. I don’t think that’s a surprising revelation to anyone who’s been building teams for a while, the reason why I think it’s infrequent that interview processes take that heavily into account is because it takes a lot of time and energy to find out if someone’s good at collaborating.

It did take me a long time to realize just how easy it is to test for technical skills vs. skills like collaborating with others, and how technical skills can be improved quicker than a person’s understanding on how to work well with others. I think you can probably get most of what you need to know from a person’s technical experience in 30 minutes, but everything else just requires way more time to figure out. And the hard truth is, a person who’s able to work well within a team but lacks technical expertise has a much better chance of thriving over time than someone’s who’s brilliant but unable to build things with others.

What we tried

Once we agreed internally on that premise we did a multi-month experiment and changed the way we hired in a pretty drastic way, in an attempt to emphasize collaboration skills over everything else. We mapped out a week-long process:

• Introduced a dozen questions into the application process and made submitting a CV optional. We didn’t look at CVs or names at all during the pre-selection process.
• Because we were asking people to spend a lot of time on our process, we committed to paying people $500 if they went through the process regardless of whether they were offered a role or not. We wanted people to feel like we valued their time.
• A brief introduction to vet basic things like English fluency and a very shallow sanity check that the information they submitted somewhat reflected their experience.
• A project to work on together that resembles the type of work that’s done in our day-to-day. We expected the project to take a couple of days with about 6-12 hours to complete and require interaction over Github, Slack and email to ask questions and for a developer on our side to have to collaborate with a few snippets of code to make the whole thing work. The goal here was to see how someone works asynchronously at their own pace while still being able to communicate about what they’re doing and why.
• If the first project showed good collaboration and a validation of their technical skills we moved on to a second, shorter and easier project that we’d try and do more or less in real-time. We’d build something small on top of the previous project and schedule a time range to work together on it. This could be over Slack, on a call, pairing, or a mix, whatever the candidate is comfortable with. The goal isn’t to see how someone works under stress, an interview process is a terrible place to test for that, but rather to validate they’re not farming out the test to someone else and to see the person collaborate in a more real-time environment.
• Last step is meeting the team they’d be working with, give them an opportunity to see if the group of people they’d spend most of their time with is what they’re looking for. This usually goes both ways, but by the time someone’s spent that amount of time with us we’re fairly certain they’ll fit in well in a team.

It took some time to get the process set up, and it took a lot of our developer’s time to support it. We loved it. The people that went through it and got to the end told us they enjoyed it. But very few people applied when they understood how much time they’d have to invest in our process, many (half or so) dropped off mid-way through, and in general people with a lot of experience didn’t apply.

We were going through a heavy growth phase where we needed as many good developers as we could find, and while we felt our new process was much better, it attracted less people and wouldn’t allow us to hire at the rate that we needed at the time. So we stripped the process into the smallest piece we felt would still capture some of what was important to us while making it less demanding on candidates’ time.

What we’re doing today

We provide some flexibility to each hiring manager in how they chose to run interviewing processes, our general approach today is:

• We kept the list of questions instead CVs for applicants, it turned out to tell us much more about candidates than their CVs ever did. We took what we looked for in CVs into questions and let people answer for themselves upfront instead of us trying to extrapolate and apply our own biases to them. The list of questions are listed at the end if you’re curious.
• The hiring manager will have an introduction call with people who seem promising, same as before, basic vetting and let them ask whatever they want to know about the company. We’ll also explain the hiring process so they know what it entails.
• We’ll send them over a small project that will take anywhere from 2-6 hours to complete, they can do it at their own pace and we encourage them to ask as many questions as they’d like. We don’t try and test here for anything more than basic technical skills, it’s much more about problem-solving and communication skills. We’d still extend an offer to someone even if they got blocked a dozen times throughout the project. People often get nervous while looking for a job, there’s no advantage in filtering those people out.
• Once they send us back the project, we’ll spend some time reviewing it and set up a call to discuss it. This is usually what resembles a technical interview the most, but instead of trying to talk about abstract technical concepts we can discuss something we’re both familiar with.
• If it all went well, we’ll extend an offer.
• All the people involved are folks you’d be working with. We don’t farm off any part of the process to an internal recruiter to filter out people.

The process ends up being two or three 30-60 minute calls, with the project in between them. It works fine, we generally hire people who end up sticking around and being great additions to the teams.
Ideally we’d want to have the longer-form process, we felt we both learned more about people, but maybe more importantly we saw people who we did not think that would be good candidates initially come way ahead of others. It felt like the smarter way to hire. We haven’t figured out a way to make the math work, maybe it’ll be easier once we’re a more well-known company and more people are willing to invest time in the interviewing process, maybe eventually we’ll find a way to sell people on the idea.

And of course, we’re still hiring!

List of questions to apply for a development role

• A Cover Letter
• How many years have you been developing software? How many of those with Python?
• Ideally, what do you want to do at work most of the time?
• How many people were on the teams you worked in in the past? (Provide a range (ie., 2-3 people, 8-10, etc). We mostly want to understand if you’ve worked primarily on your own, small or large teams.)
• What is the longest stretch of time you’ve been at the same company? (Do you like to work at companies for long periods of time (3+ years)? Do you prefer shorter periods?)
• What country do you live in? (We’re generally ok with people moving around, this is to understand where you live right now so we know what timezone & legal framework we can expect.)
• What is the largest scale of usage you’ve had on an application you have actively developed? (# of users, # of requests/sec, any metric that produces some scale. A service with 1,000 daily users? 1,000,000? Few users but millions of API requests?)
• When was the last time you changed your mind about something important? Why did you change your mind?
• What was the hardest part of your current/last role? (What was challenging for you?)
• What’s a challenging project or bug you worked on? Why was it challenging?