Let’s say you want someone who is good at setting vision and direction for a large team. A question like “How would you set vision and direction for a large team?” is only going to give you hypothetical answers.
All you will learn is how they talk about doing it, how much they’ve read around the topic of vision-setting, and also how much what they say chimes with what you believe. But we know that planning to do something a certain way and actually doing it that way are two very different things. They might say all the right things, but then not be able to follow through.
A better question would be “Tell us about a time that you set vision for a team”, because that will get them to talk about what their actual past behaviour was, which is a much better indicator of what their future behaviour will be. It also gives you an opportunity to ask questions to find out more what their approach was, like “How did you do it?”, “What was the result?”, “What didn’t go so well?” None of those questions are hypothetical, so you’re not asking the candidate to make something up that you want to hear, you are asking them what they did, and how they thought about it.
Note that it doesn’t have to be the exact same thing, because chances are this job will be a step up in one way or another. You need to work out what the important part is, and ask about that. So my question above was about setting vision for a team, not for a large team, because the important thing I wanted to know about was vision-setting.
In order to make the most of the process, you need to give a score for each question as soon as you can. Ideally immediately after the interview, so that you can remember what has been discussed.
This helps you focus on how they performed against each of the important skills, rather than being influenced by what you thought of them personally, or your general sense of whether they’d be good (which will be informed by your unconscious biases).
This also helps address the primacy/recency effect, where you will think more highly of them if they answered the first and last question well, or if they were the first or last candidate.
An example is to use a score of 0-3.