Job candidates applying for tax consulting, risk advisory, and financial advisory roles at a major consulting firm were recently asked to play an outer space game, dodging asteroids and imploding stars from a virtual starship. Players must complete six levels of intergalactic challenges in 30 minutes.
While it appears to simply test their ability to swipe and aim, the game’s creators say it is a psychometric test that measures cognitive processes such as resilience, problem-solving, how candidates respond to given situations, and whether they are a good fit for the company.
However, as one player posted in an online discussion forum, “It is a relatively straightforward process, but it is very mentally taxing, and in my tired state I worked myself up into a panic.” A pressure-induced panic is probably not the best state in which to assess applicants. It may be true for jobs that entail high pressure, such as first responders or emergency room personnel. But is it the best way to uncover financial consulting excellence?
While games can be fun and can stimulate engagement, it is important to show that a selection process is relevant to the job candidates are applying for. And it’s important to consider how serious job candidates might feel about being required to play a fantasy video game to win a professional position.
The Rise of Gamification and How to Assess the Different Technologies
Gamification is one of the hottest buzzwords in recruiting right now, and describes the idea of applying the mechanics and psychology of video games to business processes. When it comes to selecting the right job candidate there are three categories of technology – as summed up in this post (see “Three New Categories of Assessment”) – but recruiters should also ask five questions before making any investments.
What does it measure? It is often unclear what is measured in a game and whether it yields useful information about a candidate. Some gamified assessments in the market allegedly measure dozens of behavioral traits, such as a propensity to take risks or perseverance.
For example, a candidate may have to pump up a virtual balloon to make more (fake) money or decide to cash out before the balloon pops. Given the contrived nature of the scenario, does this really have any bearing on job performance? Furthermore, applicants can guess at the intended answer and perform accordingly. They can game the game, in other words.
Does it improve candidate engagement? CEB modified an inductive reasoning test by applying several game elements – an immersive storyline with 3-D animation, immediate feedback and an interactive, drag-and-drop action.
A comparative study found no differences in test scores before and after gamification, and test takers in highly gamified conditions were not more motivated to complete the assessment. Instead, participants in the most “gamified” conditions showed lower concentration and higher anxiety than candidates in less gamified conditions.
How do candidates respond? Less than 10% of candidates in a CEB survey, said they would prefer a gamified assessment, compared to 50% to 60% of candidates who prefer to complete assessments that are clearly job-related, such as job knowledge tests and work simulations. These figures are consistent across all generations of candidates, including millennials.
Is it valid? This depends on the underlying content. If recruiters take assessments that they know identify good candidates and package that into a more engaging experience, the validity of the assessment is not likely to change just because of different packaging. However, some game elements could undermine validity. For example, introducing competition, such as through public leaderboards that rank candidates, could increase candidates’ anxiety and influence their performance.
Consider also that if an assessment game is similar to other video games in the market, people who play these similar games will likely score higher than those who do not play digital games. Finally, there should be concern about any assessments that allow repeated play. Since we all tend to get better at games after playing them several times, it makes for a poor assessment because the scores will be unstable.
Is the investment worth it? Serious games require serious R&D and investment in a gaming platform – a cost that repeats every time an upgrade is needed. And there is little research that serious games successfully predict job performance.
Adding a few carefully chosen game elements to an existing assessment that has established validity is likely to produce a better return on investment than producing an original serious game for the same purpose.
There is No “One Best” Assessment Tool
Answering these questions will show recruiters that here is no simple, formulaic approach for selecting “one best” assessment. Different types of assessments measure different things, each has advantages and trade-offs, and they are often best used in combination. In short, there are four types.
Traditional assessments: These remain the most thoroughly documented and supported measurement tools in terms of validity, benefit over cost, and minimizing any adverse effects.
Today these tools benefit from technology advances such as computer adaptive testing (the assessment adapts on the fly to users’ responses), mobile device delivery and sophisticated analytic algorithms.
Gamified assessments: These may lead to more engaged and motivated candidates, but more research is needed into the validity of results and the psychological effects of this approach, such potential anxiety and any adverse impact.
Serious games: These raise some serious concerns as a predictor of performance. Unless the game has an overt connection to the role, candidate reactions can be unfavorable.
Costs to develop may outweigh benefits, since serious games have a short shelf life. For example, the US Army’s virtual reality game, “America’s Army,” had 41 upgrades in its first 12 years.
Multimedia simulations of the job environment: These provide more valid results than games, reduce any adverse impact, and are well received by candidates – but they are costly to develop.