How we build our tests: the assessment development process

Stage 1: every test starts with a blueprint

Test development starts long before the first question. Each HireValid test has a written blueprint that answers the questions a careful hiring manager would ask:

• What does it measure, and for which roles? A one-line statement of the skill and the roles it suits, shown in the test library.
• Which sub-skills? Every test is split into named sub-skills with weights, such as reading tables or spotting errors, so an employer sees a breakdown of each result without ever seeing a question.
• How long, and how hard? Most tests run 15 minutes or less. Each question pool has a difficulty mix, for example 30% easy, 50% medium and 20% hard, and the blueprint says how many questions each pool contributes.
• Which aids are allowed? An on-screen calculator or scratch paper, where the work calls for them, named on the candidate’s introduction card.
• Who is it fair for? Each test carries a fairness note that records what was done to keep it fair, and says when it needs a desktop rather than a phone.

The blueprint keeps a test focused on the job. A question that does not serve one of its sub-skills does not belong in it.

Stage 2: questions written to the blueprint

Each question pool has its own specification: the topic, the sub-skill it serves, the question formats to use and the difficulty mix. Questions are drafted to that specification with the help of AI, which lets us build pools far larger than a hand-written test could afford, so every candidate can get a different set.

Writing rules apply to every question:

• One right answer. Every scored question has exactly one defensible answer, and each wrong option is plausible but clearly wrong to someone with the skill.
• Formats that resist AI. Tables, figures, sequences and scenario details are preferred over prose that pastes cleanly into a chatbot, and at least a third of every cognitive test is built on tables or figures.
• Time limits that fit. A competent person can answer comfortably within the limit, which is short enough to make looking it up impractical.
• Fair language. No question depends on culture-specific knowledge, idioms, sports, currency or holidays beyond what the job needs. Names and scenarios are varied, and the reading level suits the role.

A new question that is too similar to one already in the pool is turned away automatically, so pools grow in variety, not in near-copies.

Stage 3: an independent review of every question

Before any question reaches a candidate, it goes through an independent review. The reviewing model comes from a different AI model family from the one that drafted the question, so the two do not share the same blind spots.

The review works in two parts. First, it answers the question blind, exactly as a candidate would, without seeing the key. If it cannot reach the keyed answer, the question is not clear enough and does not go live. Second, it scores the question against a checklist: is the key right and the only defensible answer, are the wrong options plausible, is the difficulty honest, does it test the sub-skill it claims, is it free of cultural assumptions, do tables and pictures carry what the question needs, and does every image have alt text a screen-reader user could answer from?

A question that scores 80 or more goes live. Anything lower is rejected with the reasons recorded, and the next round of writing replaces it. Automatic checks run alongside the review: spreadsheet answers are recalculated by our own formula engine, coding answers are run against their test cases, and copy is checked for US spelling and for claims we never make. When a question is edited later, it becomes a new version and goes through the same review again.

Stage 4: publishing a fixed version

A test is published only when every pool holds at least three times as many approved questions as the test draws from it, and five times for cognitive tests. That is what lets every candidate get a different selection, in a different order, with shuffled answer options. A test also needs its practice set: four easy questions covering every format it uses, shown to candidates before the scored questions begin.

Each published test is a fixed version. When an employer activates a job, the job keeps the version it started with, and every score records the scoring version that produced it. If a test improves later, earlier results keep their original meaning, and you can always explain how a past result was reached. Nothing is ever deleted: a question taken out of service is retired, with its reason, and its history is kept.

Answer keys and pools never leave our servers. Candidates receive one question at a time, and employers see sub-skill scores rather than question text, which keeps the content secure for the next candidate. The anti-cheating overview explains how that design keeps tests AI-proof.

Stage 5: live monitoring and benchmarks from real candidates

Once candidates start taking a test, a daily statistics watch looks at how every question behaves in practice: how often it is answered correctly, and whether strong candidates answer it correctly more often than weaker ones, which is the heart of item analysis. A question that misbehaves is taken out of service for review, and a fresh one takes its place.

The same data builds the test’s benchmark. HireValid’s benchmarks come from its own candidates, separately for each test version. A test shows percentiles once 50 candidates have finished it. Until the sample reaches 200, each percentile is labeled as a provisional benchmark with its sample size; from 200, it is labeled as a HireValid candidates benchmark with its sample size. Benchmarks are recalculated daily and rebuilt as each sample grows, and every result records the benchmark version it was compared with. The benchmarks page explains how to read a percentile.

Test validity and reliability: how we think about both

Reliability asks whether a test measures consistently, and validity asks whether it measures what the job needs. Both start with the blueprint: a test built around the skills a role actually uses, scored on clear sub-skills, with enough questions in each to give a stable picture.

The process above protects both. The independent review removes ambiguous or mis-keyed questions that would add noise. Large pools with controlled difficulty mixes keep different candidates’ tests comparable. Item statistics from real candidates show which questions separate stronger from weaker performance, and retire those that do not. As samples grow, those statistics give each test a growing evidence base, gathered from the people who actually take it.

For the employer, the practical rules are simple: choose tests that match the role, give every candidate the same test and time, offer an accommodation on request and check selection rates after each round. Our fairness approach and the guide to pre-employment testing cover those steps in more detail.

Fairness checks that continue after launch

Fairness is not a one-time check. Every test carries its fairness note, candidates can be given extra time, and practice questions mean nobody meets an unfamiliar format for the first time under the clock.

Employers should also check their own outcomes. After each hiring round, compare selection rates between groups of applicants; the four-fifths rule is a common first check. A difference is a prompt for a person to look more closely at the test, the role and the process, never a verdict on its own.

Nothing in HireValid rejects, hides or ranks a candidate automatically. Scores, benchmarks and integrity evidence inform the employer, who makes every decision.