How to normalise judges' scores: z-scores for strict and lenient jurors

Why strict and lenient jurors distort a ranking

Jurors use a scale differently. One gives 8 out of 10 to any solid application; another rarely goes above 6. Both may put the applications in the same order, but their numbers differ.

When every juror scores every application, this difference affects every application equally and the plain average is fair. When each juror scores only a share, an application's average depends on which jurors it drew. An application that drew lenient jurors beats an equally good one that drew strict jurors. The ranking then rewards luck in the draw.

Normalisation removes each juror's personal level and spread. It keeps what the juror says about an application compared with the other applications they scored.

The z-score formula

For each juror, take the average (m) and the standard deviation (s) of their scores in the round. Each score becomes a z-score: (score − m) / s. A z-score of +1 means one standard deviation above that juror's usual score.

For each application, average the z-scores of the jurors who scored it. Then put the result back on the scale of all scores in the round, using their average (M) and standard deviation (S), so it reads like an ordinary score.

normalised score = M + S × mean of ((score − m) / s)

m, s = average and standard deviation of one juror's scores in the round
M, S = average and standard deviation of all scores in the round
The mean runs over the jurors who scored the application.
A juror with fewer than 3 scores, or the same score for everyone, counts as 0.

This is exactly how Juryline computes the normalised score. It uses the population standard deviation (STDEV.P in a spreadsheet). A juror with fewer than 3 scores or no spread tells nothing about their scale, so their z-scores count as 0 and neither lift nor sink an application.

Worked example: three jurors, five applications

Example: five invented applications, A to E, scored on a 1 to 10 scale. Each application has two jurors. Juror 1 averages 5.00 with a spread of 1.41, Juror 2 averages 9.00 (spread 0.82) and Juror 3 averages 7.00 (spread 0.82). All ten scores together average 6.80 with a spread of 1.99. You can paste these scores into the free jury score calculator to check the result.

E has the highest average only because it drew the lenient Juror 2. Both its jurors gave it exactly their usual score, so its normalised score equals the round average, 6.80. C got the top score from both its jurors: mean z = (1.41 + 1.22) / 2 = 1.32 and 6.80 + 1.99 × 1.32 = 9.43. It ranks first once strictness is taken into account.

When the plain average is fine and when it is not

The plain average is fine when every juror scores every application, as in a small jury or a pitch final. It is also the only option when jurors have too few scores to show their scale.

Normalise when each juror scores only a share of the applications, which is usual with more than a few dozen applications. The more scores each juror gives, the more reliable the adjustment: at least 3 are needed, and with 10 or more the estimate of a juror's strictness is steady.

Normalisation has limits. It assumes that each juror's share is of similar quality on average; a juror who by chance got a weak share will have it lifted. It changes a juror's level and spread, not their order: if two jurors disagree about which application is better, normalisation does not settle that.

Decide in the rules, before scoring starts, which ranking counts. Then show both, so the jury sees where the order changes.

In Juryline, every round shows the average and the normalised ranking side by side. The round setting says which one is official, and changing it after scores are in is recorded and printed in the jury protocol.

Balanced, reproducible assignment

Normalisation works best when every juror scores a similar number of applications drawn at random and when jurors share applications with as many other jurors as possible. Then strict and lenient jurors can be compared across the round.

1. Choose 2 or 3 jurors per application. Three keep a result steady when a juror abstains or declares a conflict.
2. Exclude declared conflicts before the draw.
3. Give each juror an even share: jurors per application × applications ÷ jurors, rounded up. Example: 3 × 60 ÷ 9 = 20 applications each.
4. Mix the panels, so the same jurors do not always score together.
5. Draw with a random seed and record the seed and the inputs. Anyone can then rerun the draw and get the same assignment.

In Juryline, the assignment is a function of its inputs and a recorded seed. Each juror gets an even share (more only when an application has nobody else eligible), applications with the fewest eligible jurors are filled first and declared conflicts are never assigned. Among equally loaded jurors, the one who has shared the fewest applications with the jurors already chosen comes first. The jury protocol prints the algorithm, the seed and a SHA-256 hash of the inputs, so the same input always gives the same result.

How to do it in a spreadsheet

You need one row per score: application, juror and score.

1. For each juror, calculate AVERAGE and STDEV.P of their scores.
2. For each score, calculate z = (score − juror's average) / juror's spread. Use 0 if the juror has fewer than 3 scores or a spread of 0.
3. For each application, average its z-scores.
4. Calculate M and S: AVERAGE and STDEV.P of all scores.
5. Normalised score = M + S × the application's average z-score. Rank by it.

Or paste your table into the free jury score calculator. It does these steps in your browser, shows both rankings and uploads nothing.

Questions

Is normalising scores the same as using z-scores?

Yes. Each score becomes a z-score within its juror's scores. The average z-score is then mapped back to the scale of all scores, so a normalised 7.4 reads like a score out of 10.

How many scores does a juror need?

At least 3 for any adjustment. With 10 or more scores per juror, the estimate of their strictness is steady.

Does normalisation change the order of one juror's scores?

No. It shifts and stretches each juror's scores as a whole, so the juror's own order stays the same. Only the combined ranking can change.

Can I normalise totals of a weighted rubric?

Yes. Normalise each juror's total per application. Juryline normalises the weighted total out of 100 points of every score.

Should we use the normalised ranking if every juror scored everything?

It is not needed. When every juror scores every application, strictness affects all applications equally and the plain average ranks them fairly.