Reporting

Data Quality

Data quality refers to our measure of how reliable the data is.

We use reliability bands which are based on two pieces of data:

• The 95% confidence interval width
• The raw contributor count for that measure

The definitions are as follows:

• Insufficient: rawCount ≤ 5
• Highly Reliable: ciWidth ≤ 10
• Moderately Reliable: 10 < ciWidth ≤ 20
• Less Reliable: 20 < ciWidth ≤ 40
• Insufficient: ciWidth > 40

The ciWidth is in percentage points of n.

Concretely: if 45 of n = 200 respondents are in a category, p = 22.5%. The CI might be, say, 17%–28%, giving a ciWidth of ~11 percentage points → “Moderate”. As n grows, the interval tightens and ciWidth shrinks, which is what makes it a proxy for data quality.

In this example: 95% of the time we run this survey we’d expect the result to be between 34 and 56 (ie 0.17 * 200 = 34 and 0.28 * 200 = 56 )

Why does data quality increase near 0% and 100%?

The reliability band is chosen from the width of the confidence interval — the raw count (not the percentage) is only used for the Insufficient rule above. For the same amount of data, the confidence interval is widest for results near 50% and narrowest for results near 0% or 100%, so we are more confident about very small and very large proportions than about middling ones. This is why a category measured at 2% can earn a Highly Reliable badge while a 45% category from the same survey is only Moderate.

The intuition behind the math: when a proportion is near 0% or 100%, almost every observation falls into the same category, so the sample is very consistent — there is little variability to account for, and the result is pinned down tightly. Near 50/50 the outcome is maximally uncertain (like a fair coin), so different samples of the same population bounce around the most, and the band has to be wider to stay 95% confident. Put another way: near 50%, chance plays a big part in the exact result; near the ends, the result reflects something structural about the population rather than the luck of who happened to be sampled — so the same amount of data buys more certainty.

Two examples:

• Political polling. Survey 100 people. If 50 say they will vote for Party A (50%), the 95% interval runs from roughly 40% to 60% — an actual result of 65% would be at the very edge of what a small poll can miss by, surprising but not shocking. If 10 of those same 100 people say they will vote for Party Z (10%), the 95% interval is only about 6% to 17% — so Party Z actually winning 25% of the vote would be astonishing, even though it is the very same 15-point miss. Same poll, same amount of data, much tighter certainty at the low end.
• Under-18 producers. This category usually reports something like “1% or less, 95% certain the true value is between 0% and 8% — highly reliable.” It is highly reliable because there are structural reasons there are very few under-18 producers: nearly every observation lands in “no”, so the interval is narrow. (The raw-count rule still applies: a category holding 5 or fewer actual respondents displays as Insufficient regardless of how narrow its interval is.)

N (population) and n (sample)

On the charts we show N (the population — the total role count) and n (the sample — the respondent role count). Both count roles, not people, and both are exact counts of the underlying data — but n is published with differential-privacy noise added, while N is published exactly.

How each figure is defined, what is eligible to be counted, and the rules applied at every stage of the pipeline are documented on N and n: population and sample.

Benchmarks

The three different benchmarks are calculated like this:

• Population: reads from a preset figure in our population concordance config.
• Prior Year: takes 12 months data from the year ending with the “from” date in the selected report. Filters to display only projects accessible to the logged in user that are approved and delivered.
• Industry: takes 12 months data from the year ending with the “from” date in the selected report. Displays data for ALL projects that are approved and delivered.

Confidence Interval Calculation

We use hypergeometric score interval (finite-population Wilson analogue) to calculate confidence intervals.

This is for two reasons:

• Wald confidence intervals are inaccurate for small samples sizes.
• Wilson confidence intervals do not take into account “census” style sampling where we are confident we have surveyed the whole population. It doesn’t use “N” (population size) at all, so if you have 100/100 you get a confidence interval of 3.7%

We apply a correction to Wilson confidence intervals to correct for finite population sizes (ie, N). The correction factor is (N - n) / (N - 1), so a genuine census where the sample equals the population (e.g. n = 100, N = 100, 100 “true” values) drives the interval to a width of 0 — which is what we expect, since surveying everyone leaves no sampling uncertainty.

However, on the charts we only apply this correction when N is a trustworthy finite population: it must exceed n by a safety margin of at least 10 roles or 5% of n, whichever is larger (see when N is not shown on a chart). An N at or near n is treated as a data artifact rather than a real census and the correction is skipped, because clamping N up to n would collapse the factor to 0 and produce a zero-width interval that the data-quality bands would then label as highly reliable — falsely presenting noisy data as certain. In that case we fall back to an effectively infinite population, giving an honest, wide interval. The simulator below applies the raw correction directly, so it will show the zero-width census result when you set N = n.

Confidence Interval Simulator

Experiment with different inputs and methods below: