Reporting

Data Quality

Data quality refers to our measure of how reliable the data is.

We use reliability bands which are based on two pieces of data:

• The 95% confidence interval width
• The raw contributor count for that measure

The definitions are as follows:

• Insufficient: rawCount ≤ 5
• Highly Reliable: ciWidth ≤ 10
• Moderately Reliable: 10 < ciWidth ≤ 20
• Less Reliable: 20 < ciWidth ≤ 40
• Insufficient: ciWidth > 40

The ciWidth is in percentage points of n.

Concretely: if 45 of n = 200 respondents are in a category, p = 22.5%. The CI might be, say, 17%–28%, giving a ciWidth of ~11 percentage points → “Moderate”. As n grows, the interval tightens and ciWidth shrinks, which is what makes it a proxy for data quality.

In this example: 95% of the time we run this survey we’d expect the result to be between 34 and 56 (ie 0.17 * 200 = 34 and 0.28 * 200 = 56 )

Why does data quality increase near 0% and 100%?

The reliability band is chosen from the width of the confidence interval — the raw count (not the percentage) is only used for the Insufficient rule above. For the same amount of data, the confidence interval is widest for results near 50% and narrowest for results near 0% or 100%, so we are more confident about very small and very large proportions than about middling ones. This is why a category measured at 2% can earn a Highly Reliable badge while a 45% category from the same survey is only Moderate.

The intuition behind the math: when a proportion is near 0% or 100%, almost every observation falls into the same category, so the sample is very consistent — there is little variability to account for, and the result is pinned down tightly. Near 50/50 the outcome is maximally uncertain (like a fair coin), so different samples of the same population bounce around the most, and the band has to be wider to stay 95% confident. Put another way: near 50%, chance plays a big part in the exact result; near the ends, the result reflects something structural about the population rather than the luck of who happened to be sampled — so the same amount of data buys more certainty.

Two examples:

• Political polling. Survey 100 people. If 50 say they will vote for Party A (50%), the 95% interval runs from roughly 40% to 60% — an actual result of 65% would be at the very edge of what a small poll can miss by, surprising but not shocking. If 10 of those same 100 people say they will vote for Party Z (10%), the 95% interval is only about 6% to 17% — so Party Z actually winning 25% of the vote would be astonishing, even though it is the very same 15-point miss. Same poll, same amount of data, much tighter certainty at the low end.
• Under-18 producers. This category usually reports something like “1% or less, 95% certain the true value is between 0% and 8% — highly reliable.” It is highly reliable because there are structural reasons there are very few under-18 producers: nearly every observation lands in “no”, so the interval is narrow. (The raw-count rule still applies: a category holding 5 or fewer actual respondents displays as Insufficient regardless of how narrow its interval is.)

N (population) and n (sample)

On the charts we show N (the population — the total role count) and n (the sample — the respondent role count). Both count roles, not people, and both are exact counts of the underlying data — but n is published with differential-privacy noise added, while N is published exactly.

How each figure is defined, what is eligible to be counted, and the rules applied at every stage of the pipeline are documented on N and n: population and sample.

Benchmarks

The three different benchmarks are calculated like this:

• Population: reads from a preset figure in our population concordance config.
• Prior Year: takes 12 months data from the year ending with the “from” date in the selected report. Filters to display only projects accessible to the logged in user that are approved and delivered.
• Industry: takes 12 months data from the year ending with the “from” date in the selected report. Displays data for ALL projects that are approved and delivered.

Confidence Interval Calculation

We use a hypergeometric score interval (finite-population Wilson analogue) to calculate confidence intervals, extended with an extra variance term for the differential-privacy noise carried by the published counts (see below).

The hypergeometric score method is used for two reasons:

• Wald confidence intervals are inaccurate for small samples sizes.
• Wilson confidence intervals do not take into account “census” style sampling where we are confident we have surveyed the whole population. It doesn’t use “N” (population size) at all, so if you have 100/100 you get a confidence interval of 3.7%

We apply a correction to Wilson confidence intervals to correct for finite population sizes (ie, N). The correction factor is (N - n) / (N - 1), so a genuine census where the sample equals the population (e.g. n = 100, N = 100, 100 “true” values) drives the interval to a width of 0 — which is what we expect, since surveying everyone leaves no sampling uncertainty.

However, on the charts we only apply this correction when N is a trustworthy finite population: it must exceed n by a safety margin of at least 10 roles or 5% of n, whichever is larger (see when N is not shown on a chart). An N at or near n is treated as a data artifact rather than a real census and the correction is skipped, because clamping N up to n would collapse the factor to 0 and produce a zero-width interval that the data-quality bands would then label as highly reliable — falsely presenting noisy data as certain. In that case we fall back to an effectively infinite population, giving an honest, wide interval. The simulator below applies the raw correction directly, so it will show the zero-width census result when you set N = n.

Accounting for differential-privacy noise

The counts feeding these intervals are not exact observations. Every count we publish has a small amount of random noise added to it for differential privacy, so the number on a chart sits within some distance of the true count — and the scale of that distance (the noise variance, σ²) is known and recorded. Treating a noised count as if it were exact understates the uncertainty, badly so for sparse categories where the noise can be a large share of the count. The confidence interval therefore accounts for both sources of uncertainty: sampling error and DP noise (TEP-687).

Where the noise variances come from

The Difference Engine records the noise variance alongside every count it publishes, reading σ² directly from the DP framework’s own noise information rather than recomputing it. Each output document carries:

• privacy_budget.noise_variance — σ² of the noise on every count cell in that document
• data_quality.respondent_role_count_variance — σ² of the noise on the respondent count (n) stored beside it

The API sums these across the documents it aggregates and exposes them on qualityMetrics:

• countCellVariance — σ² of the noise on every leaf cell of counts
• respondentCountVariance — the variance of the noise on respondentCount (n), resolved to match how n was actually produced: for single-select questions n is the sum of the count cells, so Var(n) = countCellVariance × (number of cells); for multi-select questions n comes from a separate distinct-role query and carries that query’s own variance

Both fields are nullable, and the null is meaningful: null means unknown, and the client then falls back to the legacy sampling-error-only interval rather than pretending the counts are exact. null must never be read as 0 — 0.0 is a different, positive claim (“these counts are exact, no noise was added”, as produced by non-DP validation runs).

How the noise enters the interval

The score equation the interval is solved from gains a noise-variance term in its denominator. Without noise we solve for the p where:

(x − n·p)² = z² · n·p·(1−p)·f

With noise the right-hand side becomes:

(x − n·p)² = z² · [ n·p·(1−p)·f  +  Var(x) − 2·p·Cov(x,n) + p²·Var(n) ]

where f is the finite-population factor above and the added bracket is the variance of the DP noise on x − n·p:

• Var(x) = m·σ² — when a displayed category is a fold of m raw count cells (grouped answer buckets), each cell contributes its own σ².
• Cov(x, n) — the correlation term matters. For single-select questions, n is the sum of the very cells x is drawn from, so their noise is perfectly correlated and Cov(x, n) = Var(x). For multi-select questions n comes from an independent query, so the covariance is 0. Dropping this term is not a harmless simplification: with many cells it can understate the interval by more than 2× at mid-range proportions — exactly the “noisy data badged reliable” failure this work removes.
• If the API’s respondent count is unusable and the client falls back to using the sum of the counts as the denominator, the denominator’s noise is re-derived to match (Var(n) = K·σ² over the K summed cells, fully correlated with x), since the originally reported respondentCountVariance describes a denominator that is no longer in use.

When the variances are known and zero, the added term is a literal 0 and the result is identical to the pure sampling interval.

What this changes on the charts

• Sparse categories get honestly wider intervals. The reliability bands are chosen from the interval width, so a bar that is mostly noise can no longer be badged “Highly Reliable” — the noise width keeps it in a lower band.
• A census no longer collapses to zero width. Surveying everyone removes sampling uncertainty, but the published counts still carry their noise, so with DP noise present the interval keeps the noise width even when n = N.
• Older data degrades gracefully. Counts generated before the variance was captured report null variances, and the client falls back to the sampling-error-only interval — the same behaviour as before this change.

Why publishing σ² does not leak anything

The noise variance is a function of public configuration only — the privacy budget and the query’s sensitivity, both fixed by the analysis configuration and the taxonomy before any data is read. It is not a measurement of anyone’s data, so publishing it reveals nothing and spends no privacy budget. The detailed argument (and the condition it depends on) is documented in the Difference Engine’s Privacy documentation.

Confidence Interval Simulator

Experiment with different inputs and methods below. Note that the simulator implements the sampling-error interval only — it does not include the DP-noise term described above: