Work to date

PHES-EF is in its second step: generating the candidate evaluation items that panellists will rate in the e-Delphi survey. Two scoping reviews have mapped how surveillance systems are evaluated across public health, animal health, One Health and wastewater surveillance. The terms they identified have been reconciled into a draft framework, which the executive group is reviewing before the survey opens in 2027. This page summarizes that work and shows what panellists will see.

74

Terms in the dictionary

73

Wastewater sources coded

1,462

Coded annotations

68

Draft candidate items

How the framework was built

  1. 1ReviewTwo scoping reviews: existing evaluation frameworks, and lessons learned from wastewater surveillance during COVID-19.
  2. 2ExtractEvaluation terms, definitions and measurement approaches coded from each source.
  3. 3ReconcileTerms matched across disciplines, recording synonyms and related terms.
  4. 4OrganizeTerms arranged into themes, domains and attributes, then reviewed by the executive group.

Reconciling terms across disciplines

Evaluation terms are not used the same way in every field. The same word can mean different things, and the same idea can go by different names. Each term in the framework records its synonyms (“also known as”) and related terms (“see also”), so readers from any discipline can find it.

Same name, different meanings

“reliability”
Stabilitythe system keeps working when it is needed Consistencydata mean the same thing over time

Different names, same meaning

acceptability engagement participation
Acceptability and engagementThe willingness of individuals and organizations to participate in the surveillance system and the depth of their active involvement (CDC 2001; Calba 2015; Goutard 2022; Hoinville 2013).

Bringing disciplines together

Each surveillance tradition has its own evaluation focus. Public health frameworks emphasize how quickly and accurately a system detects cases; animal health frameworks add costs and laboratory performance; One Health frameworks emphasize governance and collaboration across sectors; and wastewater surveillance adds early warning, coverage and ethics. PHES-EF brings these together into one comprehensive view of public health environmental surveillance.

Anatomy of a Delphi candidate item: Timeliness

Timeliness is one of the best-evidenced items in the draft framework. Here is everything a candidate item records, with a short definition of each field.

Delphi candidate item

Timeliness

Attribute A measurable property or characteristic.

Where it sits in the framework

Theme

Surveillance effectiveness

A high-level grouping of related concepts that define broad areas of public health evaluation.

Domain

System performance

A major area of focus within a theme.

Subdomain

None (not needed here)

A specific aspect or perspective within a domain.

What it means

Definition A concise definition derived from the scoping review literature that captures the essential meaning of the concept.

The time elapsing between any two defined steps in the surveillance process. (CDC 1988; CDC 2001; Groseclose & Buckeridge 2017)

Exact synonym Terms with the same definition as the main schema term.

Reactivity (European public health guidance)

Broad synonym Terms with broadly the same definition as the main schema term.

None recorded

See also Related terms that provide additional context or relevance to the main schema term.

Simplicity; early-warning lead time

Evidence from the scoping reviews

DescriptionAn explanation of the term that builds on the definition.

Timeliness measures how much time passes as a health event moves through the surveillance system toward public health action. (Azofeifa 2018; CDC 1988) How much timeliness a system needs depends on the health event: rapid reporting is critical for acute, highly transmissible diseases, whereas less urgent conditions allow slower reporting. (Declich 1994; CDC 2001) It is influenced by how the surveillance system is designed, the resources available, and the volume and complexity of the data. (Groseclose & Buckeridge 2017; Declich 1994; Sosin 2003) Delays in laboratory results or data flow can hinder a timely response, particularly in resource-limited settings. (Innes 2022; Sahal 2009) Poor timeliness can also reduce the usefulness of the data and the acceptability of the surveillance system. (Sahal 2009; ECDC 2014; Buehler 2004)

BackgroundThe historical development, theoretical foundations and evidence base that establish the concept’s importance in surveillance system evaluation.

Timeliness has been part of surveillance evaluation since the earliest public health guidelines, which described it as “the speed or delay between steps in a surveillance system” and set the interval between a health event and its reporting, the identification of trends or outbreaks, and the implementation of control measures. (CDC 1988; Thacker 1988) The updated guidelines kept this framing, noting that “the time interval linking any two of these steps can be examined”, and judged the attribute by the availability of information for control. (CDC 2001) Later guidance anchored the interval as early as exposure to the agent and carried it through to the start of control measures. (Buehler 2004; Health Canada 2004; WHO 2006) Standardized terminology for animal health surveillance then settled on a common formulation: “the time between any two defined steps in a surveillance system”. (Hoinville 2013; Drewe 2015)

In public health surveillance, timeliness is treated as a core attribute, measured as the interval between defined steps and interpreted according to the purpose the system serves. (Groseclose 2010; Groseclose & Buckeridge 2017; Sosin 2003) Animal health frameworks use the same definition, applying it through both quantitative and semi-quantitative assessment methods. (Hoinville 2013; Muellner 2018; Peyre 2019) One Health evaluations adopt the same formulation while noting practical constraints, notably the laboratory delays that lengthen the measured interval. (Innes 2022; Peyre 2022) Wastewater-based surveillance recasts it as “the ability of the WS to deliver timely results and to provide an early warning signal compared to other surveillance systems”, gauged by the lead time between environmental and clinical indicators. (Amato 2023) The underlying definition is consistent across disciplines; differences lie mainly in which steps each setting measures, though European public health guidance also labels the concept reactivity. (Groseclose & Buckeridge 2017; Peyre 2019; ECDC 2014)

Evaluation methods (10)The methods or tools used to quantify an attribute.
  • Interval measurement between defined steps: measuring the time elapsing between any two defined steps in the surveillance process, with the steps chosen to be epidemiologically meaningful for the system’s purpose (CDC 1988; CDC 2001; Hoinville 2013; Drewe 2015).
  • Median reporting delay: calculating the median number of days between a reference date, such as disease onset or diagnosis, and the date of report to the receiving agency (Jajosky 2004).
  • Benchmarking against the disease time course: expressing the measured interval as the percentage of cases reported within one or more incubation periods for the health event (Jajosky 2004; Buehler 2004).
  • Proportion reported within a target window: the share of reports or outbreak notifications submitted within a predefined schedule, such as 24 or 48 hours (WHO 2006; ECDC 2014).
  • Interim-milestone timeline: mapping fixed milestones along the surveillance chain to locate and compare the intervals that drive overall timeliness (Buehler 2004; Crawley 2021).
  • Alert lead-time comparison: comparing the time at which a statistical alert is generated against detection by an established surveillance component (Buehler 2008; Groseclose 2013).
  • Normalized timeliness score: modelling the measured interval with an exponential decay function to obtain a score between 0 and 1 (Arinik 2023).
  • Trend analysis of median timeliness: applying Cox proportional hazards regression to assess changes in median timeliness over time (Crawley 2021).
  • Time-to-detection against a reference indicator: estimating the lead time between an environmental or surveillance signal and a clinical or gold-standard indicator (Amato 2023; Arinik 2023).
  • Qualitative key-informant assessment: interviewing system operators to judge whether the achieved intervals meet the surveillance purpose (Harper 2011; Groseclose 2010).

What the survey will look like

In Round 1, panellists see the items for their discipline subgroup, grouped by domain, and rate whether each one should be included in the framework. Try the example below.

Example only. Nothing you select here is saved or sent.

Surveillance effectiveness · System performance

Timeliness

The time elapsing between any two defined steps in the surveillance process.

How relevant is this attribute for inclusion in the evaluation framework?

Choose a rating to see how your answer is recorded.

In Round 2, items without consensus are rated again on four properties: relevance and practical utility; scientific rigour; feasibility; and equity. See the e-Delphi study page for how consensus is defined.

Draft framework structure

Draft, subject to change The draft framework has 7 themes, 21 domains and 68 candidate items. Themes organize the domains for this overview and are not rated by panellists.

System structure and processes

4 domains · 14 candidate items

Surveillance effectiveness

7 domains · 22 candidate items

Partnerships and community engagement

4 domains · 14 candidate items

System capacity and resilience

2 domains · 10 candidate items

Financial performance and resources

2 domains · 3 candidate items

Overall performance

1 domain · 4 candidate items

System description

1 domain · 1 candidate item