Delphi study
A Delphi technique will be used to develop an internationally accepted evaluation framework for public health environmental surveillance. The Delphi method is an iterative multi-round approach that uses a series of sequential surveys, interspersed by controlled feedback, to elicit consensus among a group of individuals while maintaining anonymity (1–4). An electronic Delphi (e-Delphi) method will be used to overcome geographic barriers and allow us to engage panellists internationally across various time zones.
The study protocol is published in PLOS ONE. The preprint is also available on medRxiv.
Key dates for the survey and the rest of the study are in the timeline on the Study design page.
Recruitment of panellists
We will recruit a multinational, multidisciplinary panel of wastewater-based surveillance experts, knowledge users, and engaged members of the public to complete the e-Delphi survey. Panellists will be selected to capture the multiple perspectives of those that influence the design, implementation, evaluation, use, and reporting of wastewater-based surveillance activities, including the following discipline subgroups: public health, infectious disease and epidemiology; environmental and physical sciences; mathematical sciences; social sciences; and communication, knowledge translation and exchange.
We will aim to recruit at least 50 panellists, preferably with at least 8 to 10 people per discipline subgroup. Registration will open about four weeks before Round 1, and recruitment will continue during both rounds. Panellists will complete a short screening questionnaire and then receive an anonymous link to the survey. The study working group will monitor the distribution of registered panellists based on their demographic information and will try to distribute appropriately across discipline subgroups and other demographic markers.
Eligibility criteria for e-Delphi survey panellists
Content experts
Expertise subgroups:
- Public health, infectious disease and epidemiology
- Environmental and physical sciences
- Mathematical sciences
- Social sciences
- Communication, knowledge translation and exchange
- An adult1
- Proficient in English
- Has a graduate degree in one of the listed specializations2, or ≥ 3 years of professional experience, or ≥ 2 peer-reviewed publications relating to wastewater-based surveillance
Knowledge users
- An adult1
- Proficient in English
- A professional who does not have specialized training or qualifications in wastewater-based surveillance, but who uses surveillance to inform policy and action in their workplace
Engaged public
- An adult1
- Proficient in English
- Has relevant lived experience
1 Adult: ≥ 18 years of age.
2 The list of specializations associated with each discipline can be found in S2 Table of the published study protocol.
Definitions: Professional experience: paid employment or professional practice in a listed specialization (current or former). Graduate or professional degree: Master or PhD in a listed discipline/specialization.
Procedure
We will conduct a two-round e-Delphi survey to generate consensus on evaluation criteria (see the figure below). Summaries of Round 1 will be compiled for the subsequent round. Custom expertise pathways will be generated for each panellist group and discipline subgroup, so panellists will be shown the candidate items most relevant to their expertise. Within each expertise pathway, panellists will also have the option to skip items or self-declare that they are not qualified to assess certain candidate items.
Round 1
Panellists will rate each candidate item shown to them in their expertise pathway on whether it should be included in an evaluation framework, using a 7-point Likert scale. Candidate items will be generated from scoping review results and consultation with the study executive group. Free-text boxes will be included for panellists to provide feedback or suggest new candidate items for the next round. Items that reach consensus in Round 1 will not be re-rated in Round 2.
Round 1 will be open for about five weeks, take about an hour to complete, and include two email reminders.
Round 2
Regardless of whether they participated in the previous round, panellists will be invited to participate in Round 2 of the e-Delphi survey. All panellists will be invited to rate new items and re-rate previous items that did not reach consensus. Items in Round 2 will be rated across four rating properties (see below). When re-rating their level of agreement, panellists will be presented with their previous round scores alongside the aggregate group results. Anonymous feedback from Round 1 will also be compiled and presented during Round 2. Any newly suggested items during Round 2 will be deliberated on during the consensus meeting.
Defining consensus
In Round 2, candidate items will be assessed on the following four rating properties: (1) relevance and practical utility; (2) scientific rigour, validity and reliability; (3) feasibility, adaptability and resource implications; and (4) equity, inclusiveness, and mitigation of bias.
A 7-point Likert scale (1 = highly irrelevant to 7 = highly relevant) will be used to rate each property. A summary score for each candidate item will be created by calculating the median of the four property ratings. In Round 1, the single inclusion rating is used directly. Panellist ratings will then be categorised as excluded item (irrelevant: 1–2), further discussion (equivocal: 3–5), or core item (relevant: 6–7). Consensus for each item is defined as ≥ 70% of the panellist votes falling within the same category (1–2, 3–5, or 6–7).
Reviews
Initial candidate items
Method
Anonymized e-Delphi survey of candidate items
Consensus1 reached
- Core item relevant
- Further discussion equivocal → consensus meeting
- Excluded item irrelevant
No consensus
- Further discussion → Round 2
Reviews
New and non-consensus items
Method
Anonymized e-Delphi survey of candidate items
Consensus1 reached
- Core item relevant
- Further discussion equivocal → consensus meeting
- Excluded item irrelevant
No consensus
- Further discussion → consensus meeting
Reviews
Items for further discussion from Rounds 1 and 2
Method
Hybrid or online with discussion and Delphi survey
Consensus1 reached
- Core item relevant
- Optional item equivocal relevance
- Excluded item irrelevant
No consensus
- Excluded item
1 Consensus: at least 70% of panellist votes fall within the same category (irrelevant, equivocal, relevant); no consensus if agreement is below 70%.
Item relevance: Core item highly or moderately relevant · Further discussion slightly relevant, neutral or slightly irrelevant · Excluded item moderately or highly irrelevant
Why these thresholds
The approach to use a 7-point fully labelled Likert scale – a higher number of categories compared to many studies – is informed by the consideration that panellists are professionals and engaged members of the public with good cognitive skills; therefore, more categories and labels will be more discriminating and reproducible (5–8). The ≥ 70% consensus cut-off – a lower cut-off compared to many studies – is informed by the rapidly evolving nature of environmental surveillance for public health and the wide range of disciplines invited to the panel; therefore, a high level of agreement may not occur (5–8).