Delphi study

A Delphi technique will be used to develop an internationally accepted evaluation framework for public health environmental surveillance. The Delphi method is an iterative multi-round approach that uses a series of sequential surveys, interspersed by controlled feedback, to elicit consensus among a group of individuals while maintaining anonymity (1–4). An electronic Delphi (e-Delphi) method will be used to overcome geographic barriers and allow us to engage panellists internationally across various time zones.

The study protocol is published in PLOS ONE. The preprint is also available on medRxiv.

2

Survey rounds

50+

Target panellists

Key dates for the survey and the rest of the study are in the timeline on the Study design page.

Recruitment of panellists

We will recruit a multinational, multidisciplinary panel of wastewater-based surveillance experts, knowledge users, and engaged members of the public to complete the e-Delphi survey. Panellists will be selected to capture the multiple perspectives of those that influence the design, implementation, evaluation, use, and reporting of wastewater-based surveillance activities, including the following discipline subgroups: public health, infectious disease and epidemiology; environmental and physical sciences; mathematical sciences; social sciences; and communication, knowledge translation and exchange.

We will aim to recruit at least 50 panellists, preferably with at least 8 to 10 people per discipline subgroup. Registration will open about four weeks before Round 1, and recruitment will continue during both rounds. Panellists will complete a short screening questionnaire and then receive an anonymous link to the survey. The study working group will monitor the distribution of registered panellists based on their demographic information and will try to distribute appropriately across discipline subgroups and other demographic markers.

Eligibility criteria for e-Delphi survey panellists

Content experts

Expertise subgroups:

  • Public health, infectious disease and epidemiology
  • Environmental and physical sciences
  • Mathematical sciences
  • Social sciences
  • Communication, knowledge translation and exchange
  • An adult1
  • Proficient in English
  • Has a graduate degree in one of the listed specializations2, or ≥ 3 years of professional experience, or ≥ 2 peer-reviewed publications relating to wastewater-based surveillance

Knowledge users

  • An adult1
  • Proficient in English
  • A professional who does not have specialized training or qualifications in wastewater-based surveillance, but who uses surveillance to inform policy and action in their workplace

Engaged public

  • An adult1
  • Proficient in English
  • Has relevant lived experience

1 Adult: ≥ 18 years of age.
2 The list of specializations associated with each discipline can be found in S2 Table of the published study protocol.

Definitions: Professional experience: paid employment or professional practice in a listed specialization (current or former). Graduate or professional degree: Master or PhD in a listed discipline/specialization.

Procedure

We will conduct a two-round e-Delphi survey to generate consensus on evaluation criteria (see the figure below). Summaries of Round 1 will be compiled for the subsequent round. Custom expertise pathways will be generated for each panellist group and discipline subgroup, so panellists will be shown the candidate items most relevant to their expertise. Within each expertise pathway, panellists will also have the option to skip items or self-declare that they are not qualified to assess certain candidate items.

Round 1

Panellists will rate each candidate item shown to them in their expertise pathway on whether it should be included in an evaluation framework, using a 7-point Likert scale. Candidate items will be generated from scoping review results and consultation with the study executive group. Free-text boxes will be included for panellists to provide feedback or suggest new candidate items for the next round. Items that reach consensus in Round 1 will not be re-rated in Round 2.

Round 1 will be open for about five weeks, take about an hour to complete, and include two email reminders.

Round 2

Regardless of whether they participated in the previous round, panellists will be invited to participate in Round 2 of the e-Delphi survey. All panellists will be invited to rate new items and re-rate previous items that did not reach consensus. Items in Round 2 will be rated across four rating properties (see below). When re-rating their level of agreement, panellists will be presented with their previous round scores alongside the aggregate group results. Anonymous feedback from Round 1 will also be compiled and presented during Round 2. Any newly suggested items during Round 2 will be deliberated on during the consensus meeting.

Defining consensus

In Round 2, candidate items will be assessed on the following four rating properties: (1) relevance and practical utility; (2) scientific rigour, validity and reliability; (3) feasibility, adaptability and resource implications; and (4) equity, inclusiveness, and mitigation of bias.

A 7-point Likert scale (1 = highly irrelevant to 7 = highly relevant) will be used to rate each property. A summary score for each candidate item will be created by calculating the median of the four property ratings. In Round 1, the single inclusion rating is used directly. Panellist ratings will then be categorised as excluded item (irrelevant: 1–2), further discussion (equivocal: 3–5), or core item (relevant: 6–7). Consensus for each item is defined as ≥ 70% of the panellist votes falling within the same category (1–2, 3–5, or 6–7).

e-Delphi consensus process Round 1: initial candidate items are rated in an anonymized e-Delphi survey. With consensus, items are excluded (irrelevant), sent for further discussion at the consensus meeting (equivocal) or kept as core items (relevant); items without consensus go to Round 2. Round 2: new and non-consensus items are rated the same way; equivocal and non-consensus items go to the consensus meeting. Consensus meeting, hybrid or online with discussion and a Delphi survey: with consensus, items are excluded, optional (equivocal relevance) or core; items without consensus are excluded. Round 1 e-Delphi Review of initial candidate items Anonymized e-Delphi survey of candidate items Consensus1 No consensus Excluded item Further discussion Core item Irrelevant Equivocal Relevant Further discussion Round 2 e-Delphi Review of new and non-consensus items Anonymized e-Delphi survey of candidate items Consensus1 No consensus Excluded item Further discussion Core item Irrelevant Equivocal Relevant Further discussion Consensus meeting Hybrid or online with discussion and Delphi survey Consensus No consensus Excluded item Optional item Core item Irrelevant Equivocal relevance Relevant Excluded item
1Round 1 e-Delphi

Reviews

Initial candidate items

Method

Anonymized e-Delphi survey of candidate items

Consensus1 reached

  • Core item relevant
  • Further discussion equivocal → consensus meeting
  • Excluded item irrelevant

No consensus

  • Further discussion → Round 2
2Round 2 e-Delphi

Reviews

New and non-consensus items

Method

Anonymized e-Delphi survey of candidate items

Consensus1 reached

  • Core item relevant
  • Further discussion equivocal → consensus meeting
  • Excluded item irrelevant

No consensus

  • Further discussion → consensus meeting
3Consensus meeting

Reviews

Items for further discussion from Rounds 1 and 2

Method

Hybrid or online with discussion and Delphi survey

Consensus1 reached

  • Core item relevant
  • Optional item equivocal relevance
  • Excluded item irrelevant

No consensus

  • Excluded item

1 Consensus: at least 70% of panellist votes fall within the same category (irrelevant, equivocal, relevant); no consensus if agreement is below 70%.

Item relevance: Core item highly or moderately relevant · Further discussion slightly relevant, neutral or slightly irrelevant · Excluded item moderately or highly irrelevant

e-Delphi consensus process for the Public Health Environmental Surveillance Evaluation Framework (PHES-EF)

Why these thresholds

The approach to use a 7-point fully labelled Likert scale – a higher number of categories compared to many studies – is informed by the consideration that panellists are professionals and engaged members of the public with good cognitive skills; therefore, more categories and labels will be more discriminating and reproducible (5–8). The ≥ 70% consensus cut-off – a lower cut-off compared to many studies – is informed by the rapidly evolving nature of environmental surveillance for public health and the wide range of disciplines invited to the panel; therefore, a high level of agreement may not occur (5–8).

References

1.
Tricco ACT, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation. Annals of Internal Medicine [Internet]. 2018 Oct 2;169(7). Available from: http://dx.doi.org/10.7326/M18-0850
2.
Nasa P, Jain R, Juneja D. Delphi methodology in healthcare research: How to decide its appropriateness. World Journal of Methodology [Internet]. 2021 July 20;11(4). Available from: http://dx.doi.org/10.5662/wjm.v11.i4.116
3.
Shang Z. Use of delphi in health sciences research: A narrative review. Medicine [Internet]. 2023 Feb 17;102(7). Available from: http://dx.doi.org/10.1097/MD.0000000000032829
4.
Niederberger M, Spranger JS. Delphi technique in health sciences: A map. Frontiers in Public Health [Internet]. 2020 Sept 22;8. Available from: http://dx.doi.org/10.3389/fpubh.2020.00457
5.
Diamond IR, Grant RC, Feldman BM, Pencharz PB, Ling SC, Moore AM, et al. Defining consensus: A systematic review recommends methodologic criteria for reporting of delphi studies. Journal of Clinical Epidemiology [Internet]. 2014 Apr;67(4). Available from: http://dx.doi.org/10.1016/j.jclinepi.2013.12.002
6.
Belton I, MacDonald A, Wright G, Hamlin I. Improving the practical application of the delphi method in group-based judgment: A six-step prescription for a well-founded and defensible process. Technological Forecasting and Social Change [Internet]. 2019 Oct;147(N/A). Available from: http://dx.doi.org/10.1016/j.techfore.2019.07.002
7.
Weijters B, Cabooter E, Schillewaert N. The effect of rating scale format on response styles: The number of response categories and response category labels. International Journal of Research in Marketing [Internet]. 2010 Sept 1;27(3). Available from: http://dx.doi.org/10.1016/j.ijresmar.2010.02.004
8.
Lange T, Kopkow C, Lützner J, Günther KP, Gravius S, Scharf HP, et al. Comparison of different rating scales for the use in delphi studies: Different scales lead to different consensus and show different test-retest reliability. BMC Medical Research Methodology [Internet]. 2020 Feb 10;20(1). Available from: http://dx.doi.org/10.1186/s12874-020-0912-8