Study design

PHES-EF is being developed using an established, internationally recognized consensus methodology. The study follows its published protocol and will be reported using the ACCORD guideline for consensus studies, with PRISMA-ScR for the scoping reviews and GRIPP2 for public involvement. This approach ensures:

  • Evidence-based content through two systematic scoping reviews
  • International consensus via a two-round e-Delphi survey and a consensus meeting
  • Diverse perspectives from content experts, knowledge users and engaged members of the public
  • Transparent development with open sharing of materials and progress on the Open Science Framework
  • Broad applicability across public health environmental surveillance systems, using wastewater-based surveillance as an example

The framework will be developed in five steps.

Study steps

1

Establish the executive group

  • Agree on the scope of the study and develop the study protocol
  • Agree on terms of reference for executive group members

Completed

2

Generate candidate evaluation items

  • Conduct scoping reviews to extract candidate items for the e-Delphi survey (searches are being updated)
  • Review candidate items with the executive group for organization, completeness and redundancy

In progress

3

Conduct the international e-Delphi survey

  • Recruit, screen and register international panellists (registration will open in January 2027)
  • Run two online survey rounds in spring 2027

Upcoming

4

Hold the consensus meeting

  • Hold a hybrid or virtual consensus meeting in September or October 2027
  • Reach expert consensus on the final list of items and agree on dissemination strategies

Upcoming

5

Disseminate the evaluation framework

  • Post a preprint to OSF, then submit to an open-access peer-reviewed journal (late 2027), under a CC BY-SA 4.0 licence
  • Engage researchers and knowledge users, and support adoption of the framework

Upcoming

Step 1: Establishment of the executive group

A multinational and multidisciplinary executive group oversees the development of the evaluation framework. The group brings expertise from various fields, including public health, infectious disease and epidemiology; environmental and physical sciences; mathematical sciences; social sciences; communication, knowledge translation and exchange; knowledge users; and an engaged public.

The executive group is responsible for providing feedback on the framework development process (e.g., framework organization, completeness, additions, redundancy), supporting and advising working groups in their area of expertise, connecting the study to other related projects, recruiting e-Delphi panellists, participating in the consensus meeting, disseminating the evaluation framework, advising and/or participating in post-publication activities, and promoting the adoption of the evaluation framework among their professional networks. Co-chairs are also responsible for decision-making when the executive group cannot reach a consensus, leading executive group activities, and the overall framework development process.

For more information about the members of the study’s executive group, please visit the Team page.

Step 2: Generation of candidate items

We are systematically compiling a list of e-Delphi candidate items through two scoping reviews, content analysis and extraction, and executive group consultation. The scoping review searches are being updated. See Work to date for progress so far.

You can view relevant scoping review registrations and publications on our Documents page.

Step 3: International e-Delphi survey

PHES-EF will use an e-Delphi survey, an online version of the Delphi method, in which panellists rate candidate items over two anonymous rounds with feedback between them (1,2). Running the survey online allows panellists to take part from anywhere, across time zones.

Key dates are in the timeline below. For details on recruitment, eligibility, rating and consensus, see the Delphi study page.

Step 4: Consensus meeting

In preparation for the consensus meeting, the study working group will categorize the remaining candidate items for consideration into a preliminary evaluation framework. This will include merging and/or modifying the remaining items based on gathered feedback from the e-Delphi rounds.

A hybrid or virtual consensus meeting will be held in September or October 2027, using the Surveylet Delphi survey platform, after the results of the e-Delphi have been compiled and analysed. The primary objective of the meeting will be to achieve expert consensus on the final list of items for PHES-EF, through review and discussion of salient items. This process will be guided by the empirical evidence that was identified during the scoping reviews, and the opinions gathered during the e-Delphi process. The secondary objective will be to discuss publication and dissemination strategies for the final evaluation framework.

Steps to produce the final list of items are as follows: (i) present the results of the e-Delphi exercise (name, rationale, and score of each item); (ii) discuss the rationale and relevance for including the items in the framework; and (iii) vote on equivocal and non-consensual items. PHES-EF will be developed based on the final list of items that receive consensus during the executive group consensus meeting.

Step 5: Dissemination of evaluation framework

The development of the evaluation framework will be reported in a statement document that will include the rationale and a brief description of the meeting, and the panellists involved. While the publication strategy will be finalized based on the consensus meeting discussion, the preliminary approach will be to post to OSF, as a preprint, before submission to an open-access peer-reviewed journal in late 2027. During this time, we will actively seek feedback from experts, knowledge users, and the public within our networks and via social media. PHES-EF will be made available under an open licence (CC BY-SA 4.0).

Timeline

Fall 2026

Preparing the survey Now

Finalizing the list of candidate items, then building and piloting the e-Delphi survey.

January 2027

Panellist registration

Panellists will complete a short screening questionnaire. Recruitment will continue during both rounds.

March 2027

Round 1

Panellists will rate each candidate item for inclusion and suggest new items. Round 1 will be open for about five weeks and take about an hour.

April to May 2027

Round 2

Panellists will rate new items and re-rate items without consensus across four rating properties, with group results from Round 1.

Summer 2027

Analysis

Results will be analysed and a preliminary framework will be prepared.

Fall 2027

Consensus meeting

A hybrid or virtual meeting will be held to agree on the final list of items and the dissemination plan.

Late 2027

Framework and manuscript

A preprint will be posted on OSF, then submitted to an open-access journal under a CC BY-SA 4.0 licence.

Preparation e-Delphi survey Consensus Dissemination

Dates are targets and may change.

References

1.
Tricco ACT, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation. Annals of Internal Medicine [Internet]. 2018 Oct 2;169(7). Available from: http://dx.doi.org/10.7326/M18-0850
2.
Nasa P, Jain R, Juneja D. Delphi methodology in healthcare research: How to decide its appropriateness. World Journal of Methodology [Internet]. 2021 July 20;11(4). Available from: http://dx.doi.org/10.5662/wjm.v11.i4.116