New Study: A Sociotechnical Framework to Assess Patient-Facing eHealth Tools

Among the thousands of eHealth tools available, the vast majority do not get past pilot phases because they cannot prove value, and only a few have been systematically assessed. Although there are various initiatives working on finding ways to assess the quality of eHealth tools, there is limited information and methods describing how to realistically assess and evaluate these tools in practice. Many of the existing initiatives have not been validated with the relevant stakeholders, and/or are conceptual without granular guidance on how to use and apply them in day-to-day decision making.

A study led by a research team of the FHNW School of Business "A sociotechnical framework to assess patient-facing eHealth tools: results of a modified Delphi process", published in Nature Digital Medicine, aimed to address some of these challenges by validating and refining an initial list of 55 assessment criteria based on previous frameworks through a two-round modified Delphi process with in-between rounds of interviews.

The project team is currently working on developing a web-based assessment instrument based on the outcomes of this study, where the assessors can find step by step guidance on how to assess each of the evaluation measures, input their assessment of each criterion and download a report comprising the assessment outcomes and dashboards. This work aims to empower decision-makers with an assessment instrument tailored to support their decision-making based on their specific needs and priorities within the distinct contexts where they are evaluating a tool.

About the Study

The international panel included 57 experts from 18 countries and nine concerned parties. A consensus was reached on 46 criteria that were classified into foundational and contextual criteria. The 36 foundational criteria focus on evaluating the eHealth tool itself and were grouped into nine clusters: technical aspects, clinical utility and safety, usability and human centricity, functionality, content, data management, endorsement, maintenance, and developer. The ten contextual criteria focus on evaluating the factors that vary depending on the context the tool is being evaluated for and were grouped into seven clusters: data-protection compliance, safety regulatory compliance, interoperability and data integration, cultural requirements, affordability, cost-benefit, and implementability.

Source: Jacob et al. A sociotechnical framework to assess patient-facing eHealth tools: results of a modified Delphi process. npj Digit. Med. 6, 232 (2023).

The classification of criteria into foundational and contextual helps us assess not only the quality of an isolated tool, but also its potential fit in a specific setting. Criteria subscales may be particularly relevant when determining the strengths and weaknesses of the tool being evaluated. This granularity enables different concerned parties to make informed decisions about which tools to consider according to their specific needs and priorities.

Further Information

This innovation project is sponsored by Innosuisse (the Swiss innovation agency), with the FHNW University of Applied Sciences and Arts Northwestern Switzerland as the research partner and Roche and KPT Insurance as the practice partners. For more details and resources, please check the project page.