Systematic reviews are syntheses of the best available evidence on the effects of health care interventions. Cochrane Reviews are high quality reviews that can provide valuable information for clinicians and policy makers, but are poorly suited for decision makers in time-pressed contexts. Condensed, tailored summaries of reviews may help, but there is little research about how to design such summaries, how users will experience them, and what their effect is.
In this set of four studies, users’ experiences of evidence-embedded artifacts were explored and measured. Multiple methods were employed, including a user experience framework from design practice domain. The first study is a set of user tests, observing health professionals’ user experience of the Cochrane Library (where Cochrane systematic reviews are published). The second study involves exploring user and stakeholder feedback to inform iterations of a Summary of Findings Table for Cochrane Reviews. The third study is an evaluation of the effect of including a Summary of Findings Table in a Cochrane Review (compared to a Review with no table) on user satisfaction, understanding and time spent to find key messages. The fourth study explores user and stakeholder feedback to inform the development of a template for short summaries of systematic reviews that are tailored for health policy makers in low and middle-income countries.
Findings from the user feedback uncovered a number of important comprehension problems, such as misunderstanding of document type and risk presentations, as well as confusion caused by unfamiliar language. However, in two small trials we demonstrated that correct comprehension of the main results in a Cochrane Review were improved by including a Summary of Findings Table (in comparison to a review with no table). Key findings were also quicker to find in a review with a table. User studies helped us understand how summaries of evidence from reviews could be made more useful, by layering the information, broadening scope or adding task-relevant details. One of our main challenges regarded resolving the tension between stakeholders’ concern for precision in data presentation and users’ needs for simplicity. Drawing on fuzzy trace theory, this can be reframed as users’ need to easily be able to extract the gist (the meaning to them) of the information; evidence should be designed to enable easy gist extraction rather than exclusively focusing on representing the verbatim information.
Designing for complex (digital) artifacts necessitates a multidisciplinary approach where the design role is highly distributed and the product nearly invisible. This challenges our current view of designers’ professional identity, the way we judge design quality, and what are legitimate outcomes of design work.
This body of work resulted in the evaluation and improvement of specific evidence-embedded artifacts, as well as a revised user experience framework and reflections relevant to the field of knowledge translation and multidisciplinary design work.
Access thesis here.