Policy Evaluation Network

Background and aim

The Joint Programming Initiative on a Healthy Diet for a Healthy Life (JPI HDHL), involved 28 research institutes from seven European countries and New Zealand combined their expertise to form the Policy Evaluation Network (PEN). PEN, true to its vision, has improved knowledge, tools and capacity among its network and beyond to identify, evaluate and benchmark policies designed to directly or indirectly address physical inactivity, unhealthy diets and sedentary behaviour while accounting for existing health inequities.  

Through a series of consortium-wide activities, network meetings, presentations at (inter)national conferences and capacity building workshops with policy makers PEN has met, and indeed exceeded, its deliverables. PEN has published the first EU Food Environment Policy Index (Food-EPI) for Europe, and the first Physical Activity Environment Policy Index (PA-EPI).  PEN has developed a set of Selected Instruments for Multilevel PoLicy and impact Evaluation (SIMPLE) modules for physical activity and diet, and created an expert-led Methodologic Competence Platform to guide and sustain the data harmonization process across Europe long term.  PEN scientists have created simulation models, applied multiple research designs and used innovative online research methods –during COVID-19 restrictions– to deliver 50+ peer-reviewed publications. These include meta-, systematic- and narrative-reviews advancing, synthesizing and consolidating state-of-the-art evidence on policy determinants, frameworks, tools and research methods.  PEN has advanced our understanding of the unintended effects of policy interventions on persons with low income, and as part of its Early Career Network provided mentorship and prompted changes within JPI programmes to support early career professionals.  The Policy Symposium on NCDs prevention in Brussels (June 2022), organised by PEN, the H2020 projects CO-CREATE and STOP, and the Joint Action Best-ReMaP, culminated in a Call To Action outlining key policy, research and funding priorities necessary to prevent obesity.  In summary, PEN has provided a comprehensive roadmap for policy improvement for health promotion and prevention of non-communicable diseases, including obesity. 

Expected impact

WP1 Policy mapping and EPI development

The current health status of people living in Europe, lifestyle factors and inequalities in health outcomes must be urgently addressed through the food and physical activity environment policy reform. Policy action is needed by government to create healthier food and physical activity environments. It is now clearly understood that to prevent chronic disease, we need to understand and address the “foodscape” the wide range of interconnected factors such as food production, processing, marketing and distribution, that characterise our food system and largely determine our dietary intakes and the daily determinants and our physical activity ’scape’, factors which determine our physical activity levels including (but not limited to) physical environment, access to facilities transport policies and urban planning policies. WP1 has provided a better understanding within both the scientific and the policy-making communities of existing policies that directly or indirectly affect physical activity (PA) and food environments and behaviours.

The implementation of the Food Environment Policy Index (Food-EPI) in 5-European countries (11 if you include the collaboration with the STOP partners) and at the EU has increased the understanding of what good benchmarking indicators are for the food policy environment. The Food-EPI evidence documents and accompanying reports in Norway, Ireland, The Netherlands, Poland, Germany and the EU received considerable attention from policy makers and media circles. These discussions extended to discussions with policy makers at national level on the key findings of the reports. In Ireland, the recent report of the Commission on Taxation and Welfare cites the Food-EPI (Ireland) Evidence Document and accompanying report as important documents providing evidence of support for policies to improve the food environments.

During the development of the Food-EPI evidenced reports and the launch and publication of the Food-EPI score cards we have highlighted the gaps and priority recommendations to create healthy food environments across European countries. Within each country each team has engaged with key stakeholders and policy makers, in particular in the Ministries of Health, to highlight the findings from the Food-EPI work.

The outputs from task one and the mode of engagement with key stakeholders, in particular policy makers has the potential to influence real change in the policy making environment.

Capacity building and collaboration is to the fore in Task 1.1. In collaboration with STOP partners, the 11-country comparison paper, a first in Europe, highlights the priority gaps in food policies to improve the food environment s in Europe. This has been accepted for publication in the Lancet Europe. Further, as a result of PEN, Task 1.1 partners were invited to contribute to two Horizon Europe applications based on building healthy and sustainable food environments. One of these applications has been successful.

The development of the PA-EPI prototype and the implementation of PA-EPI in Ireland (with more countries to follow) provide for the first time an opportunity to analyse Government action to improve the physical activity environment across the all domains which influence citizens physical activity and sedentary behaviours. The PA-EPI is conceptualised as a two-component ‘policy’ and ‘infrastructure support’ framework. The two components comprise eight policy and seven infrastructure support domains. The policy domains are education, transport, urban design, healthcare, public education (including mass media), sport-for-all, workplaces and community. The infrastructure support domains are leadership, governance, monitoring and intelligence, funding and resources, platforms for interaction, workforce development, and health-in-all-policies. Forty-five ‘good practice statements’ (GPS) or indicators of ideal good practice within each domain concludes the PA-EPI. A potential eight-step process for conducting the PA-EPI is described. Once pre-tested and piloted in several countries of various sizes and income levels, the PA-EPI GPS will evolve into benchmarks established by governments at the forefront of creating and implementing policies to address inactivity.

Task 1.2, building on the Food-EPI framework will develop the first PA Environment Policy Index (PA-EPI). Following on from the two publications on 2021, a further four publications are currently under review (two of which are part of the PEN Special Issue).

This has resulted in two further publications in 2021

• Woods CB, et al. The evidence for the impact of policy on physical activity outcomes within the school setting: A systematic review. J Sport Health Sci. 2021 Jan 19: S2095-2546(21)00006-5. doi: 10.1016/j.jshs.2021.01.006. Epub ahead of print. PMID: 33482424;
• Volf K, et al. Policy Evaluation Network (PEN): Protocol for systematic literature review examining the evidence for impact of policies across seven different policy domains [version 2; peer review: 1 approved]. HRB Open Res 2020, 3:62 (https://doi.org/10.12688/hrbopenres.13089.2)

A systematic mixed studies review examining the evidence of the impact of sport policies on PA and sport participation is currently under review with the Int J Sport Policy Politics. The evidence identified seven unique ‘policy actions’ which were categorised into the ‘policy areas’ Facilities, Financial, Collaboration and Exhortation. Policy actions to promote PA and sport participation have demonstrated qualified success but there is limited evidence of success in engaging hard to reach groups. Therefore, policymakers utilising sport to increase PA should treat it as a complementary intervention alongside other policy actions based on a systems perspective.

The impact of mass-media campaigns on PA: a review of reviews through a policy lens, is currently under review as part of the PEN Special Issue. Results indicate that the most consistent evidence was found for the effectiveness of mass-media campaigns on proximal outcomes, while the evidence for distal outcomes was mixed. Good practice statements were derived: 1) to achieve behaviour change, mass-media is an important component of larger, multilevel, and multicomponent strategies, 2) mass-media strategies should be coordinated and aligned at local- and national-level, and be sustained, monitored and resourced at these levels, 3) media should be tailored to reduce socioeconomic inequalities. Findings suggest that mass-media can play an important role in the promotion of PA. In general, evidence was more inconsistent for effectiveness on distal outcomes than for proximal outcomes. The policy-relevant recommendations identified will serve to inform the PA environment policy index (PA-EPI), a tool for monitoring, evaluating and benchmarking government progress in implementing public policies.

A Systematic Review examining which transport policies influence PA of the whole of society is currently under review with the J Transport Health. Three main transport policy areas with 51 individual policy actions were identified that had a direct or indirect effect on PA. These were: convenient transport infrastructure development, active travel promotion, and shift of transport mode. More than half of the policy actions identified had a positive effect on PA. Results suggest that PA levels can be increased by implementing policies that provide convenient, safe, and connected walking and cycling infrastructures, promote active travel and give strong support to public transport. There is also clear evidence that active travel policies work best when implemented in a comprehensive way. This may include infrastructure and facility improvements as well as educational programmes to achieve substantial shifts towards active modes of travel.

Combining the outputs of Task 1.1 and 1.2 will provide a comprehensive description of the food and PA policy landscape in five European countries.

WP2 Monitoring and surveillance

With its work, WP2 successfully provides a sustainable scientific contribution to the harmonization of health indicators in Europe and connects the growing scientific interest in investigating interconnections between data – harmonised across several countries – and policies on different levels (e.g., supranational, national, communal) to improve individual health behaviour and inform policies to prevent obesity. This mainly concerns the evaluation and development of survey instruments and methods according to the latest research findings as well as the needs and current status of method development of each of the surveillance systems. In order to achieve this, prioritized indicators and their availability in existing European datasets were made usable for the scientific community and various European initiatives that use and/or collect and make available monitoring data themselves, e.g. the European Food Safety Authority (EFSA), Organisation for Economic Co-operation and Development (OECD), the European Commission (DG SANTE and EUROSTAT).

Furthermore, SIMPLE modules were developed which collate unified questionnaire items to harmonise collected data and assess comparable health-related data across European surveillance systems that will facilitate the evaluation of temporal and regional changes in physical activity, dietary behaviour, and associated health outcomes (individual level).

Finally, a close exchange has already been established with leading representatives of ongoing major European surveillance initiatives such as WHO-COSI, HBSC, STEPS and Nordic Monitoring System. A Methodological Competence Platform was established fostering the cooperation between WHO Collaborating Centres and counselling the WHO in health monitoring and methodological advancement of the WHO-COSI methodology.

The implementation of the developed SIMPLE modules and their further methodological development will help researchers to analyse the relationship between diet, physical activity and health across European countries and to identify factors affecting health-related and environmentally-sustainable behaviours. This will also help to inform policy makers about the effectiveness of health policies and thus can contribute to the aim that all citizens will have the motivation, ability and opportunity to consume a healthy diet from a variety of foods, and have healthy levels of physical activity.

WP3 Estimation and simulation of policy impact

WP3 is expected to have impacted and to generate a continued impact on the scientific community and policy audience over the next years. The WP has provided tools for better evaluation, hence better policies. It has provided a better understanding of the quantitative approaches to policy evaluation, especially in relation to the assessment of fiscal policies (soda tax), with a thorough comparison of experimental methods, ex-ante simulations and ex-post quasi-experimental methods.

The impact on the targeted audience and the interest in the project findings has clearly emerged from the final WP3 workshop in Rimini, where the findings of the project have been presented. The workshop gathered 82 registered participants from 21 different countries: Argentina, Belgium, Estonia, Finland, France, Germany, Hungary, Ireland, Italia, Lithuania, Luxembourg, Mexico, Netherlands, Norway, Portugal, Romania, Slovenia, Spain, United Kingdom, United States. Among represented public institutions there were the European Commission, the Estonian National Institute for Health Development, the Hungarian National School Sport Federation, the Belgian Federal Public Health Ministry, the French Agency for Food, Environmental and Occupational Health & Safety. the Finnish Institute for Health and Welfare, the Irish Food Safety Authority, the Lithuanian Ministry of Health, the Romanian National Institute of Public Health, the Portuguese Directorate-General of Health, the Mexican Instituto Nacional de Salud Pública, and the World Cancer Research Fund International. The aim of the international workshop was to discuss challenges, differences and synergies of different approach for policy impact quantification: simulation models, quasi-experimental methods, and virtual and field experiments. Speakers included Franco Sassi (Head of Director of the Centre for Health Economics and Policy Innovation at Imperial College London, and co-ordinator of the STOP project), Pierre Dubois (Toulouse School of Economics and Institute of Fiscal Studies), David Frisvold (University of Iowa), Federico Perali (University of Verona), Timothy Beatty (University of California, Davis) and Annalisa Belloni (World Health Organization and Public Health England).

The forthcoming publication of an article containing methodological recommendations (Emmert-Fees et al., 2022), envisaging and integrated evaluation strategy combining experimental, quasi-experimental and simulation methods has the potential to foster further research in this area, and to provide tools that may address unresolved issues in impact evaluation studies, such as the assessment of longer-term health impacts, the consideration of heterogeneous effects, and the evaluation of multi-level policies. The paper and the main conclusions were presented at the final multi-project symposium in Brussels, along with evidence on experimental methods.

WP4 Policy implementation evaluation

WP4 systematically described and evaluated policy implementation processes. A compilation of existing theoretical frameworks and key aspects as well as barriers/facilitators of policy implementation processes will help the scientific community to better understand what core aspects should be taken into account when (1) translating research into practice, (2) specifying influences on implementation outcomes and (3) and evaluating implementation success. The identified frameworks for implementation can be used by researchers, development actors, policy makers, NGOs, practitioners and other stakeholder of the public health area who aim to analyse and guide the implementation of a health policy in their setting.

Based on the results of all three sub-tasks of this work package and several case studies conducted in WP6, researchers and policy makers were provided with practical guidelines for policy implementation evaluation and a checklist for policy implementation in form of two application-oriented final reports of WP4. In addition, two manuscripts have been prepared for the scientific community that will be submitted in Sept. and Oct. 2022.

WP5 Equity and diversity of policies

WP5 aimed at achieving a strong emphasis in policies on those in lower socioeconomic groups and in ethnic minorities. Substantial improvements of population levels of PA, diet and sedentary behaviour can be made by prioritizing policies with an impact in these groups. Yet, there is a large variety of interventions and policies with relatively little effects. A structured and guided approach is needed, in which expertise about (mechanisms of) health inequalities is further incorporated in policy making. The impact of WP5 is that researchers and policy makers are increasingly aware of, and able to assess the implications of policies for those in lower socioeconomic groups and ethnic minorities in all stages of the policy cycle (including monitoring, surveillance, benchmarking, evaluation and implementation of policies). Our work might support the scientific and policy community to focus on actions beneficial for the above-mentioned target groups. We learned that the ‘systems approach’ is a promising approach in which both scientists, policy makers and stakeholders are involved. It leads to new entry points for interventions and policies and might prevent undesired effects of policies. Dissemination of the approach used and the findings will impact on the choice of interventions and policies. In order to make impact however, and to go beyond the statement that the problem is complex, methods are needed and applied to derive policy recommendations from these systems models. The approach provides way to consider not only actions targeting the elements (as done for many public health challenges), but also the mechanisms, structure and paradigm of the system. Results of our work are and will be disseminated in scientific papers, and in a symposium at the end of the project.

WP6 Policy in practice - Selected case studies

The results of the SSB-tax case study (Task 6.1) have provided (and still provide) direct input into the discussion about the implementation of a sugar tax that is currently going on in several European countries, for instance in the Netherlands. The case study for Poland coincides with the actual implementation of a sugar tax in this country. The study may provide a very early evaluation of the implemented tax, as well as may have impact on the public perception of the health-related taxation policy, being currently subject to controversies and distrust.

The systematic review on effects of Sustainable Urban Mobility Plans on physical activity (Task 6.2) helped to determine the level and type of evidence for policies within the transport area that promote physical activity. The interviews about SUMP adoption in different European cities have great potential to inform research and policy practice alike. From a scientific perspective, evaluation tools are currently lacking that assess SUMP implementation by probing the impact of various stakeholders. As SUMP take-up requires the interplay of several actors in diverse domains such as politics/administration, civil society and business, accounting for their underlying – and contrasting – rationales is pivotal. This in-depth analysis can boost policy evaluation. In a similar vein, the rigorous examination also allows for specific policy recommendations. By uncovering drivers of and barriers to SUMP adoption, we provide a concise yet specific overview of potential levers for implementing sustainable mobility regimes. Lastly, our approach covers multiple cities with differing progress in SUMP take-up. Consequently, we present a diverse picture applicable to a myriad of urban contexts. This outcome can encourage transport-related physical activity and contribute to positive health impacts. Regarding effect evaluations, we only identified data on motorisation rate, modal split, and public transport use to broadly compare the influence of SUMP implementation on PA across cities. This demonstrates the potential for developing additional uniform indicators to assess indirect changes in PA e.g., obesity rates, transport education, bicycle and pedestrian network, and PT infrastructure. Additionally, quantitative data on the effectiveness of transport-related intervention inducing a shift in the mode of travel from car to transport modes enhancing physical activity provided by meta-analysis may be of great importance when developing Sustainable Urban Mobility Plans.

The results of the literature review on evaluations and implementation of School Fruit schemes (Task 6.3) contribute to the field of implementation science in particular through the use of Consolidated Framework for Implementation Research (CFIR) for the synthesis of existing research in regard to the implementation of fruit and vegetable interventions in schools. The paper highlights the importance of the following CFIR constructs, as determinants in the implementation of fruit and vegetable interventions in schools: ‘design quality and packaging’, ‘adaptability’, ‘cost’ (intervention characteristics domain), ‘cosmopolitanism’, ‘external policy and incentives’ (outer setting), ‘implementation climate’, ‘readiness for implementation’ (inner setting), ‘knowledge and beliefs about the intervention’ (characteristics of individuals domain) and finally, ‘engaging’(process domain). The study of the implementation of the EU SFVS in selected countries found that flexibility in how the scheme is designed and implemented enables country-level implementation. Newly established cooperation between implementing ministries is a potential facilitator. However, timing of the yearly top-down budget allocation is a barrier. Taking EU funding for granted is a potential disincentive to improvement, although the EU funding facilitates sustainability. Despite agreement on what the overall goals of the scheme are, there is some ambiguity as to what the primary goal is, which may influence design (duration, frequency) as well as implementation at country and school level. The qualitative system dynamics map of impact of EU SFVS is likely to provide useful information for strengthening the implementation and reach of the scheme, as well as input for ongoing evaluation of the scheme in 2023. The results have been presented to the EU-level authorities both at the Brussel conference of the PEN project and in an online meeting for all countries taking part in the EU SFVS.

WP7 Network coordination and dissemination

In terms of disseminating the results, PEN has and will continue disseminating the results, reaching all targeted audiences, among others through following activities:

The special journal issue titled “Creating evidence for public policies to improve dietary and physical activity behaviours in Europe” is to be published by the Eur J Public Health over autumn 2022. It includes 13 original articles presenting final results of all PEN work packages, including a paper on the development of the overarching theoretical PEN framework. It was important for the consortium to allow access to all interested groups by publishing this supplement Open Access.

As one of the concrete results of the symposium jointly organised with the projects STOP and CO-CREATE and JA Best-ReMaP, a document was compiled summarising the main messages and key achievements across the projects and recommendations of next steps for policy makers, stakeholders and funding programmes (see annex). This document is currently being broadly disseminated by the 4 projects and is available at the PEN website (https://www.jpi-pen.eu/images/reports/Policy_Symposium_Final_Statement_Oct2022.pdf). The collaboration with STOP and CO-CREATE and JA Best-ReMaP will continue to be explored for identifying opportunities for joint dissemination of results.

The PEN factsheets summarise the main results, conclusions and recommendations of the research areas covered by each PEN work package. This brochure will be disseminated to researchers, policy, and practice stakeholders, e.g., national funding bodies, public health decision makers at the national and European levels, public health promotion organisations at national and European levels. The PEN factsheets (16) are publicly available at the project website and have been and will continue being distributed as hand-outs at international conferences whenever the opportunity is given.