3. The evaluation process

3.1. Assessors and content

The evaluation process is based on existing Nordic and Finnish processes. They will be applied to the process where applicable. Our objective has been to establish an evaluation process that is as transparent and as light touch as possible. The core working group comprises the Early Intervention science editor and an expert panel, referred to as the Scientific Panel.

The role of the Scientific Panel is to offer guidance and support to the editor in developing evaluation criteria for early support interventions. The Scientific Panel is responsible for carrying out an overall evaluation and making final decisions on classification based on proposals put forward by the editor. The council membership comprises national experts from the fields of psychology, psychiatry and education: Kristian Wahlbeck, Chair SMS; Eeva Aronen, University of Helsinki; Ilse Julkunen, University of Helsinki; Mirjam Kalland, University of Helsinki; Jorma Komulainen, Duodecim; Marjukka Mäkelä, National Institute for Health and Welfare; Raija-Leena Punamäki, University of Tampere; Nina Sajaniemi, University of Helsinki; Christina Salmivalli University of Turku; Päivi Santalahti, National Institute for Health and Welfare; Marja Holmila, National Institute for Health and Welfare).

3.2. The evaluation process

Collecting data on interventions

We start the evaluation process by collecting information on the interventions. This information will be used to evaluate the interventions in terms of the scientific evidence available and their effectiveness and usefulness. The documentation may be presented in a number of different formats but, importantly, it should also be available and accessible to practitioners using the intervention. The information might include a manual and relevant scientific reports and other publications. We will also seek information from a number of relevant databases. Our aim is to complete an evaluation that is as accurate, appropriate and objective as possible. Under the Finnish model, the science editor is responsible for collecting the information.

The evaluators

Under the Finnish model, the evaluations are carried out by an expert panel, known as the Scientific Panel, whose members represent a broad range of disciplines relevant to child and family support services. The evaluation process is based on objectivity and impartiality. This means that people who have been involved in developing, disseminating or providing training on the intervention under review or who are associated with a rival intervention are not eligible to contribute to the evaluation process. However, it is desirable for evaluators to have knowledge and experience of the relevant interventions. In addition to theoretical expertise, at least one of the evaluators should have significant prior experience of psychometrics.

Evaluation meeting

The Scientific Panel meets twice a year. The expert panel conducts the evaluation process on the basis of summaries of the interventions prepared by the science editor. These are issued with a proposed classification. In drawing up the summaries, the science editor works with the developers and/or practitioners using the intervention along with one member of the Scientific Panel.

The summary comprises a written summary and a draft classification. This is based on an assessment of the quality of the evidence available, along with its effectiveness and usefulness. The Scientific Panel members work together as a group on the basis of the written materials provided and with view to achieving a consensus decision on each evaluation. This is done to ensure that all members give the same weighting to the criteria on evidence, effectiveness and usefulness. All panel members are committed to carrying out their work with objectivity and impartiality. At the end of the evaluation process, the panel members draw up a list of each intervention’s strengths and weaknesses, with a particular focus on practical aspects, before assigning them to a classification category. In addition, the expert panel can put forward suggestions for improvements.

Evaluation follow-up

Once the expert panel has assessed the materials, Early Intervention will complete a draft summary. The draft summary, along with the proposed classification, is sent to the individuals or organisations representing the intervention, which allows them the opportunity to correct any errors that may have occurred during the evaluation process. The Early Intervention editorial team will then finalise the summary and classification.

Creating a classification

The purpose of the evaluation and classification system is to offer support to practitioners and guide their decision-making. The classifications are recommendations drawn up by experts on the basis of scientific evidence that are intended to provide information on how useful and effective the interventions are in the practical or clinical setting. The aim is to ensure that practitioners are able to identify clearly targeted and highly effective interventions that will be chosen in consultation with the client, ensuring that their voice is heard and that they are fully informed of the options available. The classifications system is perhaps best described as a practical, evidence-based toolkit aimed at professional practitioners.