Why is this needed?
Any service should benefit from quality monitoring and evaluation against its identified outcomes.
Monitoring and evaluation are important ways of identifying the need for change and improvement.
As noted in the introduction, adoption support services are in the ideal place to identify potential improvements in the wider adoption and looked after systems, which may reduce the need for adoption support services in the future. It is therefore important not only that evaluation takes place but that the learning from this is used to improve the wider system (see section 17)
15.6 Adoption agencies seek feedback from service users on the success of the service provision. This feedback is recorded centrally and on the case record of the service user.
25.6 The executive side of the local authority, the voluntary adoption agency’s/Adoption Support Agency’s provider/trustees, board members or management committee members:
- receive written reports on the management, outcomes and financial state of the agency every 6 months;
- monitor the management and outcomes of the services in order to satisfy themselves that the agency is effective and is achieving good outcomes for children and/or service users.
Adopters not only want to experience improvements in service delivery but are often very willing and able to evaluate the services and systems which they receive for the benefit of others.
Family Futures has an annual evaluation as well as end of treatment evaluation process for parents, children and professionals (see appendix nine). Along with this, cases are reviewed on an annual or six-monthly basis depending on their level of support. In this way adopters feedback important information into the service about the support they are offered. As part of our assessment process children, parents and school setting complete a batch of psychometric evaluations which are then repeated annually during the treatment process to evaluate progress.
The Family Futures Neuro-Physiological Psychotherapy model (NPP) is one of the few therapeutic interventions to have been evaluated, peer reviewed and published. See McCulloch et al (2016), Vaughan et al (2016) and McCulloch and Mathura (2019).
OAWY had developed a Practice Improvement Framework across all of their services (available on request) which incorporates:
The views of services users (children and families, adoptive parents, birth parents, adopted adults)
Practice wisdom and knowledge of staff, adoption panels & learning from disruptions
The findings of external and internal inspections, audits and evaluations of practice. This includes professional audit activity using case files and direct observation with practitioners to assess the quality of practice
The framework has three questions at its heart:
How much did we do?
How well did we do it?
Is anyone any better off?
It connects to decision making bodies in the five participating local authorities and Health and Well-being Boards. Reporting takes place on a quarterly, six monthly and annual basis to varying levels of detail.
Adoption Counts has developed a Quality Improvement Framework but with a more specific focus on practice and file auditing, including audit tools and schedules for adoption support assessments, support to adopted adults, letterbox activity and the number of birth parents accessing independent support services. Monitoring is conducted using a performance management framework linking outcomes, qualitative and quantitative measures and feedback from services users including children, adopters and birth parents. In addition, Adoption Counts has developed an outcomes monitoring framework (See appendix nine) which links the outcomes and aims of their Centre of Excellence Grant with performance indicators, allocating responsibility for monitoring and measurement between delivery partners.
There is significant interest, not least from the DfE, in gathering evidence about which specific adoption support interventions are the most effective with adoptive families and children. As the Tavistock (DfE 2016b) study demonstrates, even a comprehensive literature review has struggled to draw firm conclusions in this regard.
Adopt East and Birmingham CT use a range of psychometric tools. These include:
Assessment Checklists for Children and Adolescents (Tarren-Sweeney, 2012) which include: ACC (age 4-11) ACA (age 12-17) and ACC and ACA plus (strengths checklist)
Strengths and Difficulties Questionnaire (SDQ) – This is widely used by CAMHS but widely acknowledged as a poor indicator of change
Thinking about your child questionnaire (also called Carer Questionnaire)
Brief parental self-efficacy scale
Adoption Plus suggest that there should be a focus on capturing change in the relationship between the child and the parent and intends to carry out a validation exercise on the ‘Thinking about your child questionnaire’.
Given the time required to implement, record and analyse this kind of data, the Centres of Excellence have focused on a ‘goals-based outcome’ approach. This allows parents and children to identify their own goals and measure progress towards them. It has the advantage of being applicable to almost any intervention and can be used to measure progress in services delivered by external providers. The experience of the Centres of Excellence has been that the most important determinant of ‘success’ is the perception of adopters and young people. Whilst ‘objective’ or psychometric measures have a role to play, they often locate the ‘problem’ with the child, rather than the environment in which the child is functioning. Frequently progress is made when the adults (or the institution) changes its perception of the child, even if the child’s progress is not measurable or even deteriorates. This approach prioritises the perceptions of parents and children (“if they feel it is making a difference, it is making a difference”).
Both Centres of Excellence have followed the lead of Birmingham CT by including two common goal-based outcomes to all assessments and evaluations. These are:
How do you rate the stability of your family unit at the moment?
How do you rate the stability of your child’s school placement at the moment?
Not only are these the two most common areas of concern, but they also allow an element of comparison between different approaches to offering support. Both Centres of Excellence are undertaking evaluations. OAWY have engaged the University of East Anglia. Adoption Counts are using a research assistant supported by the University of Manchester.
Adoption Counts is in the later stages of compiling an evaluation of their Centre of Excellence which will be made available once complete.
Both OAWY and Adoption Counts have conducted adopter surveys. There seems little reason why a standardised survey should not be conducted by all adoption support agencies on a national basis, perhaps as part of an approach to inspection.
The development of national performance indicators (as with recruitment, assessment and family finding activity) in relation to adoption support should be a key priority in driving service development.