As well as undertaking research and policy development on financial exclusion and its causes, The Financial Inclusion Centre (FIC) has a strong track record in providing robust, independent evaluation of interventions.
This new research provides an assessment of the feasibility of carrying out a robust financial and social/societal evaluation of local welfare assistance (LWA) schemes that links back to a Theory of Change.
The report can be found here: Developing an Evaluation Proposition for LWAS in London – Final Report July 2021
The research was commissioned by the Greater London Authority (GLA) and London Councils (LC) to feed into the work of the London Recovery Board. FIC undertook a best practice review covering the use of data and evaluation within local welfare support schemes, and assessed current approaches to measuring social impact and cost benefit analyses. This was done via desk research and consultation with councils participating in the Local Government Association’s Reshaping Financial Support programme.
From this research and assessment, FIC has proposed an approach for developing a robust evaluation framework that would allow the impact of LWA schemes to be evaluated.
Most councils now deliver some form of LWA scheme which, despite the differences in name, are broadly similar in terms of the type of support they provide in helping households deal with immediate hardship. Undertaking a robust evaluation of schemes requires the right data, the right evaluation model(s), and a Theory of Change.
There is limited use of robust or systematic data collection amongst Councils to effectively drive delivery or allow for evidence-based targeting of specific cohorts for support. As it stands, data is utilised primarily for the purposes of assessing LWA scheme applications. Evaluation of LWA schemes is very limited.
The research identified only a handful of examples where formal models are used, or systematic measurement of the impacts and cost-benefit generated by the delivery of support undertaken, to maximise value of local support schemes. Notwithstanding the limited use of formal models or systematic approaches, the research did identify examples of data monitoring and a range of metrics within local welfare support schemes that could be utilised to: assess the effectiveness of local welfare schemes; better identify and target the most vulnerable residents ahead of a crisis; better understand clients and identify wider support needs; and contribute to an effective overall evaluation of LWA support provision.
There are a handful of well-developed tools / models that could be utilised to formally measure the impact of LWA schemes in terms of both: social value – with five outcome measures identified as being most relevant that could be adopted to determine the benefit to the household of the local welfare support schemes through changes in: financial comfort, relief from being heavily burdened with debt, higher levels of confidence, feeling in control of life, and relief from depression/anxiety.
Calculating the value for money achieved by LWA interventions using an established cost benefit analysis (CBA) tool allows the ‘financial case’ to be made by quantifying economic benefits that are generated for individuals and organisations.
Eligibility to hardship funding for No Recourse to Public Funds households appears to be limited across the Councils consulted with access having mostly been a temporary change made in response to Covid-19.
Wider research and learning on the behavioural and psychological impacts of poverty is important to inform the evaluation approach to LWA schemes. A new Theory of Change (ToC) for LWA schemes would need to be established that addresses the outcomes and goals the organisations are trying to achieve and the parameters or limitations for the intervention as well as determining the evaluation framework and type and amount of data to be collected to allow evaluation.
The research identifies a number of appropriate data categories that would allow for an evaluation of: operational outputs, targets, and goals; organisational benefits; and policy outcomes and goals.
There appears to be no single evaluation framework that could be lifted directly and used for LWA schemes. An evaluation framework that is fit-for-purpose for LWA schemes could be developed by drawing on elements from the handful of models identified above plus building in elements that are specific and relevant to LWA schemes.
The adapted framework would consist of the identification of: the baseline set of conditions prior to intervention; costs of that intervention; outputs, organisational goals, and policy outcomes that are to be measured to evaluate the intervention; data that is to be used to evaluate the intervention (pre and post intervention, ongoing intervention); sources of that data and who is responsible for collecting and analysing data; methods for quantifying the impact and actual model to be used to evaluate the impact; and limitations of any evaluation framework.
To undertake proper comparative analysis of an LWA scheme it is important to contextualise the environment in which the intervention operates. An LWA scheme operating in a particular local authority area which scores badly on multiple deprivation indicators or is intended to support individuals with multiple challenging issues might find it more difficult to make an impact than a similarly constructed scheme operating in a less deprived area or designed to support individuals with comparatively simple, less difficult issues.