Why governments should run experiments to manage public funds better

Listen to this podcast on Spotify | Apple Podcasts | Google Podcasts

Public funds should never be spent on a hunch. Yet more often than not, politicians in power base part of their strategic decisions on subjective opinions or intuitions.

In this podcast episode, Toni Roldán, Director of the Esade Centre for Economic Policy and Political Economy, and Pedro Rey, Associate Professor of Behavioural Economics, discuss why governments should run experiments to make evidence-based decisions. The experts advocate for replacing hunches with evidence-based data to improve public policies and handle public money more effectively.

TRANSCRIPT

Toni Roldán: My name is Toni Roldán, I'm the new Director of the Centre for Economic Policy and Political Economy. I'm here with Pedro Rey Biel, an amazing behavioural economist who also works at Esade. We're going to talk about policy evaluation, behavioural economics and why it is so important. Over the past two decades there has been a big transformation in economics, both in the methods used and the way we approach data and experiments. This year’s Nobel Prize was actually about this.

Pedro Rey: The Nobel Prize was for poverty-reduction policies based on previously-run experiments that provide an idea of what is – and what isn't – going to work. 

Toni Roldán: You're alluding to a very important element, which is trying to identify and be more precise about the causalities. In the past, we used to have lots of analysis, but you couldn't isolate the cause of things. Why are these experiments so important and how have they transformed economics?

Pedro Rey: The idea comes from the fact that too many policies are based on either intuition by a politician who thinks “let’s try to see what works,” or, on the other hand, different ideas about what may or may not work. What this approach is trying to do is to move away from ideologies and intuition and try to benefit from the fact that much more data are available now that everything is digital.

Too many policies are based on intuition by a politician who thinks 'let’s try to see what works'

Today, we have much more data ex-ante and exposed. This allows us to use pilot experiments to try out whether or not a policy is going to work, quantify when it works and what factors make it work. Having so much more data allows us to evaluate the impact, not only in experiments, but also in policies that have already been implemented. The availability of data added to the fact that our analysis tools are much more powerful nowadays allow us to be much more precise and to identify why exactly something works. Finding the causality is also very important. 

Toni Roldán: In my experience as a politician, when we worked on projects and we tried to analyse policies, we found out that more often than not, nobody was actually evaluating anything. We were just proposing policies, spending lots of money and then we didn't even know whether there was an impact. The assumption was that because something grew, it worked, but we didn't actually know what the driver of that growth was. I think there's a cultural transformation, particularly in Spain, that still needs to take place. In several other countries they are already running these pilot experiments and there has been a big change – they are using more modern techniques to evaluate public policies and make better use of public resources. But, in Spain, we are somehow still very far away from that.

Pedro Rey: I don't want to blame you as a former politician, but what you’re saying is a little scary and sad because you're revealing first-hand experience of how you were trying to do things that you thought would work. You obviously were not doing things blindly – you were trying to get access to good data – but part of the problem with analysing exposed policies is that data were not generated under optimal conditions.

In order to establish causality, sometimes phenomena occur and there are correlations between things that are going on, but even if things go on in the same direction at the same time, one thing may not be causing the other.

Having so much more data allows us to evaluate the impact, not only in experiments, but also in policies that have already been implemented

Toni Roldán: Can you give us an example? In education, for instance, let's say you have 200 schools with a big dropout rate problem. You decide to run an experiment and apply a treatment, which might be to add an extra teacher in the classroom to reduce these rates.

Pedro Rey: You mentioned the key word, which is treatment. In an experiment, you first need to have the conditions under which a policy was being run. When you’re trying something new, you should keep everything as similar as possible but have a treatment group which is exposed to the thing you have some intuition or theory about in order to be able to predict whether or not it’s going to work.

Going back to your example about schools, imagine we want to analyse the use of incentives in the classroom, either for teachers or for encouraging students to get better grades or even just to attend school. These things may or may not work because they may create different effects. For instance, perhaps giving incentives for teachers to ensure that students get better grades may not actually work because the professors could be corrupt and award better grades just to get the incentives.

In experiments, what we try to do is to isolate one factor instead of mixing things up. Sometimes, when a policy is implemented, many factors change at the same time. So, you observe an effect, but you don't know which of the many aspects you have changed is causing the effect. What we try to do when we run an experiment is to isolate things and make one change at a time. 

Sometimes, when a policy is implemented, many factors change at the same time

But this doesn't mean that we are only testing one thing: we might have a control group, where everything is kept the same and, at the same time, many experimental treatment groups. In those experiments, only one thing is changed at a time. This allows you to evaluate and measure the effect of the only thing that you're changing, and that gives you causality.

Toni Roldán: Following your example, you would increase the number of teachers in the classroom, randomise the choice of schools and then apply the treatment by isolating one factor. This would allow you to compare the schools that didn't receive the experiment with those that did in order to see what, if any, effect there was on dropout rates. In Spain, though, when you’re trying to choose a school for your kids, you don’t have access to this type of data.

Pedro Rey: It's very private; that's another problem. Many policies are trying to protect people’s privacy, which is obviously a big concern but it is also important to have more data available to researchers. Of course, such data should be anonymised so that no one can misuse it.  

Toni Roldán: You just have an anonymous code, without names, to ensure complete privacy protection. In many countries, they are using anonymous data.

Pedro Rey: In Denmark, for example, the availability of public data is amazing. The social security data of everyone who has ever worked in Denmark is made publicly available to all researchers who satisfy certain conditions. And, of course, they are very careful in terms of anonymity and confidentiality. 

You have to be careful with the data but, if you set up the correct protocols, it allows you to do many things. For instance, the case you were mentioning about your kids going to a school with fewer students per teacher is interesting because intuition tells us that it would be better to have private tutoring.

The cost of not implementing the correct policy is huge, but it's never measured

But this year's Nobel Prize winners Esther Duflo, Abhijit Banerjee and Michael Kremer conducted experiments which showed that the key to increasing attendance and how much kids learn in school is not resources – i.e. the number of teachers per student – but things you wouldn’t have thought of, such as the curriculum or teaching students things that are useful to them in whatever setting and country they live in. This has a much better effect than changing the ratio of teachers to students.

The reasons why a school may be doing well may be related to factors such as location, household income and many other aspects. By keeping those factors constant (choosing schools with similar location, similar income, etc.) we could then isolate the one thing that we want to examine to see whether or not it has an effect. In this way, we can establish the effect of a specific action and not the effect of many things changing at the same time.

Toni Roldán: Spain has one of the highest unemployment rates in Europe. The country spends around €6 billion per year on policies aimed at offering opportunities to the unemployed. Money is allocated to some courses that are run by trade unions, but we don’t know whether these are useful in offering opportunities to workers. In fact, the results show that when we have a bad evaluation, the employment rate for those courses is extremely low. By using these experimental techniques, you could find out which of these courses might work and which skills you need to focus on.

Pedro Rey: Exactly. Imagine that you are designing two different courses and you want to see which one is most effective in helping people to find a job. You would have two very similar populations and you would offer a different course to each of them. Then you would follow these people to evaluate whether they liked the course. This is often termed as policy evaluation, but the important thing is whether these people obtained employment. So, you could trace them through their job-market experience to see if they fond work. This is actually a huge problem in active labour market policies because decisions are often based on things like how long a subsidy should be for. These decisions are very often based on whether you’re on one side of the ideology spectrum or the other.

Toni Roldán: You're pointing to a common problem. One of the reasons why policy evaluation experiments are not widespread in our countries is probably down to their cost. Is there a way to evaluate impact with less expense? 

Pedro Rey: Actually, the cost of not implementing the correct policy is huge, but it's never measured. There's always a cost associated with running an experiment but it can be minimised by doing well-defined pilot tests that are based on the correct sample population.  

The same thing happens when you run pilot projects in firms. Many business managers are reluctant to experiment, but we should change the discussion. First, it's not so costly if you design it properly. Second, the cost of not making the optimal decision – as we have been doing for years with many policies – is so much higher, it's ridiculous. 

One of the things we are advocating for, and the reason why I think you and I are talking right now, is that we are pushing for an approach that is already happening in the UK, the European Commission and the US. They are all moving towards doing experiments and measuring outcomes to test whether things are working or not.

Toni Roldán: As part of the new centre we're opening in Madrid, we are launching a unit – the Policy Impact Lab. One of its goals will be to design and improve public policy using the techniques that we are discussing. Our ambition is to start changing the culture in Spain. We are heading towards a time when the country will have very high debt and very high public spending on, for instance, the elderly. It’s very important that we pay serious attention to how we are spending public money. This is a crucial change that we need to make. As director of this new unit, can you tell us about your main ideas?

Pedro Rey: I'm really excited that, after so many years of lobbying for this, we are creating this unit at Esade. The Policy Impact Lab will evaluate existing policies and help in designing new ones through experiments. This will be a scientific, ideology-free unit that provides data availability, designs policies in a scientific manner, runs experiments and measures outcomes, which we can then offer to different administrations. 

When this movement progresses and starts to extend through different administrations, we will be able to run additional experiments to compare the different policies that different governments have implemented in order to see what works and what doesn't.

Toni Roldán: So, moving beyond left-wing and right-wing debates and towards evidence-based policy-making, which is the crucial transformation we need to make.

Pedro Rey: I mean, it’s fine if we keep our own ideologies because that's part of our personality. 

Toni Roldán: Of course.

Pedro Rey: But when we are working with so much public money and dealing with so many important topics, let's put the ideology aside and concentrate on what works and what doesn't. And let's measure it. That's what we are trying to do. 

Toni Roldán: Fantastic. Thanks very much, Pedro. And thanks very much everyone for listening.

All written content is licensed under a Creative Commons Attribution 4.0 International license.