Business decisions: the benefits of using experimental evidence

A growing number of American companies are embracing experimentation in their decision-making processes. This evidence-based approach is the key to understanding the effectiveness of business decisions in terms of cause and effect.

Pedro Rey Biel

In recent months, and especially since the Nobel Prize in Economics was awarded to Richard Thaler, I have seen how this drive to use experimentation in decision-making is gaining momentum all over the world, including in Spain.

Companies, public institutions, NGOs and financial institutions have started to create internal behavioural economics units. Those of us who work in this field have been surprised to receive frequent calls from organisations interested in adopting an experimental approach in their day-to-day business activities.

Experimentation in decision-making is gaining momentum all over the world

This interest in evidence-based decisions is also flourishing in the field of education. New courses on "experimentation in companies" have been cropping up in all sorts of master's degrees, which usually require a final project involving an experimental application.

It used to be that we researchers had to proactively seek out "let's-do-an-experiment" collaborations. Therefore, we are pleased to be asked to run experiments with real application outcomes that benefit organisations' production processes.

Now is a good time to reflect on both the potential and the difficulties of using the experimental method to make evidence-based decisions rather than relying on intuition or copying what has worked for others.

What, exactly, is an experiment? It's is a small-scale test in which the result of a single measure applied to a randomly selected sample is compared with the result from a control group, which has also been selected randomly.

Our experiments are very similar to clinical trials

In order to prove that the variable in question is actually the cause of the observed effect, it must be truly exogenous. The treatment group and the control group must be perfectly comparable – hence the randomisation. In other words, one would expect a priori not to observe any differences between the two groups. The measured variable is only aspect that distinguishes one group from the other, so it must be the cause of the result.

Our experiments are very similar to clinical trials, in which some patients are given a medication while others are given a placebo without their knowledge. The difference is that our experiments involve people, not pills.

Lab experiments
Illustration: Holdentrils/Pixabay

Here's a practical example from the business world. Imagine that your intuition tells you that a certain incentive programme can increase your employees' productivity. Instead of applying the programme to all employees at once – risking possible losses if your intuition is wrong – you can test its potential on a small group of employees. This group should have the same characteristics as the staff as a whole. Ideally, they should also be isolated from members of the control group to prevent interactions that could contaminate the results.

If you observe that this group's productivity increases compared to other employees, you can conclude that the results are scalable and that the incentive programme should be rolled out to other parts of the company.

Similar experiments can be performed in many areas: price changes, special deals, communication policies, donations to charitable campaigns, and even organisational hierarchies. For this approach to succeed, both the experimentalist and the organisation must understand the market very well and be willing to run tests based on informed intuition.

Why are companies hesitant to adopt experimentation?

Their reluctance may be due to a combination of factors. They may be wary of applying a new method. They may be concerned about the short-term loss of benefits during the experiment (even though the cost of an experiment is nothing compared to the losses associated with not taking optimal measures). Or it may be difficult for managers to admit that they don't know the answer to a problem and need to experiment in order to find it.

Two of the greatest experts in the experimentation field, John List (co-author of The why axis) and Steve Levitt (co-author of Freakonomics), have written articles that give practical advice on how to perform useful experiments. These articles, Why economists should conduct field experiments and Field experiments in economics, draw on the authors' academic knowledge to explain how experimentation can be applied to real-life decisions.

Business executives want to learn what works and what doesn't to make quick decisions

These insights show that the objectives of academia and the corporate world are compatible, if not necessarily identical. Academics want to understand exactly why something happens, while business executives want to learn what works and what doesn't to make quick decisions.

Whenever academics and business executives collaborate on an experiment, it is crucial to establish a solid agreement among the parties involved. In most cases, a confidentiality agreement should be reached to establish what results are publishable and under what conditions, with the aim of ensuring that competitors cannot benefit from the knowledge generated.

It is also important to understand that the parties' incentives are multiple: behavioural economists are primarily interested in publishing their findings, but the inability to publish can be offset by access to a novel data source or economic compensation.

By way of example, this article explains how a colleague and I studied ways of encouraging customers of a supermarket chain to answer questionnaires about consumer habits. I consider this to be one of my most interesting academic works, because it enabled us to understand a market and a type of incentive that were poorly understood at the time.

I encourage you to perform an exercise of modesty and honesty: consider making decisions based on evidence from experimental tests rather than what your intuition tells you will work.

Even if you are an expert in your field and your intuition is correct, there are still various degrees of freedom – method, sample size, expected effect quantification – that make experimentation interesting.

Remember: experimentation does not involve changing many things at once. Making multiple changes only serves to obscure the real cause. With patience, you can learn much more by applying the scientific method to truly understand the reasons behind a phenomenon.

Read the original knowledge article in Spanish on Nada es Gratis.

All written content is licensed under a Creative Commons Attribution 4.0 International license.