IQ is just the threshold: Measuring leadership effectiveness requires a variety of viewpoints

... And, reasons why development tests should include not only feedback from professional colleagues but also from personal relations.

This interview is based on research by Joan M. Batista-Foguet

Effective leaders have a sense of self-awareness, a desire to learn and grow and develop resonant relationships that inspire others. They create a sense of well-being and trust around them and activate people's potential by focusing on the latter's strengths, dreams and unique capabilities.

In his research findings published in Frontiers in Psychology, Joan Manuel Batista-Foguet (Director of Esade's Leadership Development Research Centre) and his colleagues shed light on why leadership assessment has to go beyond IQ tests and why development questionnaires to measure leadership effectiveness should include feedback from different observers and contexts.

Leadership assessment has to go beyond IQ tests

Do Better: How useful are IQ tests to measure leadership effectiveness?

Joan M. Batista: During the First World War and, specially, after the Second World War, psychometrics exploded, and tests became a very popular means to scientifically measure human abilities. People were qualified as more or less intelligent based solely on how capable they were of answering tests. For instance, a fireman would be judged as capable of putting out a fire just by his test score. There is plenty of academic work that argues against this narrow test approach (Stephen Jay Gould's is among the most entertaining).

IQ test
Photo: Ben Mullins/Unsplash

Tests can do a lot of damage...

There was a point in time in which people believed that intelligence was the exact score on an IQ test. Some have seen their careers fail because of tests, and I'm afraid we're still doing the same nowadays.

For example, to access graduate management programmes, students are required to take the Graduate Management Admission Test (GMAT) which is supposed to measure a set of verbal and logical competencies required to complete an MSc or MBA programme at any university.

But, did you know that you can improve your score each time you take the test? The second time is better than the first, and so on. This threat to the validity of the inferences you can make about applicants' competencies is known as testing.

Actually, applicants can demonstrate an increase in their competencies in one area by improving the way they take the exam. Some academies that prepare students for the GMAT or other tests do not teach them content; they teach them how to pass the exam.

Some people have seen their careers fail because of tests

How can tests be improved?

People working in social sciences have a hard time accepting something important. We often try to analyse things that we don't see, concepts such as attitudes, poverty, wealth, health, competency, love. They are abstractions, which we agree to anchor to reality through observable indicators (we often operationalise them by means of the responses to items included in a questionnaire) and call them constructs.

However, we need to realise that these responses are intangible constructs, which means that they are just invented for us to understand and communicate with each other. And, as such, people can interpret them in different ways.

So, when we refer to constructs, we have to observe them from several points of view (triangulation). This is best illustrated by the "competency" construct which is typically assessed in leadership development programmes.

How so?

In classical performance tests, evaluators usually analyse competencies related to job performance only through professional colleagues (subordinates, peers, bosses, etc.) in the same specific context (work competencies). They always look for convergence among raters, since agreement among them can be interpreted as evidence of the construct's validity and its measurement (i.e., what we measure is a valid inference of that competency's existence).

However, when we assess leadership competencies, in general, we normally use questionnaires, which are the basis for the development programmes. These questionnaires are based on feedback from others – from the individuals' work and personal environments –, but the diversity of competencies, contexts and raters' perspectives makes the convergence of all these viewpoints meaningless. All perspectives are valid and, eventually, complementary.

If you want to know if I have self-control, don't ask my boss

What do your research findings reveal?

In our research, we demonstrate that there are more appropriate rater perspectives and contexts when assessing specific competencies. For example, if you want to know if I have self-control, don't ask my boss whether I have it or not. Ask my subordinates and peers, or, better yet, my wife and children!

However, if you want to know about my level of organisational awareness, ask my work colleagues instead of my wife, since she would probably have a hard time assessing this competency.

The person evaluating you has different criteria when assessing, for instance, flexibility. This will depend on whether the rater is your subordinate, your boss or your friend.

Development questionnaires are like a broken mirror

Development questionnaires are like a broken mirror: To get a global picture you have to put all the pieces together. If you want to give leaders proper feedback, you need all the possible sources of information about the person's behaviour. Besides, this can help them contingently apply the most appropriate leadership style.

We know that what friends see, compared to peers or subordinates, is different because their observations come from completely different contexts. You not only act differently based on who you talk to, but the person observing you probably has different criteria in terms of empathy, flexibility and so on compared to other people.

What contributions does your research make?

The poet, Ramón de Campoamor, was right. When we measure a construct like love, health, empathy or flexibility, context is key. We shouldn't analyse a competence by itself; we should also take into account the context surrounding it because the lens each of us uses to observe reality is different.

If you want to give leaders proper feedback, you need all the possible sources of information about the person's behaviour

For development tests to be effective, empathy, self-awareness, flexibility and so on have to be measured in different contexts. We show that a competency measured by the individual herself, by someone from the latter's personal environment or from the professional domain has very different results. For example, empathy ratings are usually low when assessed by bosses, peers and subordinates compared to partners, friends, classmates and family.

What are the takeaways from your findings?

If we are aware that not everyone rates the same competence in the same way, we can find environments that are better suited to evaluate each specific competency. For example, for organisational awareness, don't ask your friends; ask the boss directly. For leadership or teamwork, you can ask friends.

Adopting this new approach in development questionnaires allows us to customise our 360º surveys for specific environments depending on the competence. And this results in shorter questionnaires for each rater group – nobody likes long questionnaires! – which should improve data quality as well as feedback quality.

This article is based on joint research by Esade, Pompeu Fabra University and Case Western Reserve University.

All written content is licensed under a Creative Commons Attribution 4.0 International license.