Leadership and engineering: the missing pieces in the machine learning era

The most sought-after profession at any organisation interested in implementing machine learning is that of data scientist. Leadership and engineering are essential, but their role in the successful deployment of machine-learning models in production is often overlooked. Industry is realising that the only valid machine-learned models are those in production. In this article, we discuss how leadership and engineering are essential to monetising machine-learning initiatives.

Marc Torrens

Nowadays, we refer to the capability of machines to be extremely competitive at a specific task in a particular domain as artificial intelligence (AI). As Daniel Dennett argues in his book From bacteria to bach and back: The evolution of minds, we are creating extremely competent machines that have no understanding of the world whatsoever. Nonetheless, machine learning (ML) - a subfield of AI - is transforming all sorts of organisations, businesses and government bodies in unprecedented ways.

This is a natural consequence of two factors:

  • The availability of data as a result of the digitisation of words.
  • An increase in computing power.

Both of these factors are growing at an exponential pace, so this is just the beginning of a new machine learning age. However, although the press surprises us every day with new machine learning applications that perform better than experts at certain specific tasks, ML techniques are reaching a plateau.

Machine learning is transforming all sorts of organisations, businesses and governments in unprecedented ways

The innovations produced at the global scale are merely incremental. A few years back, most newcomers to the field saw deep learning as the solution to all AI problems. Nowadays, more people have started to understand the limitations and can see how far we are from general or strong AI. The focus thus turns to robust engineered solutions that make use of well-known techniques.

This revolution is currently being led by technology companies and mostly being implemented by data scientists. This is why data scientists are among the most sought-after professions nowadays. Data scientists understand data and know how to create ML models that discover hidden patterns in large data repositories. They are enthusiastic about fine-tuning models and finding the right parameters for a specific scenario.

Data scientists are among the most sought-after professions

However, data science is not enough for organisations to benefit from machine learning in production. We are seeing a process of democratisation of ML: various platforms and libraries are lowering the barrier to the incorporation of machine learning in non-technology companies.

From my point of view, the data-science and experimental part of ML will be less relevant and good leadership and engineering practices will make all the difference. The ML industry therefore also needs a new kind of profiles corresponding to leadership and engineering.

Tech leader

There is a huge difference between building a machine learning model for a specific task and embedding such a model in the heart of an organisation. In other words, there is a huge gap between having a model that works in the lab and preparing a model for people to use in their products and services. Most organisations aim to include ML in their businesses, but few are effectively embedding those models into the organisation's actual processes.

As ML has matured, it has become clear that leadership and software engineering are essential to any ML initiative. Although most attention has been focused on data science, specialised leadership and engineering are also needed in order to bring ML into the heart of industrial organisations.

The machine learning industry needs a new kind of profiles corresponding to leadership and engineering

Software engineers are responsible for embedding ML into production IT systems to make model outputs actionable for businesses. This requires integration with data sources, automation of ML workflow, and links between ML predictions and production IT systems.

Leadership and business development are essential to the ML lifecycle. Machine learning initiatives must be driven by the needs and goals specified by business developers. This new job requires a new type of leadership with knowledge of ML from a business point of view. New leadership should:

  • Understand the overall ML lifecycle, from problem definition to maintenance in production.
  • Be familiar with the main types of ML models, in particular what they do, what they need and what output they produce.
  • Understand the various ways to evaluate a model in order to estimate the benefit for the business of a given model and task.
  • Be able to interpret a model that is gaining new knowledge of a domain in order to improve decision-making.
  • Understand that machine learning is based on an experimental science that is not yet automated or industrialised. ML projects should therefore be managed iteratively to deal with uncertainty.

There is no doubt that machine learning is changing all sorts of organisations in unprecedented ways. In the recent years, most attention has focused on data science, which is the expertise needed to develop models to discover hidden patterns in large datasets. However, this new era requires a new type of leadership and engineering that is capable of a) making business sense out of ML through new actionable business insights, and b) bringing ML models into production IT systems at the heart of organisations.

All written content is licensed under a Creative Commons Attribution 4.0 International license.