The world’s weather is presenting a paradox. In many regions, conditions are unprecedented, different to anything seen before. Yet they are also more predictable than ever.

This summer’s extreme heat across Europe and the US was foretold days in advance. Last year, four days before Britain experienced 40C for the first time, the UK’s national weather service, the Met Office, issued a red warning — announcing an 80 per cent chance of record-breaking temperatures.

“The actual forecasting, and deciding how to message the forecast, was one of the easier things we had to do,” says Will Lang, the meteorological service’s head of situational awareness. “It was almost scary how clear-cut it was.”

Economists and political pollsters regularly see their predictions upturned by real-world outcomes. In contrast, weather forecasters have managed steady improvements in modelling a chaotic system.

The Met Office encapsulates this process. Remembered by many Brits for failing to forecast a 1987 storm that killed 18 people in the UK, it is today ranked as one of the world’s best-performing public weather services. Based in Exeter, south-west England, and largely funded by the UK government, it says its four-day forecast is now as accurate as its one-day forecast was 30 years ago.

Much of the credit goes to the expanding network of satellites, which provide data on cloud cover, humidity, wind and other variables, and increases in computing power, which allow more complex calculations. Already a five-day forecast in the US is accurate about 90 per cent of the time.

Better forecasts are giving governments and companies more opportunity to mitigate the costs of heatwaves and extreme weather, which are already large enough to be seen in gross domestic product figures. So far this year, the US has experienced 12 weather-climate events costing more than $1bn, including 10 severe storms.

But even “normal” weather causes all manner of shortfalls and inconveniences, from failed crop harvests to delayed flights. Increasingly accurate forecasting has the potential to improve performance across a swath of sectors.

Abundant data has created new opportunities for specialised or “hyper local” forecasts, mostly for business. Since the weather holds huge sway over things like heating oil and natural gas, when Citadel, the world’s largest hedge fund, hired Sebastian Barrack to expand its commodities trading in 2017, one of his first moves was to hire a 20-strong team of weather forecasters. Citadel now credits that forecasting team for much of its success in commodities trading, which accounted for about half of its $16bn in returns in the first quarter of this year.

Despite leaps in forecast accuracy, blind spots remain. Forecasters struggle with certain weather phenomena: localised thunderstorms, which can trigger flash flooding, are still at the outer limits of their abilities. Google, Nvidia and Huawei are among the technology companies developing AI-based forecasting, which aims to bridge some of the gaps.

Not to be outdone, the Met Office is also investing in machine learning and supercomputing power. Beyond being the UK’s national service and archive, it provides its forecasts to other countries and sells insights to private clients in fields as wide-ranging as aviation, retail, construction, agriculture, mining and insurance.

The investment has already aided the Met Office in its most important task: forewarning. The 1987 storm, which the office forecast would hit France rather than the UK, was difficult to predict because of a sting jet — a burst of powerful winds lasting just a few hours.

Today, its forecasters would be able to identify the storm hitting the UK as a low probability event, allowing a warning to be issued. “We work on a mantra of ‘no surprises’,” says Lang.

‘Pushing against chaos’

The science of weather forecasting dates back to the mid-19th century. In 1854, Robert Fitzroy, formerly the captain of HMS Beagle, on which Charles Darwin sailed, founded the Met Office. Its initial role was to improve safety for ships, by gathering a few scattered observations and charting them.

By the 1920s, there was an attempt at more sophisticated prediction. Lewis Fry Richardson, a British mathematician, honed a series of equations, based on Newtonian physics, that could calculate how pressure, temperature and other variables interact.

Being an intrinsically chaotic system, the weather is highly sensitive to small changes in initial conditions: the proverbial butterfly can cause a hurricane. As a forecaster, “you’re pushing against chaos”, says Paul Davies, the Met Office’s chief meteorologist. It wasn’t until the 1960s that better data and computing power allowed a step-change.

Data quality is crucial. The UK has a few hundred observation stations containing meteorological instruments, often at airports. These are now complemented by sensors on aircraft, ships and buoys, as well as remote sensing by radar, satellite and weather balloons. (Satellites alone struggle to read the lowest part of the atmosphere.)

Everyday devices are now capable of feeding into forecasters’ models. “We’re not at the stage yet of using the temperature from your car, but most cars are measuring temperature and it would be technically quite easy to share that data,” says Douglas Parker, a professor of meteorology at the National Centre for Atmospheric Science, University of Leeds.

Weather forecasting also marks successful international co-operation. Through the UN World Meteorological Organization, established in 1950, countries agree to share their weather data.

In 1997, the Met Office worked with a grid of boxes 90km wide. That resolution has gradually shrunk to 10km today for the global grid. For the UK, the resolution is 1.5km. These boxes are stacked vertically, into the troposphere and the lower stratosphere. Meteorologists now debate whether there is an optimum resolution, below which the accuracy of forecasts no longer improves significantly. “There might be other ways that we can use our computer power to increase the accuracy,” says Helen Roberts, a Met Office forecaster.

One of those ways is so-called ensemble forecasting. Traditional models are “deterministic”: observations are plugged into a model, which provides a single forecast. In ensemble forecasting, the observations are tweaked in random ways, and the model is run multiple times simultaneously on the different data sets.

The random tweaks compensate for the limitations of the observations and of the model. They give a probabilistic account of different outcomes; reasonable worst-case scenarios can be picked up earlier. “Introducing randomness into this incredibly complex system actually improves it,” says Roberts. “It gives us a plausible range of scenarios.”

The Met Office’s global ensemble forecast currently uses 12 different scenarios. This involves vast amounts of computing power and electricity. By the end of 2025, the service will only use ensemble forecasting, enabled by one of the world’s largest supercomputers, initially budgeted at £1.2bn. (The supercomputer, based in Microsoft data centres and intended to process 60 quadrillion calculations per second, was due to be ready last summer, but was held back by Covid chip delays. The Met Office now expects it to be online in spring 2024.)

Human forecasters retain a role. They compare the models’ outputs to real weather observations and adjust them for biases. “For the time being, human meteorologists provide so much added value and insight above raw model data,” says Roberts, adding that their intervention is most necessary in unprecedented events.

But the models should perform robustly even as the world warms. Extreme weather is not necessarily hard to predict: July’s European heatwave involved a “loopy” jet stream, which allowed high pressure to settle for many days in five regions of the northern hemisphere. “We believe we’re solving equations of physics that are fundamental. So if we move into a hotter world, the equations should still be valid,” says Leeds university’s Parker.

Some places are easier to forecast than others. Britain has comparatively unpredictable weather because it sits off the edge of a continental landmass and next to a large ocean. It is affected by six air masses, large volumes of air, including ones from north Africa and Greenland that bring different levels of humidity and temperature.

But prediction is easier than in the tropics, where there are fewer observation stations, where the effect of the Earth’s rotation, a stabilising force on the weather, is weaker, and where thunderstorms can be frequent.

“In a chaotic system, there is a timescale over which [things] are no longer predictable,” says Parker. “We think for our mid-latitude world it is something like two weeks. For the tropics, it might be a day [at least for rainfall].”

The Met Office has meteorologists embedded with customers, particularly airlines and airports, who want tailored interpretation of forecasts. Demand for tailored services is likely to grow. As the UK’s share of renewable energy increases — from 11 per cent in 2012 to more than 30 per cent in 2022 — understanding how hard the wind will blow or the sun shine has become key to ensuring the lights stay on.

The UK’s National Grid Electricity System Operator is investing in increasing the frequency and quality of the forecasts it receives, including using machine learning. It is also working on analysis of satellite imagery of clouds to see if its solar radiation forecasts can become more accurate.

Whiffle, a specialist forecaster that grew out of a project at Delft University of Technology in the Netherlands, has taken the grid concept and supercharged it. Rather than focusing on 10km boxes, it says it can predict the weather based on areas as small as 10 metres squared.

This can give wind farm operators advance notice of how much they are likely to generate from individual turbines, aiding their trading in power markets. It can also help with the optimal placing of wind turbines within a bloc.

“The margins on these projects are not normally huge, generally in the region of 6-10 per cent,” says Remco Verzijlbergh, Whiffle’s co-founder. “So if the yield you get from the wind is lower than you hoped for, the economics can quickly blow up.” The company works closely with General Electric and Shell, among other customers.

Better weather ahead

Last month, exactly a year after accurately predicting record-breaking temperatures, the Met Office found itself under scrutiny — from England cricket fans. The fans wanted dry weather to enable England to try to beat Australia in an Ashes Test match in Manchester.

The Met Office suggested that no play would be possible on the third day of the Test, a Friday. A rival forecasting service, AccuWeather, offered a more optimistic view, which was quickly adopted by fans. The England team itself judged that an app called Home and Dry, which blends data from the Met Office and the European Centre for Medium-Range Weather Forecasts, was the most accurate guide. “It’s impossible to know with weather forecasts,” sighed a BBC radio commentator.

Overall, the Met Office’s forecast was largely accurate, and England’s rain-affected match finished in a draw. But the episode highlights one of the shortcomings of current weather forecasts: pinpointing exactly when and where rain will fall.

The physics behind rainfall is an “extremely non-linear process,” says Parker of Leeds university. Most rain starts as ice crystals or snowflakes and turns to rain as it descends within a cloud. It is affected by multiple factors, including interaction with other water in the clouds, the topography of the land below and the level of aerosols in the atmosphere, including salt from the sea and smoke from cities. It is so difficult to track rainfall that the Met Office mines social media data for mentions.

Given the difficulties in forecasting, attention is being paid to “nowcasting”: tracking rain over the extremely short term. In 2021, DeepMind, an artificial intelligence arm of Google, and the Met Office published a paper in the journal Nature suggesting that machine learning could outperform traditional, physics-based models at nowcasting rainfall within the next 90 minutes. “Two hours’, three hours’ time — that’s where the real power of machine learning could kick in,” says Davies, the Met Office’s chief meteorologist.

Machine learning has also been trialled over longer time horizons. Researchers at Nvidia said last year that their model was able to pick up extreme weather events such as hurricanes two to six weeks in advance. It required much less computing power than traditional forecasting, running 320 ensemble forecasts in just three minutes. Nvidia now claims its week-long forecasts outperform those of the European Centre for Medium-Range Weather Forecasts for variables such as precipitation.

“The interesting thing is how [AI-based forecasts] opens up for the private sector to compete with national weather services, because you don’t need the same infrastructure of scientists or computing experts to make that work,” says Parker.

But models trained on historic data may not be reliable in a changing climate. “If we have an AI that’s a statistical fit to the climate that we know, and then we have a different climate, it’s not so clear that that model will continue to be accurate. We’ve got no guarantees,” Parker adds. For example, the possible collapse of the Atlantic Meridional Overturning Circulation, the system of ocean currents that helps to stabilise Europe’s climate, could create unprecedented conditions and challenge any models that have not been trained for such scenarios.

Improvements in medium-range forecasts, whether by AI or other methods, are a focus for meteorologists. Such probabilities could help retailers to decide what to stock, for example, and farmers to decide what to plant. “We might not be able to make a really strong statement about what the weather’s going to be like this summer or this winter, but even if we just talk about a slight shift in the odds — a 10 per cent increased chance of it being a hotter summer — combined with a lot of other information that you’re getting, that might just be enough to tip the balance,” says the Met Office’s Lang.

But mid-range forecasts would not help individuals plan holidays or weddings. “You’re never going to be able to forecast your barbecue a month in advance,” says Parker.

Conveying such nuanced weather information is a challenge. Already the public has access to an array of weather apps, often with hourly forecasts and shiny maps that foster an illusion of certainty. The Met Office is studying how to give a sense of the probabilities involved.

With extreme events, communication can be counterintuitive. In 2017, a small but intense storm hit the Cornish coastal village of Coverack, causing flash floods. On the radar, “you could hardly see it — it was just a little point”, says Davies. As a result, people nearby were unaware and did not try to flee the area. “Had we produced the perfect warning, we might have lost lots of lives,” he adds. People nearby “might have rushed to the car and been drowned”.

Since the Coverack storm, the UK has endured a number of unprecedented weather incidents. In 2020, a study by the Met Office had concluded that, while hot summers were rising, “the current chance of seeing days above 40C is extremely low”. Two years later, it happened. Now the service says 40C temperatures “could be the norm” by the middle of the century, even if the world takes some steps to limit carbon emissions.

Experts emphasise that even accurate warnings are no substitute for long-term adaptation and mitigation strategies, which help society cope with intense heat and rainfall. “Underneath is climate change. It affects everything,” says Davies. “How do we communicate that, but also present it in such a way people don’t see us as nanny state?”

Graphic visualisation by Bob Haslett and Kari-Ruth Pedersen