The BBC's Weather Test, designed to assess the accuracy of UK weather forecasters, has finally agreed a protocol by which the forecasters' results will be judged. After more than 18 months of debate and public consultation, the project has at last formulated a protocol. The idea is to demonstrate to the public how much (or how little) the predictions of how hot or cold, wet or dry or windy compare with reality.
The Weather Test, which is being funded by the BBC and run by the BBC's Today Programme, is being evaluated by the Royal Meteorological Society and Royal Statistical Society. The BBC environment analyst Roger Harrabin, who devised the Weather Test, explains the new rules: "The forecasts will be evaluated on the accuracy of their one, three and five day forecasts as well as their longer term predictions. A number of forecasters have either signed up for the testing programme or are involved in negotiations to do so. Once the project is underway it will have a life of its own, overseen by the royal societies, myself and a senior editor on the Today Programme. It will be judged statistically by Leeds University."
So just who is likely to get involved? Clearly the Met Office would be a likely candidate and they have been informally approached during the public consultation. A Met Office spokesman said: "We are awaiting to agree a protocol about how this will be done. We have seen some drafts but a final version has yet to be produced."
Other likely candidates include Joe Bastardi, a well-known American forecaster and Chief Forecaster at WeatherBell Analytics LLC; and WeatherAction, run by Piers Corbyn who Harrabin refers to as a maverick weather forecaster.
Harrabin goes on to explain that "This project will compare the long-term performance of several forecasters. A comparison of forecasters is not currently available, and indeed our steering group is having difficulty agreeing a protocol to compare different forecasters. They will be judged on how accurately they predict rain, wind and temperature in a project lasting several years."
The results of the project will be staggered over a number of years: the one to five day forecasts will appear three years after the launch and the seasonal forecast results will be five years after the launch, which is hoped to be in the spring. All the forecasts will be kept confidential until the announcement of the results.
Before the launch date can be confirmed the project will need forecasters to commit to supplying information. The forecasters who will be invited to take part will be asked to submit predictions for the 35 stations from the UK's national climate observing network for which data is publicly available.
There are some concerns about how likely the project is to provide any value. How impartial the project will be when the key players behind it have commercial and academic partnerships with each other? And the project does not weight the forecasts appropriately to capture major weather events that have a high impact rather than just the majority of occasions when the weather can be relatively benign. Watch this space for more information.
The details in the agreed protocol for the BBC Weather Test can be found here.