The Met Office is the UK’s meteorological organisation and was founded in 1854. The organisation is responsible for the UK’s national weather forecasting, and their Hadley Centre for Climate Prediction and Research is a world-leading climate research centre.
We spent two days visiting the Met Office making observations of the working environment and speaking to people who work in a range of different roles, but primarily with climate data. We also spoke to a number of Met Office people on the phone, and chatted with a former senior policy maker – now government adviser – who had been involved with trying to ‘open’ weather data held by the Met Office.
Read through the ‘Data Journey’, ‘Culture’, ‘R&D’ and ‘Policy’ tabs below to find out more and then add a comment to the discussion at the bottom of the page.
In the R&D tab you can read about the Hadley Centre for Climate Prediction and Research and two innovative Met Office projects (WOW and ACRE). The people that work in these sections process and use data that flows into the Met Office from various sources including Weston Park, Old Weather, Amateur Observers and the Climatic Research Unit.
Once you have finished, follow either the Red line to the IPCC (Intergovernmental Panel on Climate Change) or the Green line to Weather Market Data Supplier on the right.
Or, if you are interested in data that are flowing into the Met Office from other locations follow either of the Blue lines on the left to Old Weather or Amateur Observers, or the Red line to the Climatic Research Unit.
Our temperature datum arrives at the Met Office via the World Meteorological Organisation’s global telecommunication system (GTS).
Weather and climate observation data from automated weather stations such as Weston Park in Sheffield are all transmitted via the international GTS. The UK Met Office and other national meteorological organisations then download these weather station observation data from GTS and ingest them into their own systems, alongside data from radar, weather balloons, aircraft, satellite and buoys.
Data for forecasting are the first priority. Near real-time observations from synoptic weather stations are downloaded from GTS, decoded, formatted and incorporated as quickly as possible into the numerical weather prediction (NWP) system used by weather forecasters.
Climate data from synoptic stations also arrive straightaway via GTS. However, climate data from the network of co-operating climate observers, such as Weston Park Museum, arrives at a slower pace – typically daily or monthly.
This short Met Office video discusses how the organisation processes the weather observations that arrive at its data centre from stations such as Weston Park.
Once observation data have been received by the Met Office, they go through a rigorous quality control process. This aims to ensure consistency and accuracy, and to remove errors.
Three types of automated consistency checks are performed:
- Whether the observation is compatible with other weather variables – e.g. is the temperature compatible with the pressure and rainfall readings.
- Whether the observation falls within expected limits for the climate.
- Whether the observation is consistent with nearby stations.
…looking for errors, consistency. So there’s a set of automated checks they’ll do for, you know simple things, range values, numbers that are clearly in error. They’ll take information from the automated quality control that the forecasts provide, and use that as well. So there can be all sorts of reasons why the observations will be in error, there may be a rain gauge has got stuck. Then if every other rain gauge in the vicinity has 20 millimetres and that one has zero– then you can be fairly confident that it’s probably an error. [MO_01]
Any data that are flagged as inconsistent by these automated checks are then examined by trained meteorologists in a dedicated Met Office quality assurance team based in Edinburgh.
Climate observation data are typically checked on a monthly basis, and then re-assessed a few months later.
Whilst quality checks are carried out, it is important that errors are avoided as far as possible through management of the weather station site, annual site inspections and equipment maintenance.
Nearby trees, roads and buildings, and site elevation, can all impact on observation data. These characteristics are therefore recorded in the station metadata. Any movement in the location of the station is also logged and evaluated for the possible impact on the observations.
Archiving the Data
Following quality control, all real time observation data are stored in the MetDB synoptic database. Climate observation data from automated stations are then integrated from MetDB into the MIDAS database (Met Office Integrated Data Archive System) twice daily. Non-automated climate records are manually added to the MIDAS database at a later date.
MIDAS is specifically designed to support the needs of various climate applications. It covers 1853 to the present and incorporates UK land and marine observations, and more recently, global climate observations data.
MetDB is the backbone of all weather forecasting undertaken by the Met Office. Similarly, MIDAS supports all Met Office climate-related activity. MIDAS and a subset of MetDB are also made available for UK-based academic research funded by the Natural Environment Research Council (NERC), via the British Atmospheric Data Centre (BADC).
National Climate Data Processing
The National Climate Information Centre (NCIC) based in the Met Office Hadley Centre is responsible for producing UK gridded climate datasets and a range of summaries and analyses of climate data.
Our focus and our remit is very much UK climate monitoring, rather than global or international. So the wider team that we sit in build global datasets. So some of the well known global temperature series and so on. But our remit is very much on the UK monitoring aspect. So on a day-to-day basis we’re sort of maintaining and developing these datasets. [MO_01]
Observation data from weather stations such as Weston Park in Sheffield feed into these datasets:
So we take the data out of this MIDAS database. So all the observations for a particular day, or a particular month, and we create the climate statistics for individual stations. So for any observing stations, particularly the long running stations, we’ll produce climate statistics. So that might be the average temperature for the month, and it’s the long term averages for different seasons of the month. So we provide the sort of base line reference dataset for individual stations. But actually for a lot of what a lot of users are interested in is sort of integrated statistics for the county– was the UK particularly warm or cold? So we need to average up all of the observations we’ve got for the country. [MO_01]
The team at NCIC work to make sure all the grid squares across the UK have a data point:
Our approach to handling this process is that we take all of the observations and we interpolate them onto a uniformed grid. So we’re essentially making estimates of what the observation would have been at ten and a half thousand locations across the country, based on the observations we’ve got. [MO_01]
They also ensure that the data set is consistent over time. The team complete a process called data homogenisation to make sure that any changes in weather stations over the years – for example, location and instruments – do not impact upon the accuracy of the observation data in the climate record:
Now, for a climate record, obviously, one of our key considerations is the homogeneity of the record at the time. And one of the biggest sources of inhomogeneities in that, are the fact that the number and location of observing stations changes all the time. [MO_01]
NCIC’s work is used by policymakers, public sector and commercial organisations interested in planning for and mitigating against climate change. For example, the national gridded climate datasets support the UKCP09 Climate Projections initiative, which aims to support researchers and decision makers tackling climate change impacts and adaptation.
Much of the data produced for the purpose of climate science is shared openly amongst academic researchers.
UK monthly and annual climate summaries and averages, UKCP09 gridded datasets and monthly climate records for historical weather stations such as Weston Park, are all made available for the public by the NCIC on the Met Office website.
Central repositories, such as the BADC in the UK and the National Climatic Data Center (NCDC) operated by NOAA in the USA, facilitate sharing within the scientific community.
BADC provides access to several weather and climate datasets produced by the Met Office and the Climatic Research Unit, although a licence and proof of academic research status is required to gain full access.
Selected sets of UK observations, forecast, climate and forecast data are also available as open data from the Met Office DataPoint API. They are licenced under the Open Government Licence, which means they can be downloaded and used by anyone. However, there are limitations on the volume of data that can be downloaded from DataPoint. High-volume users, such as the weather market data supplier on the green line, still need to arrange a wholesale data contract with the Met Office. These contracts have conditions and fees attached.
Whilst meteorological science is at the heart of the UK’s Met Office, over recent years the context that this science exists within, like many public institutions, has shifted towards an increasingly business orientation.
The fusion of science and business
Senior management within the Met Office have been keen to develop the relationship between the science and business sides of the Met Office, and to create a synthesis of the two known internally as the “Bow-tie”.
As one climate scientist we spoke to described:
Well, it’s service driven…but, underpinned by the integrity of the science and the methods that are applied. [MO_1]
These efforts to seamlessly draw together the two ends of the ‘bow-tie’ appear to have impacted the way science is imagined by some within the organisation. For example, the language used by one scientist to discuss their work was heavily shaped by a managerial discourse of “our remit… function” and the term “climate services” [MO_1] was enthusiastically embraced. Similarly, a discourse of “compliance“ [MO_2] was evident in relation to the organisation’s relationship with government policy makers and regulators.
Efficiency was also a concern for some. When discussing the reduction of the rain gauge network by nearly 50% since the 1970s, the effectiveness and efficiency of the observing network were the primary considerations for one individual. The idea of a weather station being an important part of a local community and culture as at Sheffield Weston Park was not explored.
This suggests that as our datum enters this ‘big’ data infrastructure its socio-cultural context has shifted towards a more technocratic space that is less about people, and more about utility, functionality and economic reasoning.
The Met Office’s corporate identity is central to its functioning. A variety of techniques aimed at generating and presenting a strong and cohesive identity are observable.
A strong emphasis is placed by the organisation on brand identity. Met Office colours and logos saturate public spaces and offices within the buildings.
In the two images to the left you can see the heavy use of the ‘swoosh’ logo on office surfaces and plant pots, and the use of banners to present a corporate identity
The change in name to Met Office in 2000 further represents the importance placed on brand identity by the organisation.
The organisation’s public identity is also shaped by restricting the external communications of employees. Engagement with potential critics, including the press and academics, is heavily controlled by the organisation’s Press Office. This indicates a level of guardedness within the internal culture of the organisation.
Technology is at the heart of Met Office culture. Weather forecasters based in the Operations Centre can have as many as 11 computer screens to work from, and large screens display the latest updates from forecasts and social media. On the top floor of the Met Office, an impressive visualisation of the global climate is displayed on a huge spherical globe which sits in its own specially constructed alcove outside of the Hadley Centre.
Many forecasting and climate models rely on the processing power of a supercomputer – a relatively small computer that fills only a small portion of a vast, largely empty room. In 2014, the government confirmed that the Met Office will soon be acquiring a new supercomputer to further increase its processing power. The new technology will cost £97million and the decision to fund it was subject to an extensive UK government Science and Technology Committee hearing.
The Met Office was excited by these developments and is keen to demonstrate its technological capabilites to visitors – the supercomputer is part of the official Met Office tour and features on its own wesbite. Watch the video below in which the history of supercomputers at the Met Office is introduced.
Uncertainty and vulnerability
The Met Office’s effort to build and present a strong public identity points to vulnerabilities faced by the organisation within contemporary forms of public sector governance.
The development of a corporate identity and a restricted discursive space are representative of broader trends in public sector governance that have been pushed within the UK’s public sector over the last three decades.
A strong corporate identity has become perceived as necessary to strengthen both the Met Office’s position in the commercial market place and its reputation amongst domestic and international networks of policy makers, funders, and potential collaborators.
In the case of the Met Office, the managerial impulse to control discursive space can also be understood as a response to the current vulnerability of the institution as it comes under threat from some groups of policy makers who believe some of its functions should be outsourced to private sector management companies. These issues are explored in more depth in the Met Office – Policy tab.
Spaces of innovation
Whilst the corporate identity of the institution is strong, we also discovered innovative projects run by small groups of scientists within the Met Office that seemed to exist both within and outside the restricted structures of their parent organisation. As one scientist described his team:
The organisations work a bit like the Royal Navy, we work like Pirates of the Caribbean. (MO_6)
Some of these projects such as ACRE and the WOW project are explored in more depth in the Met Office R&D section.
In this section we introduce specific sections and projects at the Met Office that are focused on climate research and innovation. We begin at the Hadley Centre for Climate Prediction and Research, one of the world’s leading climate research centres, and explore the ACRE project – an international collaborative project which runs out of the Hadley Centre. We then introduce the Weather Observations Website (WOW) project which collects weather data from amateur observers around the country.
The Met Office Hadley Centre for Climate Prediction and Research sits on the third floor of the Met Office. The culture is less corporate than on the lower floors of the Met Office. The absence of any branding within the working environment is particularly noticeable.
The research undertaken by the Hadley Centre includes a range of activities including creating and managing climate datasets such as CRUTEM4, HadCRUT4, and the UK’s national gridded climate datasets. Scientists are also involved in modelling and analysing data including the development of new techniques such as reanalysis.
The scientific research of the Hadley Centre is clearly aligned with academic principles and practices. Academic journal articles are a primary mode of dissemination for research findings. Communication with the general public, businesses and policy makers also appears to be an important part of scientists’ work.
The activity of the Hadley Centre is clearly focused on science. The role of data in enabling that science to take place is unquestionable. Some scientists have a deep affinity with the data work they are engaged in:
Fundamentally I think I’ve been a data scientist since before data science was fashionable. Right. What I like to do is to find new information, and to see our picture of the world improving as we get a clearer image of what’s going on. [MO3]
I always find the data analysis side of it really, really interesting….So that’s definitely something that I–, yeah it keeps me going, keeps me interested. [MO_4]
Science is necessarily engaged in the pursuit of new knowledge, and this can be seen in the way that climate datasets are periodically updated to new versions – for example the publication of HadCRUT3 in 2006 and HadCRUT4 in 2012 – in order to refine and improve the scope, certainty and knowledge about the underlying data that climate scientists work with:
Well, there’s always room for improvement…In the change from [HadCRUT]3 to 4 we had a large increase in the amount of temperature data going into the datasets…We had refinements to the way that we presented the uncertainty of information to make it more useable by scientists trying to use the data…. we had, from the sea surface temperature side of things, some updates to understanding of the way that observations made by different platforms differ and how that affects the temperature measurements…there’s continuous updates to the kind of methods available…progress in various different related fields and statistics. And we try to gradually, as techniques improve try to incorporate them into the datasets. So it’s more of a kind of gradual kind of research process as we gain more information and try to incorporate that into the datasets to improve them as we go. [MO_4]
Whilst the scientists recognised the value of more and better data, the broader funding environment they work within tends to be more focused on scientific results and technology, rather than development of the underlying data infrastructure:
There’s a big push on climate modelling because of weather forecasting, and of course global change more recently. And so it’s been, there’s been a heavy emphasis on that, which is important and fine, but I think unfortunately at times the data side of things has suffered. [MO6]
ACRE – Atmospheric Circulation Reconstructions over the Earth – is a project run out of the Hadley Centre. The Met Office is one of nine core international partners, although there are several hundred organisations from around the world involved in the project. The project makes a significant contribution to the development of climate reanalysis techniques.
Reanalysis is a technique which potentially offers many benefits for improving our understanding of climate change. Whilst data from climate stations such as Weston Park are valuable in order to improve the accuracy of the technique far more data about historical weather conditions is needed to feed into the reanalysis models.
One significant area of activity for ACRE is therefore the recovery of historical weather observation data from around the world. Some of this work involves working collaboratively with data holders and demonstrating the benefits of sharing weather and climate data with climate scientists. Other work involves the discovery and transcription of archived weather observations, for example the shipping logs that volunteers on the Old Weather citizen science project are working on.
Watch the video below by Hadley Centre climate scientist Philip Brohan to see where data recently recovered from archives (yellow dots) have increased scientists’ confidence in the climate reconstructions they have produced using reanalysis techniques (yellow highlights).
Despite the importance of historical data for reanalysis, funding to support data recovery efforts is difficult to find:
The old questions of money and personnel become critical. And we really do work on a shoestring, there’s no smoke and mirrors out there in what we do…They all think it’s great…and they all want the result, but they’re not really prepared to pay for it (MO_6)
I guess what we suffer from is the thing that I guess we’re not seen as sexy…Especially the data part of it, you know, everybody’s sort of, “Oh data,” you know. (MO_6)
The frustration we’re having is because when we find there’s a block of data and we’ve got to think how the hell are we going to get this imaged and digitised and into the database, because you’ve got to find people with money. (MO_6)
Despite this constrained environment, the ACRE project is able to “chip away” (MO_6) at the problem due to its freedom to innovate. To a large extent the project seems to have escaped many of the corporate constraints of its parent institution:
I’ve been very lucky to be in a position where I can do this without too much from high saying, you know, what are you doing and why are you doing? You know, I just do it [laughs], and it made it happen…So you know, it’s about freely doing things, or more freely doing things than perhaps most other people in the organisation. (MO_6)
The culture of ACRE diverged quite significantly from that of its parent organisation and similar institutions. As one scientist described:
The organisations work a bit like the Royal Navy, we work like Pirates of the Caribbean. (MO_6)
This enterprising form of science prioritised flexibility, serendipity and freedom over careful planning:
You’ve just got to sort of be completely flexible, and keep thinking laterally, and seeing where things might pop up. (MO_6)
The sort of serendipitous, if that’s the word, nature of this…there wasn’t great planning that ended up with this result. It’s just something that’s evolved, and you know, and it’s starting to have value. (MO_6)
With several hundred organisations around the world involved in the project, including a number of citizen science projects such as Old Weather, ACRE functions by adopting a bottom-up, organic form of organisation:
It’s up to them if they want to be involved…We try not to send out dictums to people to do this or that, but sort of say, you know, link in with the work that we’re doing now, you know, so that we all benefit. (MO_6)
I try to keep it on sort of on a bit of a knife edge here. We work in the twilight zone a little bit at times…I think in a lot of ways it’s the informality that has given us strength in different ways that allows it to sort of develop and then to be more flexible.(MO_6)
Despite this organic nature, ACRE is dependent upon a leader who can “keep all the balls in the air” (MO_6). In order to keep the project sustainable, it is hoped that other people will come forward to take on more of a leadership role in the future:
I would like to see…more younger people taking on more of the mantle of it. (MO_6)
It is also hoped that as institutions such as the Met Office begin to see more value in ACRE, the project will “get more traction and embedded more in to institutions like the Met Office and others” (MO_6). Although it was perceived that whilst this would be beneficial for the long term sustainability of the work, it was uncertain whether the Met Office, and similar institutions, would be able to maintain the organic nature of the project that was perceived to be one of its key strengths:
I don’t know what they’d do, I don’t know whether they’d be able to do it in the way that we do it, or I do it, because they would try to formalise it more. (MO_6)
The people behind the Met Office’s Weather Observations Website are working to improve the accuracy of weather forecasting, particularly in relation to extreme weather events at the local level. In order to do this they need to collect large volumes of local observation data, however the official observation network does not have the required level of granularity and funding is not available to invest in the development of the infrastructure.
The WOW project was therefore set up to collect weather data from thousands of amateur observers around the country:
To date we’ve received getting close to 300 million observations from about 6000 different observing sites. [MO7]
Scientific understanding and accuracy are at the heart of the WOW project. Yet, the project exists within a constrained environment and aims to meet the increasing data demands of meteorologists within a context of reduced budgets and demands for increased efficiency:
At that time there were various clear drivers that more observations were required, the forecast models increasing in resolution, super computers getting more powerful, forecasters are always keen to have access to as much live information as possible to compliment the information they’ve got from models or from existing data sources. And there was also sort of as across most organisations there were pressures to reduce costs and become more efficient. [MO7]
Data Quality and Quantity
WOW recognises that for certain meteorological activities there are benefits in simply having as much data as possible, however this is tempered with a need to maintain data quality.
A note of caution is present in the treatment of the amateur data. Significant effort is made to verify the quality of the data, and to assess it against official data sources, before it can be considered for Met Office use.
There is something of a tension between making the WOW website accessible and easy to use for anyone who wants to upload observations, and incorporating data standards to ensure the data has provenance and a minimum level of quality so that it is useful for the Met Office as well as the general public.
Overall there is a strong desire to ‘get it right’, in terms of using data standards, verifying data quality, and providing technical support.
The WOW project has been relatively cautious in its development. Investigations into its initial feasibility and likely acceptance were lengthy, and consultations were undertaken with various interested communities.
At the moment the project is still in ‘beta’ after 3 years of development. The data collected by WOW is not yet integrated into forecasting systems, however academic research is currently being conducted to gauge the suitability of the amateur data for forecasting purposes.
Whilst the project has attracted a high number of contributors, there has been little public promotion of the web site to date:
Still to date we haven’t done much in the way of announcing its existence to the world. [MO_7]
However, despite this caution, interest is developing in the international meteorological community and the Australian meteorological service have recently developed their own WOW platform in an effort to gather their own amateur weather data.
UK government and European Union regulations and policies have a significant impact on the processing and flow of data through the Met Office. In this section we explore two important areas of public policy: Data policy and Public Sector Governance.
Data that are produced and owned by the Met Office, including those generated by the Met Office equipment at Sheffield Weston Park, are subject to a number of different policy and legislative developments that impact upon their processing and re-use as public sector data. These include:
- EU Directive 2007/2/EC on Infrastructure for Spatial Information in the European Community (INSPIRE)
- EU Directive 2003/98/EC on the Re-use of Public Sector Information (its replacement Directive 2013/37/EU is currently being transposed into UK law)
- UK Open Government Data policy
- Various related frameworks and schemes at the UK government level including Exceptions to Marginal Cost pricing, the UK Government Licensing Framework and the Information Fair Trader Scheme.
These data policies and legislation also intersect with broader policy areas including competition law, and the Trading Fund model of public data production which is governed by the Managing Public Money framework.
The complexity of the data that are produced and processed by the Met Office mean that compliance with the developing legislative frameworks has been a difficult task, and Met Office policies and practices are still in development in relation to some of their data holdings.
INSPIRE (Infrastructure for Spatial Information in the European Community)
EU Directive 2007/2/EC on Infrastructure for Spatial Information in the European Community, better known as INSPIRE, is a Directive that aims to develop the infrastructure for spatial information in the European Union. The types of data covered by INSPIRE include a wide range of environmental data, including some types of meteorological data. The Directive should improve access to such data and make it easier to share and link with other data. The ultimate objective is to create a data infrastructure that can contribute to better informed environmental policy. There are a range of milestones for INSPIRE compliance, including set dates for when metadata, data and networked access to data will be available.
In order to move towards compliance with the INSPIRE Directive, the Met Office are working to overcome a number of challenges. These include developing a means for people to access their data and contributing to the development of INSPIRE standards.
Meteorological forecast data is different in nature to a lot of other data … you issue a forecast, but your forecast then has various different time elements to it. It’s the date that it was issued, the date that it’s valid for, and then you have forecast time steps going out, you know, five days or whatever. So it makes it quite difficult when everyone else is just going this is the data and this is the time that it was created, and therefore that’s when it’s valid….We need more than one time thing. So we’re working with the relevant bodies that are trying to develop the standards for INSPIRE. (MO_2)
Despite the challenges of INSPIRE compliance, the Met Office is confident about its ability to comply and enthusiastic about the benefits for interoperability with other organisations and easing re-use by third parties. It has therefore decided that the organisation will apply the INSPIRE principles to a wider range of data than those required under the regulations:
But we have just started a programme of work … for making sure that by 2020 we are compliant with INSPIRE, as we should be with the various milestones as we go forward. (MO_2)
We’ve decided that yes okay, INSPIRE’s only supposed to be about your public task, but actually well, it makes sense…To actually do it across everything, as long as it’s appropriate to do it. (MO_2)
Re-use of Public Sector Information
Directive 2013/37/EU on the Re-use of Public Sector Information amended the 2003 Directive (2003/98/EC) of the same name. The 2013 Directive is currently being transposed into UK law to replace the Re-use of Public Sector Information Regulations 2005. At present it is regulated in the UK by The National Archives.
The Directive covers data and information produced by public bodies (other than public service broadcasters, educational or research establishments) as part of their public task.
Since 2003, the Directive has stipulated that such data must be made available for re-use by others non-exclusively, without discrimination and in most cases at marginal cost. After some debate, the 2013 Directive still allows for above marginal cost charges for Trading Funds such as the Met Office, but stipulates that these charges must be calculated according to “objective, transparent and verifiable criteria”.
The Directive only applies to data and information that is produced by a public body as part of its public task. In relation to the Met Office, its public task is defined in relation to the agreed outputs of the Public Weather Service part of the organisation. These include things such as national severe weather warnings, public weather services, civil contingency services, and some international services.
Definitions of public sector organisations’ public tasks have proved to be an area of tension in recent years. Some have argued that public sector bodies do not have a well defined public task. This has meant it is sometimes unclear which data the Re-use of Public Sector Information regulations apply to. There has therefore been a push from The National Archives, as regulators of the re-use of Public Sector Information (PSI), to make public sector bodies define their public task. This process has had both advantages and disadvantages from the Met Office’s perspective:
Sometimes though it’s the way it’s interpreted, so it’s your public task. Okay well, how do we define our public task? Well, you define your public task. Okay. So now we’re defining our own public task that will then be used to say whether or not we’re complying with legislation or not. So in some ways, yes it is useful to have the legislation there, but then there are times when, actually, we almost have a little bit too much scope…Which makes it more difficult to kind of go, no this is clearly what we need to be doing. (MO_2)
Providing access to the high volume of data underlying the Public Weather Service is a difficult task, which can make compliance with the Re-use of Public Sector Information regulations technically challenging:
We have our raw model data…and obviously, the observations that we have. And that underpins our public weather service outputs. So for us that forms our requirement for what we need to make available. That’s big datasets – they’re being updated four times a day or more, and that’s significant volumes of data. And we are struggling a bit at the moment because our models have got so big and so detailed… And we’re now trying to work out how are we actually going to make all of this data available appropriately, now and in the future. So we’re going through a piece of work on that. (MO_2)
Whilst most re-usable PSI in the UK has been charged for at marginal cost since the Cross-Cutting Review of the Knowledge Economy (2000) this has not been the case for Trading Funds such as the Met Office which are exempt from this policy.
The rules around charging to re-use information held by UK Trading Funds are stated in a policy called Managing Public Money. Maximum charges are legislated under the re-use of PSI regulations which allow for above marginal cost pricing in the case of Trading Funds.
Trading Funds such as the Met Office must also meet the standards of the Information Fair Trading Scheme, including meeting the charging criteria outlined in the Exceptions to Marginal Cost pricing framework.
Following these various regulations and policies, the Met Office currently charges third parties for re-use of its data, although they frame these charges as service-related rather than charges for the data:
But all of that’s charged for…They don’t really pay for the data as such…They pay for the service that goes with it…It gets to a point in the processing chain and we go right, anything from there on in is particular to them getting that data. So they’ll pay for those steps in the processing chain. Creating it in the format that is appropriate for them to get it, the message switching to get it to them, so the network charges and those sorts of things. We have an account manager that manages them getting it, and we have a kind of service charge. To make sure that if they have a problem with it they can call in, and those sorts of things. So the charges are around that – Rather than the actual creating data. (MO_2)
The ability of the Met Office to charge above marginal cost for re-use of public sector information has been challenged from a number of directions including the Open Data movement and the PSI lobby which represents commercial re-users of Public Sector Information including the weather derivatives markets. Despite this lobbying, the UK government was successful in negotiating at the EU for an option for above marginal cost charging to remain in the 2013 amendments to the Directive.
Open Data policy development in the UK began under the Labour Government, and continued under the Conservative-Liberal Democrat coalition government that came to power in 2010.
For data to be ‘open’ it means that anybody can re-use the data, for any purpose, at marginal cost – which for digital data is usually free of charge. Open Data challenges exemptions to marginal cost pricing under the Re-use of Public Sector Information regulations.
The development of open data policies at the Met Office has been complex and contested.
One former senior policy maker we spoke to described how meteorological data were “certainly one of the early priorities” (GOV_01) of the Open Data policy initiative under the New Labour government. This continued under the coalition government, with the Chancellor of the Exchequer announcing a significant release of Open Data from the Met Office in his Autumn Statement of 2011:
The Met Office will, from today and for the first time, release under the Open Government Licence (OGL) as machine-readable and machine-processable for unrestricted use, the following Public Weather Service weather forecast and real-time observation datasets, which together represent the largest volume of high quality weather data and information made available by a national meteorological organisation anywhere in the world. [Cabinet Office]
In response to this push from Government, the Met Office agreed that it would make open the data that was already available on its website:
So our open data is the data that’s on our website. So that is our weather forecast data that’s paid for by the Public Weather Service…Which also includes some climate data as well, so we have things like the anomalies and averages, and those sorts of things, they are also considered open data. So that was agreed with the Cabinet Office probably about two, two and a half years ago because we needed to somehow scope and we said well, if it’s on our website and people are using it then that would seem like a reasonable kind of boundary of what is open data. (MO_2)
The Met Office has found adoption of the new open data policies challenging, and some policy makers have complained about the Met Office being “resistant” (GOV_01) on open data.
Significant barriers for Met Office open data policy adoption include the Trading Fund business model it operates under and the complexity of the data that it generates, manages and processes. As one Met Office participant described, the organisation felt somewhat threatened by the new Open Data landscape:
So we wouldn’t have chosen to have made data openly available specifically had it not been for the government’s drive for open data and the way that that was going. (MO_2)
And PWS [the Public Weather Service] were being pressed to release all of their – what was on the website that needs to be open data, you need to make all your data open. And we realised that as a trading fund that causes us issues because if we provide the data for free then it’s very difficult to…Not only sell the data, but also it means that you’re opening up a huge amount of competition, they’re getting the data for free…And because of the way the organisation is set up we’re not as agile as we would like to be in some cases, and certainly not on the commercial side. And so they were struggling that if you just gave it all away well, we’ve got no chance. (MO_2)
One critical area of contention with some policy makers has been around the opening of the Met Office’s historic data:
I don’t think they’ve implemented in spirit or probably in letter what ministers agreed and announced they would do. And particularly I think their historical stuff, which they’ve definitely digitised. (GOV_01)
Currently, historic data is only available for commercial re-use through the Met Office’s commercial team, rather than as a wholesale product available under the Re-use of Public Sector Information regulations or as open data. Although, subsets of historical data are available free of charge through the Met Office library for personal and research use:
We don’t have any historical data through our wholesale manager. There was a time when we did and it caused confusion because we have a commercial team that were also providing historical data, and the two pricing models were different. And we went this is ridiculous because actually, wholesale is not about that it’s about the provision of the stuff kind of as it happens…So we have a team that make available that commercial–, that data under commercial licence fees. We have the library and archive though, so if it’s a member of the public or someone trying to do some research then they can go to the library, and the library will do that, a reasonable amount of extraction for them…And there’s no charges for that. But it’s done under this licence for private and research use only. (MO_2)
This barrier to open historic data has been challenged by some policy makers. These policy makers have argued that open historic data is of high economic value, looking to the weather risk markets and firms such as Climate Corp that take advantage of open weather data in the USA.
What’s less clear to me is whether the historic observation data has been opened, which was definitely a public weather service product. Historical data – they’ve dragged on – but they’ve definitely digitised – it has proved in other countries to be enormous value…So, for instance Climate Corp in the US. (GOV_01)
You can read more about policy makers’ views on the relationship between open weather data and weather risk markets in the Weather Market Data Supplier – Policy section. However, the fact that different policy makers and politicians see weather risk market innovation as a driver for opening data suggest that pressure is being put on the Met Office to expand its open data policy in this direction.
The government’s open data agenda has been a driving force behind Met Office developments to improve and standardise its data policy across the organisation:
MO_2: So we have a data policy steering group …the group was set up, and [a policy manager job], probably about two years or so ago. Up until that point there had been no kind of formal policy decision joined up making. People did what they thought was appropriate.
Q: So what triggered that? Was it a change in the landscape?
MO_2: It was basically the drive for open data. (MO_02)
These developments have included a number of changes around the management of Open Data to enable decisions to be made quickly and consistently about what data should be opened or otherwise made available for third parties to access and re-use. Developments include 8 criteria for open Met Office data and a system for classifying the different types of data the Met Office handles.
So we’ve divided our data into open, managed, research, and internal… Managed data doesn’t necessarily mean paid for, but it means that we will control it in some way. The terms and conditions might be different, it might be because it contains somebody else’s data, and they’ve put restrictions on us…So our national severe weather warnings are managed data. To make sure that we have a consistent coherent message so if there is a warning out everyone gets the same message. (MO_02)
New systems have also been developed to provide appropriate access to the data that is being opened for re-use. For example, the Met Office developed the DataPoint API to allow access to near real time open data. This API was funded by the Public Weather Service Customer Group, and was developed in response to the perceived shortcomings of the government’s data.gov.uk portal which cannot handle the real time data needed for commercial applications:
One of the things that we weren’t happy with…data.gov.uk…you can download a CSV. And we said that’s really not – certainly with the forecast data if we’re making this data open, because it’s about supporting economic growth you need an API. You need the Application Programming Interface because you need people to be able to have it in real time and just automatically get hold of the data they need. So we put in place something called DataPoint. (MO_2)
Whilst this DataPoint API currently only provides access to dynamic open data, plans are in place to make “everything available through the API” (MO_2) in the future, as this will contribute to Met Office compliance with the INSPIRE regulations.
Adapting to the new Open Data landscape has been challenging for the Met Office. Yet, some initial hurdles have been overcome and the organisation is beginning to take a more proactive approach to open data policy development. In late 2014, it announced it had become a partner of the Open Data Institute’s membership programme.
At the time of writing, the Sheffield Weston Park data generated by the Met Office’s own equipment are not available via the DataPoint service, or via other Open Data portals, however it is possible that this may change over time as the Met Office’s open data programme becomes more embedded.
Public Sector Governance
The Met Office has been a Trading Fund since 1996. Trading Funds are public bodies that must gain a significant proportion of their revenue from the commercial exploitation of the goods and services they produce. They became popular in the UK during the 1990s. The Met Office’s revenue comes primarily from the services it is contracted to provide to the UK government and public sector. It also provides some commercial services to the private sector.
Over recent years the Trading Fund model has been questioned by some policy makers:
“Trading Funds…essentially self-financing agencies by the sale of information…And that I think is now recognised as a flawed model. I mean if you read the stuff in Shakespeare’s report…it’s a crazy way to finance what are essential bits of the national infrastructure” (GOV_01)
Public Data User Group
In response to some of these concerns, the Met Office and other Trading Funds have been subject to a number of governance changes since the coalition government came to power in 2010.
These began with the 2011 proposal for a Public Data Corporation that would bring together key Trading Funds including the Met Office into one single organisation. Then the Met Office, Ordnance Survey and HM Land Registry were moved into the Department for Business, Innovation and Skills (BIS) in July 2011.
Whilst the Public Data Corporation plans were abandoned, in late 2011 the Chancellor of the Exchequer announced in the Autumn Statement that three new bodies would be formed: a Public Data Group made up of the Met Office, Ordnance Survey, Companies House and the Land Registry Trading Funds; a Data Strategy Board; and an Open Data User Group.
These three bodies are at the heart of new governance processes around the opening of Trading Fund data. The new decision making process is based on a commissioning model. Advisers based in the Data Strategy Board (and its sub-committee the Open Data User Group) advise Ministers to commission data from the Public Data Group, which can then be opened up for free re-use. A similar process is also utilised for commissioning data for public sector re-use.
The Public Data Group was formed based upon its members being Trading Funds holding economically valuable data that might be opened. However, these public bodies face different challenges in relation to open data, and at times seem like an odd grouping:
The PDG is more about us working with the other three organisations…To sort of share lessons and learn from each other about how do we do it…That’s one of the issues, when you have Met Office, Land Registry, Companies House, and Ordnance Survey, their data, Ordnance Survey there are similarities with, but Companies House and Land Registry their data is so different from ours…I think one of the problems that we have with the Met Office is that we collect data, which is our observations, but then we do stuff to it, and our reason for being is not collecting data. Our reason for being is making forecasts…So although there are some things we’ve learned from each other…We have far more alignment with people like the Environment Agency, and BGS, and CEH where they’re working on environmental data…So yeah, our PDG is a bit of a weird conglomeration of people. (MO_2)
Privatisation and Outsourcing
Despite these developments in the governance structure around the Met Office, some policy makers are still unhappy. One former policy maker we spoke to complained that Trading Funds such as the Met Office
haven’t been subject to some of the disciplines that even the rest of the public services have been subject to. So essentially have been monopoly suppliers of information they can charge whatever they–, they can charge cost recovery…but they’re the monopoly supplier so eventually they’ll just be able to pass on their costs to their information. (GOV_01)
Whilst rumours have circulated for some time about the potential privatisation of the Met Office and other Trading Funds, this policy maker preferred the option of outsourcing significant parts of the Met Office to a private sector management company:
I think the er, the experience elsewhere when national information assets have been privatised has not been good…Frankly, the value of the business is in the information that it holds…so people will only pay that value for the business if they think they can exploit that information. Which ultimately means either charging for it or denying access to it…And so privatisation is probably a poor model. A better model is the sort of thing that was done with the National Savings and Investments, NSI…Which is where it’s still a government owned asset…But it’s sort of hollowed out to a private sector management company to run it in the most efficient way possible…And that is you know, something that is begging out to be done in the case of the Met Office. (GOV_01)
The governance of the Met Office as a public sector organisation has gone through many changes in recent years. One of the key reasons underlying these changes is related to the flow of data through the organisation and efforts to ‘open’ parts of the UK’s weather data infrastructure. As data policy developments continue to unfold, the future governance of the organisation remains uncertain.
- Transcript of interview with MO_1 (pending)
- Transcript of interview with MO_2
- Transcript of interview with MO_3 (pending)
- Transcript of interview with MO_4
- Transcript of interview with MO_5 (pending)
- Transcript of interview with MO_6 (pending)
- Transcript of interview with MO_7 (pending)
- Observation photographs taken at Met Office
We’d like you to take a minute to reflect on some of the things you have read above and add a comment to the discussion below. Here are some questions to get you started:
- How do the different cultural values and practices across the Met Office compare with one another, and other stations you have visited?
- What are your thoughts on Met Office scientists using citizen science generated data from projects such as Old Weather and the WOW project to understand better the weather and climate?
- How do you think the fusion of business and science in the Met Office impacts upon the processing and flow of data through the institution?
- What do you think about the Met Office being run as a commercial business that sells data and services to other organisations?
- What do you think about the Met Office open data policy? Should it open its historic weather data?
Once you have posted a comment, follow either the Red line to the IPCC or the Green line to Weather Market Data Supplier on the right.
Or, if you are interested in data that are flowing into the Met Office from other locations follow either of the Blue lines on the left to Old Weather or Amateur Observers, or the Red line to the Climatic Research Unit.
By posting a comment you confirm that you agree to our conditions of participation.