There are a small number of data supply firms that provide weather data and forecasts for the management of specialist weather-based financial products such as weather derivative contracts. You can read more about weather derivatives at the financial markets station.
The weather data supplier we spoke to was a UK based firm that supplied data to a range of clients, including those participating in the weather risk markets.
We spent a day visiting this weather market data supply firm. Whilst we were there we observed the working environment and key data processes, and interviewed two key people: the CEO of the firm and an individual that worked closely with the weather data.
Read the ‘Data Journey’, ‘Culture’ and ‘Policy’ tabs below to find out more, then add a comment to the discussion below.
Once you have finished follow the green line on the right to Financial Markets.
Sourcing the data
The data supply firm sources weather data from 118 countries around the world, including data from Weston Park weather station in Sheffield. The data production and processing practices at Weston Park and the Met Office are typical precursors to the data arriving at the firm.
Traditionally, much of this weather data was gathered from national meteorological services such as the Met Office, but increasingly data are being sourced from alternative suppliers. These alternative suppliers include commercial and private weather networks, as well as those setup for research purposes and amateur observers.
We regard the national met offices as actually, probably now in terms of the numbers of datasets we get they’re in a minority. You know, the amount–, weather data is increasingly ubiquitous, and the importance of those data sources I think will fall. They will always be higher quality than other data sources, because they’re posher instruments. But you know there are so many other networks, so you get regional networks, you get academic networks, you get the local Ag[ricultural] networks. …And then the really big contribution is these private observers who increasingly are joining up to public networks and putting their data out there for the benefit of all. [DS_01]
Despite this proliferation of new data sources, there is still a preference for the high quality data of national meteorological services. In some cases, such as the USA, this data is available for the firm to use free of charge as open data. However, in other cases, such as UK’s Met Office, data is charged for on a wholesale basis.
Sometimes there is a market need for data in a location without enough high quality weather stations to base a contract on. Similarly, the firm may suspect that weather data that is available is of low quality or open to tampering.
In such cases where quality data is needed to support a large high value contract, the firm has been involved in the installation of new observation networks set up for specific contracts. Public sector and commercial partners have also been involved in setting up and running some of these networks with the firm.
For the weather data supply firm, one significant benefit of this approach is that all of the new stations are setup and managed in a standard way, with the same equipment, location criteria, observation practices and metadata.
This large volume of historical and real-time observation data from numerous sources is aggregated by the firm.
Current weather data is ingested daily over computer networks, and is transferred into the firm’s own weather data software platform. This platform is used by the firm as a repository and data management tool, as well as for its forecasting services.
In our core database we carry data of something like well over 100 thousand meteorological stations around the world. And that data is marshalled depending on what weather element it is. So it could be temperature, it could be humidity. It could be ground temperature and so forth. And in our database we support something over, I don’t know, 100 weather elements. But then each of those weather elements, weather variables is further marshalled into up to something of the order of 50 different data quality types. And that might be the matter of data quality respecting reporting conventions, or respecting where that data has come from. [DS_01]
Once the data has been aggregated into the database it is cleaned. Data cleaning is an essential process in making the data ready for use in the weather derivative markets. Quality control processes are relatively similar to those of the Met Office and other data suppliers.
Weather data is prone to occasional inaccuracies due to things such as equipment malfunction. There can also be inconsistencies between sites and historical periods. Differences in the way that observations are taken, for example different instruments or observation processes, and changes in weather station surroundings or movement of the observation equipment all contribute to inconsistency. Other factors contributing to inconsistency include the types of observations taken, the metadata provided and differences in the speed that data becomes available to use from different stations.
Since data is acquired from multiple sources by the weather data supply firm, these inconsistencies need to be addressed and errors need to be checked for. The firm applies its own data cleaning processes to the data it acquires to ensure consistency, reliability and transparency.
Data cleaning begins daily at 7.30am on the day’s updated datasets. It is a full-time task for a small team of staff. When new data arrives at the firm it is subjected to a number of automated checks based on its feasibility and consistency with other sources. Missing data and potential errors are flagged up, and a team of trained meteorologists conduct additional manual checks and make any necessary modifications.
The manager of the data cleaning team has a long history of working with UK meteorological data, and has built up a high level of knowledge and expertise which informs the processes and decisions made in cleaning the data.
We process something like 100,000 files of weather data a day, and that’s two years old that stat, from multiple providers. We interpret that data, we load it into our database. And for roughly 5000 of those sites they then go through, essentially a semi real time quality control process. So that data will be handled by the team in the corner here…And they then use the tools that the development team have produced that alert them to data that looks out of line – that embellish the data with everything from radar images, satellite images, they’re looking at what’s going around, you know, neighbouring stations. And every data point we produce a synthetic. So if Heathrow reports 25 degrees T-max, and Northolt and a couple of other stations around are saying 20, we’re going to say that looks wrong, and we’ll investigate and look for consistency. [DS_01]
Importantly, the original data as supplied to the firm by meteorological services and other providers is kept alongside the cleaned data as an audit trail. The manually amended data is also annotated with cleaning codes and notes explaining the decision taken.
We’re going to put this through the microscope, and we’re going to tell you everything about this data. We’re going to give you the raw data, we’re going to give you the quality controlled data. Which means anything that’s dodgy we’ll change, and where there’s missing data, which we know will screw up your ability to price with it, we will fill in, yeah. And you will see the audit trail of the raw and the cleaned. But we will also expose other detail that’s material. So many of these sites have moved. You mention a site that hasn’t, but most sites have wiggled about, you know, closed, moved, instruments have changed. So we can document that, and if necessary provide them with another dataset that allows them to understand that transition. [DS_01]
Data Supply and Use
The firm supplies a range of datasets to the weather risk market via its weather data software platform. The datasets it supplies include both historic and current weather data, and may be supplied in ‘raw’ or cleaned form.
It also provides clients with station level forecasts, using outputs from the European Center for Medium Range Weather Forecasting (ECMWF) and the US National Oceanic and Atmospheric Adminstration’s Global Forecast System (GFS). Datasets are available from the firm on a subscription or ad hoc basis.
In some cases the firm might provide a bespoke dataset and supporting analysis for a specific weather risk contract:
This process will be modified to take into account any definitions that are within a term sheet [terms of the contract]… So we take account of that. It may include more inspection. It may mean we have to insert a client’s prescribed checking methodology, or infill methodology. As per the terms of their contract that they’ve agreed with their counter parties. [DS_02]
Unsurprisingly, the activities of the firm were driven towards the monetisation of weather data. The firm was largely orientated towards the needs of their clients. Many of these clients were using data products supplied by the firm to engage in the weather risk markets. The high-speed of financial transactions within these markets impacted significantly upon workflow within the firm:
The reason we have to do it daily is if a client’s got weather risk on it they need that updated now…They need to know how it’s going, how much they’re going to be losing. [DS_01]
The need for high quality, reliable data within the weather risk markets drove the firm’s quality control efforts:
They [end users] have an insistence on data that’s come from secure sites, you know, third parties that haven’t got a vested interest, and that sort of thing. So the provenance is important. [DS_02]
Members of the firm were keen to stress that they perceived themselves to be neutral and objective, and therefore an authoritative actor that could contribute to the development of trust between economic agents within the weather risk market:
The only way to actually sort out your dataset is to have somebody, you know an independent body like [company name] to clean that data, to understand it and come up with good plausible estimated in filled data. [DS_02]
We’re vested with a fair amount of authority to be the referee, and you know, we explain, but we don’t negotiate… because we’re neutral and that’s why you’ve paid us. [DS_01]
A lot of value was placed on acquisition and growth within the firm:
So we acquired [the data side of a failed public sector business venture] out of administration. With that we acquired some core datasets and one key person who’s now head of our forecasting data. And really our growth has been driven by developing that organically, and bringing in more people, building systems. [DS_01]
This spirit of acquisition was also strong in relation to the firm’s relationship with weather data. A strong emphasis was placed on acquiring as much data as possible, prior to sifting and processing:
So essentially, you know, we make it our business to collect data wherever we can, wherever we’re permitted.[DS_01]
Anything and everything….You know, you still want more and more and more [data]. [DS_01]
They’re all of interest. [DS_02]
The firm was engaged in an ongoing struggle to acquire more weather data. This activity often resulted in frustration for some individuals within the firm who perceived that they were being denied what they believed they should be entitled to:
So the data is there very often, but not available. [DS_01]
When we first got involved in this market we would go to conferences and there would always be the moan about lack of data availability and how data should be open. And you don’t hear that anymore…. But that doesn’t mean that they’re sated. [DS_01]
Not everyone has the same open data policy that varies a lot, there’s a lot of politics goes on in it.[DS_02]
Barriers to the firm’s acquisition of data included restrictions on the commercial re-use of data:
“No, our data’s not for commercial purposes.” An absolutely infuriating response, but one we get used to. [DS_01]
Corrupt practices within some countries also restricted aquisition:
Kenya is the same, huge network, but completely corrupt. So when we go to buy weather data we sometimes get the response from these guys, yeah that’s fine, you know, I need $2,000 in my personal bank account and you can have everything, and we have to say no, obviously. [DS_01]
Where data were available from organisations, it often came at a cost to the firm, and they were keen to lower that cost as much as possible:
So the more data that is open the more value we add, therefore the happier we are, and the happier our customers are because they get more data cheaper, and everyone wins. [DS_01]
The CEO of the weather data supply firm was proud of his involvement in the weather risk industry:
We were there right from the start. [DS_01]
Making a contribution towards market innovations seemed to be at the root of this individual’s pride in his work:
The availability of weather data is an important driver in the switch from indemnity to parametric. And we believe that that’s, you know, that’s quite a nice thing to be involved in. [DS_01]
The CEO was also proud that his role enabled him to “help the guys“:
[We] produced some software that helped the guys taking that risk, insurance companies, understand how to price it. [DS_01]
So in the vast majority of cases it’s the big guys for whom it works. [DS_01]
Whilst individuals located in other nodes experienced a similar sense of making a contribution to the scientific community and society in general, the activities of this firm are geared towards helping “the big guys” – powerful market actors within the global economy. This reference to “the guys” is also representative of the construction of masculinity observed within this space – traditional, self-assured, domineering and at times condescending:
There are some parts of the world…Africa, which are very difficult. And it’s not always because the data isn’t being observed, it’s because they don’t realise they need to give it away. And this is to their extreme detriment. [DS_01]
Whilst the general office environment was relatively underwhelming, we observed that there was effort being made to introduce ‘hip’ consumer goods into the environment, perhaps in an effort to bolster perceived status and develop an aspirational culture:
Very ordinary red brick office. The offices inside are also very ordinary and bland…The office has little in the way of design…There are signs at attempts at quirkiness/hipness, e.g. the boardroom table is a converted snooker table, antique clock in the boardroom, a table football machine in one of the meeting rooms, leather sofas, and in the kitchen they have a breadmaker (and today’s loaf), a delivery box of healthy snacks, fruit basket, and fresh coffee. [Observation notes]
Whilst the firm positioned itself as neutral in relation to the provision of quality weather data, the CEO of the firm expressed strong neoliberal beliefs which drove the firm’s mission to support and expand the development of weather risk markets.
We were talking about the importance of weather data in creating perfect markets. [DS_01]
How can there ever be any detriment to a contract freely entered into by informed economic counter parties… It’s only when there is compulsion attached to something that you get economic detriment. [DS_01]
The work of the firm was perceived to be making a significant contribution to the development of efficient – even “perfect” – markets:
So clearly weather is one of the most important, probably the most important element in understanding crop yields. So if the community of speculators has complete information about weather they can anticipate what’s going to happen. They can therefore freely deploy capital, and therefore the provision of weather data has an important, but not necessarily terribly obvious role in making markets efficient. [DS_01]
Whilst there was a demonstrated desire for acquisition and private profit, risk was perceived to be something that needed to be moved as far away from investing individuals as possible:
That same principle…of institutionalising risk, moving it as far from the individual as possible to the collective. [DS_01]
For the CEO of the firm the market was perceived solely as a solution to significant societal issues related to weather risk. For example, when we asked if there might be any negative social consequences resulting from trading in weather risk – the CEO did not believe there were any:
No, I mean so long as it’s done between consenting adults who understand the benefits and the risks, no. [DS_01]
Yet, as we argue in the Financial Markets station significant social risks are present in these markets that should be investigated further.
As explored in the culture tab, the CEO of the firm held strong beliefs in favour of a highly market driven economic system.
Concern was voiced around any form of compulsion from the state with regard to economic activity, as was criticism of recent developments in the regulatory frameworks impacting upon financial services:
The clamping down that we’re seeing on speculative trading, and many, many banks have pulled out of trading important world commodities because they feel demonised…This is not a constructive move, it probably does far more harm than good.[DS_01]
This individual believed that the state should be focused on the development of a robust infrastructure that could support market growth. In relation to weather risk markets, he argued this state activity should be focused on public data provision and regulation of the indexes of data underlying the financial markets:
Okay, so there’s evidently the problem of what happens if that index is fiddled…So there’s recently a paper that was put out by one of the EU bodies talking about index providers and the scope of that specifically mentions weather data as an index, which of course is exactly what we do. So it’s quite possible that what we do becomes regulated. And we’d be delighted it speaks to the approach we take anyway of openness, consistency, quality, and auditability.[DS_01]
It was strongly felt that a publicly funded infrastructure – from roads to weather data – should be made available for exploitation by commercial interests free of charge:
The very first thing they should do is to stop tolls on bridges. And if they get that then open data will come [laughter].[DS_01]
Whilst some have called for the privatisation of parts of the UK’s public data infrastructure, including Trading Funds such as the Met Office, it was argued that this would not be appropriate:
Should it be privatised? The answer is no, I mean I think, I’m essentially a Thatcherite, but I think there are limits to the scope of privatisation. And whenever you see an infrastructure good I have great doubts that you can move that infrastructure’s good service to a private environment because of the artificial nature of creating competition…and provision of meteorological data in a nation is an infrastructure good in the same way, so therefore it’s a public good and should stay in the public.[DS_01]
It would never become open if it were privatised.[DS_01]
In order to overcome barriers to acquiring data, the firm looked to the USA and some European countries as exemplars of good open weather data practice:
So the obvious example is the US. The US realises that what it loses in not charging for data it gains through the increased economic activity that follows the free availability of that data. And so many companies in the US exist because that data is free, that you do not see in Europe.[DS_01]
Simply copy the model of other nations in Europe that have moved to an open policy. And that involves, firstly a policy decision that data should be freely available because the economic benefit transcends the modest amount of money that might be charged otherwise. But also it requires the practical side of it, data delivery, you know you need a way to get at it.[DS_01]
The firm only had limited awareness of the Met Office’s open data DataPoint service, believing it was too limited to meet their needs:
Yeah. I think I have come across that [Data Point], but that’s not something we use, no. Okay, so if that were to be broadened–[DS_01]
The firm’s knowledge of the government’s open data initiative was claimed to be limited:
It’s not an area I feel qualified to comment on…I don’t know enough about political policy in those areas.[DS_01]
Lobbying policy makers
Despite these strong political beliefs, the CEO stated that the firm did not engage in lobbying of policy makers:
No. No it’s not worth it…We’re not a big enough organisation to have any sway.[DS_01]
Industry level bodies such as the Weather Risk Management Association (WRMA) were also claimed to be relatively inactive in their lobbying efforts:
They [WRMA] have lobbied in Washington on a couple of areas, but not outside of the US.[DS_01]
Instead, the role of the WRMA was understood to be a form of “talking shop” for industry.
Lists of WRMA conference attendees for 2002 and 2003 found online, suggest this “talking shop” is made up of powerful and significant players within the global economy including: Chicago Mercantile Exchange, Goldman Sachs, Ernst & Young , Citigroup, PricewaterhouseCoopers, JP Morgan, Entergy-Koch Trading, re-insurance companies including Swiss Re and AXA Re, energy companies including BP Energy and Centrica, risk modellers such as Risk Management Solutions, and data suppliers to the weather markets like the firm we spoke to. Between them these organisations have had significant influence on UK policy makers in relation to their access to weather data.
In a 2008 report by the Department of Business and Regulatory Reform, it was reported that the WRMA and some of its conference attendees, alongside other powerful players in the financial markets, had influenced senior policy makers to push for “freer access to UK and other European weather data” [BERR]. There is also evidence that the WRMA has brought together its members with the Met Office to discuss some of the barriers they are facing within the weather risk industry.
More recently, senior policy makers have been keen to promote the needs of the weather risk industry in relation to the drive to release more weather data under the Open Data policy initiative.
The commercial side, you know the classic Peter Weiss analysis from about ten years ago on the weather markets in Europe against the weather markets in the US. Show a much larger and much more active private sector weather services market in the US. It also showed things like weather derivatives. Which was very much larger in the US than in the UK. [GOV_01]
The role of this public data in supporting a rapidly growing weather risk management industry underwriting financial risk management instruments, valued at approximately $8 billion. [Making Open Data Real consultation]
Support has also come from Frances Maude MP – Minister for the Cabinet Office – who has also spoken publicly about the connection between the decision to release Met Office data as part of the government’s Open Data initiative and efforts to boost the weather derivatives market:
The opportunities for enterprise won’t always be obvious. For example when some years ago the US released its public weather service one surprise result was a boost to the insurance industry. The data helped farmers to protect their profits leading to dramatic improvements in agricultural productivity. Today the weather derivatives market in the US alone is worth $3.5billion – that’s all powered by Big Data. Last autumn, this government announced that we would publish data from all 5,000 weather stations in the UK. [Francis Maude MP]
These comments demonstrate that there is a keenness within some policy and political circles to address the frustrations of the weather risk markets, and open publicly funded weather data in order to grow the UK’s weather derivatives market and make it more competitive with the US market.
Due to confidentiality reasons no archival sources are available for this node.
We’d like you to take a minute to reflect on some of the things you have read above and add a comment to the discussion below.
Here are some questions to get you started:
- How do the cultural values at this station compare to those of some of the other stations you have explored?
- How do you feel about weather data from places such as Sheffield Weston Park being used in the weather derivative markets?
- Should the Met Office open all its publicly funded data so that the company can use it without having to pay?
- How do you think the flow of data from private weather observation networks as set up by this firm, will differ from the flow of data from public and amateur observation networks?
- Did you notice that there were no photographs, audio or transcripts linked to this station? Why do you think that might be?
Once you have posted a comment follow the green line on the right to financial markets.
By posting a comment you confirm that you agree to our conditions of participation.