1. Executive summary (Appendix)
Modernising energy data access means both understanding and responding to the fact that every asset, system, organisation and network in energy (and beyond) will be producers and consumers of data.
Systems will increase in complexity: they are not only being digitalised, they are becoming data-driven. The growth in data connections will be exponential as the market matures.
MEDA must implement an architecture which can scale in data-type, volume and connectivity, across use-cases, organisational and logistical boundaries, sectors and jurisdictions. It must deliver this in a secure, safe, robust and adaptable environment which addresses governance.
Data users are diverse: from asset managers to DNOs to consumers. Our research has confirmed:
- User needs are diverse, encompassing thousands of organisations, customers and society as a whole.
- There is no ‘single data platform’ approach that will (or should) address all needs.
- There is a material risk to implementation unless governance is addressed.
Centralised data architectures have not scaled effectively in any sector. Interviewees expressed the need for a clear roadmap to transition from a fragmented data landscape to a robust, decentralised, federated data infrastructure. They believed that “there can be no single platform for all data and use-cases” and “there will be significant barriers to adoption around the centralisation of commercial data”. With Presumed Open as a guiding principle we must also apply the precautionary principle to innovation to address potential unintended consequences (e.g. unexpected monopolies).
The architectural approach to developing domain-specific platforms, hubs, analytic networks, asset registries, catalogues, systems maps and so on requires a shift in thinking from ‘push’ to ‘pull’—as websites enable search engines to find and index them, a distributed architecture creates a dynamic market between data suppliers and consumers. This enables markets for many solutions including platforms, apps and related services, while control is retained at the organisational level.
We heard from dozens of experts that they see “no viable alternative” than to “address the upstream needs of data supply through an open, decentralised architecture with strong governance”. Further, “should that exist it would enable a multitude of solutions to emerge”.
Fortunately, we have 30-years evidence of the most successful information architecture in history: the web. We believe the energy sector must now embrace a ‘web of data’ approach.
We propose phase 2 develops a Governance Platform in concert with all projects to establish a common approach to data-sharing, a cohesive minimum viable product and viable market architecture.
2. User needs
Extensive stakeholder engagement across the energy and associated sectors (47 interviews and 4 webinars, reaching over 200 people) confirmed four clear macro trends in Energy:
- Decentralisation
- Decarbonisation
- Digital transformation
- Democratisation
As a result, data is growing exponentially at the grid edge in a rapidly evolving and increasingly complex demand and supply landscape. Interviewees reflected that a common data architecture is only one part of the jigsaw.
This data is needed to balance the network and support innovative decarbonisation initiatives. Current infrastructure is fragmented and does not easily allow data to be discovered or shared.
For the industry to fully benefit, data is urgently needed to inform decisions that efficiently decarbonise electricity, heating, cooling and transport. Data must be available in an accessible, discoverable, easily consumable way which is cost-effective, secure and trusted, with a common governance platform and processes across parties.
Our Phase 1 interviews revealed dozens of use-cases which demonstrate benefit from improved energy data sharing, and we have chosen to focus on two representative cases which illustrate different aspects of user needs while touching on the three pillars of: asset registration, a data catalogue and digital system mapping.
1: Flexibility services to balance the grid
System Operators need to balance supply and demand in geographical regions as Distributed Energy Resources (DERs) at the grid edge come online. Renewables must be supported, alongside electrification of heat and transport without expensive grid reinforcement.The ESO and DSOs require access to both existing and new data, critical for integrating these DERs. Further, as energy aggregators and flexibility services play an increasingly important role in balancing the grid, they will need to access data to aggregate capacity, identify when flexibility is provided and settle contracts. Data will inform efficient, cost-effective, timely decision making, balancing the grid in a world of high renewable penetration.
2: Localised resource allocation
Local authorities must deliver a local heat and energy efficiency strategy, and meet net-zero by 2050. They need to understand DER technology performance, optimum mix and cost in order to successfully decarbonise and alleviate fuel poverty. In particular, they need to provide DNOs with trusted-data demonstrating DERs can be effectively managed without risk, essential to overcome DNO refusal or high charges for network enhancement. Detailed information from DNOs on LV grid capacity will further streamline the grid connection process. Finally, they must demonstrate informed energy decisions to citizens relating to budget and resource allocation.
3. Constraints
Stakeholder engagement highlighted many issues which must be addressed in the delivery of widely accessible shared energy data. These include, but are not limited to:
Commercial
- Lack of incentives for regulated utilities to invest, develop and support open data systems
- Business cases for sharing specific data, or understanding of its value, are not obvious
- Investment needs to protect the national interest, security, consumer rights and/or commercial sensitivities
Regulatory/legal
- Uncertain, evolving (cross-sector) regulatory landscape regarding data
- Compliance concerns over GDPR and privacy law
- Management of consent, risks and liabilities for sharing data
Technical
- Lack of a secure, controllable environment supporting effective governance
- Discovery of appropriate data according to requirements
- Access to data in a standardised and structured way
- Data quality and trust in origination
- Management of structural change and local variation across industry
Specific prototyping constraints
Creation of a Governance Platform to support a web of energy data requires demonstrable capabilities for data discovery, a trusted access process, and standards. This will be achieved via a “directory” (as in Open Banking). Identified constraints and their mitigation strategies include:
- Developing industry understanding and support for this approach. This will be achieved through a highly collaborative, consensus-based and consultative process, shaped by a deep understanding of user needs and the benefits that must be delivered (many interviewees expressed their desire to engage this process).
- Discovery of appropriate data is currently challenging for open data and shared data. The prototype directory will ensure data will be signposted, discoverable and accessible.
- Data providers must be able to qualify requests for data and recognise that the data recipient is a trusted party. The prototype directory will demonstrate how only trusted parties can access shared (restricted) data through a pre-emptive licencing approach.
- Datasets do not follow a common standard. A prototype standard will demonstrate how this can be approached.
- Integration, while machine-to-machine, requires support. Access to technical support or commercial assistance will be demonstrated. Contact details for appropriate individuals will be enabled with a contact management system.
- The end-user benefits must be clearly demonstrated. An example will be delivered through a mocked-up service that shows how the specific end-user needs are addressed.
- Integration and technical capability will be required. This will be provided through API integration experts contracted to undertake the technical aspects.
- The end-to-end process, including the ability to provide pre-emptive licencing, will be demonstrated through a mocked-up user journey & supported with annotated wireframes.