Ref: https://drcf.org.uk/publications/papers/insightssmartdataframeworks 

Published on 26 March 2026, this report from the Digital Regulation Cooperation Forum (DRCF) provides an international comparative analysis of Smart Data implementation models to inform the UK’s strategic approach under the Data (Use and Access) Act (DUAA) 2025.

Core focus

The report reviews how jurisdictions worldwide have approached consumer data portability and Smart Data ecosystems, drawing lessons to help the UK implement its cross-sectoral Smart Data ambitions. The DUAA 2025 empowers the Secretary of State to create sector-specific Smart Data schemes via secondary legislation, building on the Open Banking precedent. As of May 2025, one in five UK consumers and small businesses were actively using Open Banking services, up from one in seventeen in March 2021.

Three implementation models

The report categorises global Smart Data approaches into three main models:

1. Regulator-mandated 

Government-led, prescriptive legislation (e.g. Australia’s Consumer Data Right, Brazil’s Open Finance). Provides legal certainty and mandated participation, but risks high compliance costs, regulatory rigidity, and potential for stifling innovation. Australia’s experience is cited as a cautionary tale of disproportionate costs relative to uptake.

2. Market-facilitated  

Industry-driven, with regulators in a facilitative role (e.g. US, Japan, Hong Kong). Fosters innovation and commercial flexibility but suffers from inconsistent standards, fragmentation, slower incumbent adoption, and uncertain liability frameworks. The US is experiencing particular instability following legal challenges to its framework, which has been seen as lacking unambiguous positioning.

3. Public infrastructure-led 

Built on foundational national Digital Public Infrastructure such as digital identity and data exchange layers (e.g. Estonia’s X-Road, Singapore’s SGFinDex via Singpass). These solve interoperability and trust-by-design but require major upfront investment and sustained political commitment. Estonia saves an estimated 820+ years of working time annually through X-Road.

A fourth category, Hybrid and Transitioning, covers jurisdictions like India (Account Aggregator / DEPA framework, with over 100 million consents by 2024) and the UAE (transitioning from market-led to a centrally mandated Open Finance framework).

Key themes from the analysis

Governance

A central coordinating body is consistently identified as critical. Without it, sector-by-sector schemes under the DUAA risk creating new data silos rather than eliminating them.

Standards and interoperability

Cross-sector fragmentation is a recurring failure mode. The risk that different UK government departments managing different Smart Data schemes could produce divergent technical standards is highlighted as a significant concern.

Consumer consent journey

Legal compliance alone is insufficient. The quality of the consent user experience is as important as the legal principle. Brazil’s experience shows how broadly drafted consent forms led to data misuse and loss of public trust. India’s AA framework shows high consent numbers but low conversion due to friction. Singapore’s integration with Singpass is held up as a model of frictionless, trustworthy consent.

International alignment

The report recommends aligning with Gaia-X (the EU’s federated data infrastructure framework) to protect the UK’s data adequacy status (renewed until 2031), maintain access to EU digital markets, and reduce compliance burdens for internationally active businesses.

Anti-competitive risks

The report flags the possibility that Smart Data schemes could inadvertently facilitate tacit collusion if they enable easy monitoring of competitor pricing — a novel risk that warrants safeguards by design.

Considerations for the UK

The report proposes five strategic pillars:

1. A central Smart Data governance body to coordinate all schemes, set baseline standards, and prevent fragmentation.
2. Phased, use-case-driven rollout prioritising energy (for Net Zero), finance, and telecoms, learning iteratively before expanding.
3. Sector-tailored implementation models — acknowledging that some sectors suit market-led approaches while others need regulatory mandates, within a unified strategic framework.
4. Interoperability by design, integrating with the UK’s Digital Verification Services (DVS) trust framework (as envisioned in the DUAA) to create a trusted, unified consent architecture.
5. Clear secondary legislation covering liability frameworks, security standards, transparent cost-benefit processes, and safeguards against anti-competitive conduct.

Relevance to IB1

The DRCF report findings and recommendations are closely aligned with IB1’s existing work and strategic positioning.  

Central governance and Trust Frameworks

The report’s core recommendation, a ‘central’ body setting common baseline standards and ensuring cross-sectoral interoperability, closely mirrors the architecture IB1 has developed through its Trust Framework and Scheme architecture. The DRCF’s concern about fragmented sectoral schemes producing new data silos is precisely the problem IB1’s governance approach is designed to solve, specifically the interoperability between them.

Energy Smart Data and Perseus 

The report explicitly names energy as a priority sector for Smart Data designation, citing its alignment with Net Zero policy objectives. IB1’s Perseus programme, enabling permissioned energy data access for SME carbon reporting and green finance, is an existing example of the kind of sector-specific, use-case-driven scheme the report advocates.

Consent and liability frameworks

IB1 has developed detailed consent and liability standards across energy, water, finance, supply chains, and other sectors. The DRCF’s finding that the quality of the consent journey is as critical as the legal principle, and that clear liability apportionment is essential for industry confidence, directly reinforces the value of this work.

Interoperability across sectors

The report warns that without strong central coordination, different government departments could develop incompatible standards across schemes. IB1’s cross-sector interoperability work, including its engagement between Open Energy, Perseus and Open Banking, addresses this risk directly.

International standards alignment

The report recommends aligning with international frameworks such as Gaia-X. IB1’s engagement with cross-border data governance and its work across multiple jurisdictions is consistent with this direction.

The DRCF report diagnosis and recommendations map closely with the approach IB1 has built, and provides independent external validation of the strategic importance of IB1’s work.

This diagram shows how sustainability data can flow through a (NOVA-aligned) structured, governed market to enable trusted decisions. It begins with raw inputs and ends with real-world outcomes, and maps the various steps in-between, including how the sharing of data is governed. 

At its foundation are codified science, raw data, and algorithms. These come from science and standards bodies via sensors or related data providers. They are structured through harmonised (using existing standards) models, taxonomies, and ontologies, and stored or managed through technical systems such including databases, data lakes, and aggregation platforms. This layer provides the basic ingredients for sustainability intelligence.

In parallel, the policy and legal layer defines the rights, responsibilities, and protections that govern the underlying data. This includes statutory and voluntary rules, contract structures, redress mechanisms, liabilities, access management, terms and conditions, and pathways for conformance and dispute resolution. Policy frames what should be done and the legal frameworks encode these requirements into enforceable commitments for all participants. Together, these provide guardrails to ensure data sharing is lawful, ethical, and accountable, enabling organisations to operate with clarity and confidence.

Sitting above this layer are the technical infrastructure and metadata practices that ensure consistency, provenance, and interoperability. Principles, standards, and commitments guide how data is produced, maintained, and exchanged. 

Together, these feed into Scheme definitions: where multilateral licences define the full rules for publishing, accessing, and using data. The Scheme rules cover policy, legal, technical, and security needs, and are the basis for pre-competitive collaboration.

To drive adoption and make the Scheme rules operational, a data governance process brings together domain experts, implementation bodies, and market catalysts, policy, assurance practices, and verification. This layer ensures that the market can ‘go far together’.

To enable scale, Schemes can be delivered using Trust Frameworks (which address identity verification, assurance processes, onboarding, monitoring and enforcement). These enable safe participation for organisations across supply chains and sectors to share data effectively.

Above this, discovery and access interfaces allow both humans and machines to find, request, and use data. Portals and aggregators can surface what is available using standard web processes for search.

The next layer brings in tools for analysis. These turn data into insight, ranging from impact assessments to risk identification, forecasting to exposure analysis, vulnerability assessments to reporting. These are the analytical bridge between raw data and meaningful insights.

All these tools can feed into a wide range of applications: analytics platforms to reporting systems, dashboards to insights engines, or tools that support financial, operational, and policy decisions. 

Using these insights, organisations can design or support financial instruments, including loans, insurance, multimodal investment, transition bonds, or other products that depend on consistent, assurable evidence (the data). 

At the top are outcomes that deliver impact: market-facing solutions, actionable research and derisked innovation, leading to better decisions: targeted investment, effective transition planning and implementation, improved resilience and adaptation, disaster risk reduction, mitigation, and reconstruction.

Combined, this diagram shows how sustainability data can be made usable, dependable, and valuable to support society-wide decision-making.

Examples:

• Perseus
• Open Energy

The responsible use of LLMs includes understanding their energy use and the net zero implications associated with them. Both LLM model creation and usage are known to use a significant amount of energy and so this guide includes considerations for reducing energy consumption in LLM usage.

Consistent with our commitment to transparency, we have built upon the excellent work of mySociety’s AI Framework to produce a practical guide for internal use. We are publishing it so partners can understand our approach, adapt it for their own contexts, and engage with us where collaboration would be helpful. The guide addresses how to use LLMs responsibly as research tools while maintaining rigorous oversight and expert review.

Written by Paul Johnston

Read our full Guide to Using LLM’s here

Access TNFD’s full report here

What does good nature data look like?

Working across multidisciplinary stakeholder teams, we identified seven key principles for high-quality nature datasets, acknowledging that nature data should be:

• Transparent and reproducible
• Credible
• Accurate and complete
• Relevant and decision-useful
• Accessible and usable
• Legal, ethical, privacy protecting
• Networked and compatible

Our Process

TNFD provided a roadmap of use cases, which we used as a foundation to work from. From this, our user-needs led approach helped define who the data users were as we developed a set of recommendations and principles that could lay the groundwork for nature data that supports financial decision-making.

“The Taskforce is very clear that its focus and contribution is on addressing use cases specific to corporations and financial institutions”

TNFD: A roadmap for upgrading market access to decision-useful nature-related data

Findings from the pilot testing of proposed nature data principles:

After evaluating 40 existing datasets against these principles, we confirmed that none fully met the standards required for global reporting or investment use. This exposed a critical gap for the financial sector: the absence of reliable, comparable, and standardised nature data needed to direct capital toward nature-positive outcomes.

“We tested 40 datasets against robust criteria for decision-useful nature data and found that none of them fully met the principles required for global reporting and investment needs.” Lewis Just, Lead Researcher

Understanding the barriers of nature-based data

A key reason for this gap lies in the complexity of nature data. Unlike carbon, which can be measured through a single metric such as tonnes of CO₂, nature spans many interconnected systems including water, soil, species, forests, and many more. Each uses different metrics, standards, and methods of measurement, making it extremely difficult to compare results across regions, sectors, or reporting frameworks.

Without harmonisation, financial institutions face a fragmented landscape where nature-related risks are hard to identify or value, and progress toward biodiversity goals is difficult to measure.

IB1’s approach brings structure and clarity to this complexity: developing guiding principles to support better quality, comparable, and decision-ready nature data that can help direct financial flows toward positive environmental outcomes.

Nature loss isn’t just an environmental issue, it’s an economic one

“Reducing nature data barriers is key to enabling effective nature-related reporting and accelerating action to halt and reverse nature loss.” (TNFD)

The impact of nature-related financial risk is widespread and financial institutions currently lack the data needed to measure and manage their nature-related financial risks. For instance, UK Banks could avoid potential losses of 4–5 % of loan-book value if they’re able to anticipate and price risks more accurately. For governments, making nature risk visible could help to avert $2.7 trillion annual GDP loss by 2030.

To unlock the power of nature data and positively shift financial investment, we need better access to high-quality, trustworthy nature data. But access alone isn’t enough, how that data is shared and governed is equally important.

Balancing openness with responsible governance

At first glance, it seems there’s a simple solution: make all nature data open and accessible, right? Unfortunately, it’s not that simple. And, in fact, not all data should be open.

For instance, sharing the precise locations of endangered species could make them more vulnerable to poaching. For example, GPS tracking data intended to help conserve rhinos could be exploited by poachers to locate the animals. Similarly, satellite-derived coral reef maps — created to support conservation — could be misused by developers and industrial fishing fleets to identify and exploit those same ecosystems.

That’s why responsible governance is crucial. Data providers must retain control and ownership over their datasets to prevent misuse and ensure that data serves its intended purpose: protecting and restoring nature.

NOVA (Networked, Open, Verifiable Architecture)

Our NOVA principles were developed to guide us to outcomes that are interoperable, scalable, and aligned with existing standards, ensuring that data can be shared and used responsibly.

Applied through the lens of nature data, this means that sensitive information, such as species locations, is only accessible to those granted explicit permission, safeguarding wildlife while still enabling actionable insights for conservation and investment.

COP30 Brazil

IB1’s principles and recommendations for this project were formed to help TNFD understand what good nature data looks like, so that financial flows can shift towards investments that help, not harm, the planet.

With COP30 in Brazil now underway, the spotlight turns to a country that holds some of the world’s richest biodiversity and largest nature datasets. At its launch event in São Paulo on Thursday, November 6th, TNFD published its recommendations for upgrading the nature data value chain for market participants. This included a blueprint to govern, launch, operate and finance a Nature Data Public Facility (NDPF).

Wellcome has now published the workshop summary report, Realising the potential of a National Data Library, which highlights key insights and recommendations from participants.

Our contribution to the report drew on our use-case–driven approach to delivering data infrastructure for net zero. This perspective was reflected in the final publication:

“Defining a strong vision for the NDL is necessary not only for intended users and the public but also to inform its design. “A clear vision is a necessary design constraint” for the NDL, state Hardinges and colleagues. They recommend documenting the vision as a “clear, tightly-bound problem statement. This will help bridge the chasm between a high-level political vision and technical execution.” Similarities can be drawn from the development of Open Banking in the UK, where the public were consulted about the problem they wanted addressed with their banking before the financial data innovation was designed.”

Realising the potential of a national data library: The Wellcome Trust

Read the full report here

Perseus will automate access to assurable SME electricity smart meter data and its carbon intensity at the time and place of use. SMEs will be able to receive emissions reports generated from these data and share them, via reporting solutions, with banks or lenders to unlock green finance.

As described in the 2024 Plan, Perseus members have been working throughout the year to uncover and explore the needs of all stakeholders in the data flow and convert these into concrete actions. These have driven detailed work on legal, technical and user experience design, assurability and process development to enable an operational pilot.

Purpose of the pilot

June marked the end of the six-month pilot phase of Perseus, which was launched to gather feedback on the technical, legal and user experience aspects of Perseus. Part of an iterative process, the pilot focused on reducing friction and enhancing the way Perseus works for SMEs, energy data providers (EDPs), carbon accounting providers (CAPs) and financial service providers (FSPs). 

What did we do?

• Tested the technical, legal and contractual elements of the trust framework.
• Explored whether the onboarding documents were sufficient and useful to organisations joining the Perseus scheme. 
• Checked the assured data flow of electricity consumption and derived emissions between participants.
• Tested the user journey flow, design, product integration and legal agreements with a CAP, an EDP and a FSP. 

How did we do it?

The testing was conducted on a one-to-one basis with participants, and feedback was gathered during calls and/or shared via email.

What did we learn?

Several important lessons emerged:

• SME UX: The overall Perseus user journey could be improved through the addition of simple but effective elements, such as a progress bar, providing the user with support and guidance throughout their Perseus journey. 

• Clarity about who hosts what: Greater clarity is required on which stages of the Perseus journey are hosted by the EDP, the CAP and the FSP and to what extent the host can brand and integrate the Perseus journey stages.     

• Engagement: attempts to engage FSPs during the pilot were not as successful as expected. This was mainly due to key stakeholders’ own constraints, needing to focus on other internal projects and not being able to put their own SME clients forward for user testing. To address this, we reached out to other stakeholders in the wider ecosystem to understand their barriers to participation and engagement and how to overcome these.      

• Onboarding support and guidance: During the pilot we discovered that organisations had more questions when onboarding that we had anticipated. We have since updated the supporting documentation and produced clearer step-by-step guidance, with “readiness” checklists.

• Cloud providers: Members whose servers are hosted by cloud providers had difficulty using private server certificates. We have now modified the technical requirements to enable them to use the same public certificate authorities as they would for any website or web service.

• Liability and data retention: Members’ legal and compliance teams were comfortable joining the pilot because both the pilot agreement and associated data retention requirements were limited to the term of the pilot. This will inform timeframe considerations for production.

• Green Lending Market: An observation from testing the pilot is that the green lending market is not as active as we anticipated for SMEs seeking green loans. However, we anticipate this will evolve as the market develops and more incentives are introduced.

• Trust and clarity: The clarity of definition that the scheme provided proved vital in helping different actors understand their roles and responsibilities.

• Case studies: A lack of engagement equaled a lack of real-life Perseus case studies from participants. As a result, IB1 increased its efforts to conduct stakeholder interviews, focusing on how stakeholders perceive their role and value within Perseus, and how IB1 can support them in becoming early adopters that other organisations can learn from.

• Support and discussion channels: Support channels proved to be a challenge with some organisations finding it hard to use Slack. To provide more opportunities for focussed discussion, we introduced more targeted and frequent working group sessions, with shorter advisory group meetings tasked with decision-making rather than detailed discussion.

• Language: Insight from the pilot emphasised that the language used to describe Perseus needs to be kept simple and appeal to different users. There is a responsibility to encourage SMEs to adopt green finance to help the financial sector deliver net zero and to support CAPs in easily calculating assurable emissions data.

Next steps

Perseus has now progressed to the ‘sandbox’ stage of the project; which officially launched on Monday 29th September. The Sandbox provides identical trust services (Registry and Directory) to Perseus in production, and includes a reference EDP that provides synthetic smart meter consumption data. It can safely be used for development and testing. 

Learnings from the pilot phase have been incorporated into the onboarding documentation. Perseus members are receiving one-to-one support and guidance through the sandbox and are encouraged to showcase their case studies.

The 2025 Perseus Report will be launched at the end of this year and will provide more detail on the Pilot, Sandbox and launch phases.

The report explores the opportunities and challenges of sharing data across UK National Infrastructure Systems, with a particular focus on energy, water, and transport. Icebreaker One was proud to contribute to this crucial piece of work, undertaking a landscaping exercise to assess the current state of the art in data sharing in energy, water and transport infrastructure, particularly in regard to the support of research, with a focus on gathering evidence from the private and government sectors.

Read the full report here

You can find our landscape analysis on national infrastructure data sharing with researchers, which contributed to the above report, here

To full implement a data governance framework, our recommended next steps are:

• Validate assurance signals through engagement with UK arable farmers and their
• supply chains
• Continue to convene the Advisory Group to ensure the effective co-design and
• implementation of a market-wide solution
• Deliver a demonstrator that shows assurable data flowing from farm to bank
• Co-design implementation of a technical demonstrator that establishes the data
• infrastructure, including assurance metadata records

Below is the case study published by Digital Catapult entitled ‘Icebreaker One advances data sharing governance for farming supply chains’.

If you’re interested in this work please don’t hesitate to reach out via: icebreaking@ib1.org

You can also stay connected by joining IB1 Constellations

We have created an open document to explore what position we (all of us) should take in the governance of data around ‘ai’ (artificial intelligence, machine-learning, et al).

Click here to contribute to our open-to-comment paper
or Download this Word Doc version and email us research@ib1.org

Artificial intelligence (AI) is rapidly transforming how data is shared, managed, and leveraged across industries, but its impact on societal resilience—our collective ability to adapt, thrive, and safeguard against systemic risks—remains an open question. As AI is integrated into financial systems, energy infrastructure, and regulatory frameworks, ensuring it is ethical, transparent, and aligned with public interest is critical.

We wish to explore how AI can enhance resilience across economic, environmental, regulatory, digital, and social domains, and arrive at key principles that should guide its development. Embedding AI within open, interoperable, and responsible data ecosystems, we can drive sustainable growth, protect critical infrastructure, and empower communities. The question now is: how do we get this right?

“Every government agency, everywhere, is working on a new system that’ll solve all data problems and will be ready to use in 18-24 months… Except it will always be ready in 18-24 months.” – Whong’s Law

In December, we submitted a response to the Wellcome Trust and ESRC’s National Data Library Technical White Paper Challenge.

The Challenge was designed to surface concrete implementation options for the proposed National Data Library (NDL). We responded to:

• Support UK Government to look beyond the abstract ideas and aspirations projected onto the National Data Library thus far. 
• Propose a feasible implementation option, based on how we approach data infrastructure for net zero at Icebreaker One.
• Describe significant challenges to delivery regardless of technical role/architecture, and ways to address them.

We’re delighted that our white paper, Delivering an effective National Data Library, was selected for publication. And, as we present our paper this week, alongside four other submissions, we wanted to publish a summary along with some further thoughts. 

Building from a strong foundation

In our response, we argued that there’s an urgent need for UK Government – as well as the wider pool of organisations seeking to influence its design – to:

• Clarify the technical problem the National Data Library should address. We decided to focus our work on a National Data Library aimed at improving the discovery of government-held data for research. We described the fork in the road between research and pursuing improved operational data access, on the basis that the needs of researchers and operational data users are unlikely to be met by this singular intervention. This feels vindicated, given the release of the UK Government’s blueprint for modern digital government, which commits public sector organisations to do various things to improve the way data is used to deliver public services (including using standard APIs to exchange data). 
• Take inspiration from existing research data infrastructures, rather than the library metaphor. We pointed to 11 types and 39 examples of data infrastructures that already serve researchers’ needs, such as by generating new datasets for research (UK Biobank), unlocking access to data held by the private sector (Smart Data Research UK) and providing access to linked or combined datasets from multiple organisations (ONS Integrated Data Service). The National Data Library won’t exist in isolation and must be built with an understanding that it’ll be one node in a network of research data infrastructures. 
• Ensure the National Data Library adds something new, or improves or replaces what already exists. In addition to these existing research data infrastructures, there are numerous pan-government initiatives working to improve public sector data access. We argued for these initiatives – plus the Library’s intended users – to be engaged early, in order to understand the needs not met by existing efforts.

Our recommendation for the National Data Library

We recommended a simple, decentralised National Data Library to improve discovery of public sector data for research.

In the spirit of the Challenge, we laid out how it could maintain a searchable catalogue (or ‘index’, ‘registry’, ‘portal’; the name doesn’t really matter) of metadata harvested from across many public sector organisations. It wouldn’t copy or store any of the research data those organisations hold, but provide a service to help researchers find relevant datasets wherever they are on the web. It would look similar to Open Net Zero, which we’ve built to make net-zero data discoverable, accessible and usable. Open Net Zero currently indexes metadata on nearly 60,000 datasets from more than 400 organisations.

To do this, this National Data Library would harvest existing metadata from sources like data.gov.uk, gov.uk API Catalogue and Administrative Data Research UK Data Catalogue. It would also collaborate with public sector organisations to make available and harvest new metadata. It would point to datasets that are openly available, as well as datasets that researchers can work with under more restricted technical, legal and commercial conditions. While we described various technical challenges – including varying quality and machine-readability of metadata, and unclear data licensing terms – we think this represents a feasible version of the National Data Library.

There’s also some interesting things happening with metadata and data discovery that the Library could build on. This includes: new metadata formats emerging from the AI community (Croissant); ways to tag datasets with information about restrictions on their use (Data Use Ontology); and new tools that enable users to search across multiple data catalogues and within datasets on them (Open Data Deep Search, HerdingCats).

Data curation must begin with users and use cases

At Icebreaker One, we start with use cases and deliver data infrastructure to enable them. The starting point for Perseus, for example, was to automate high-quality sustainability reporting for every SME in the UK to enable them to access over $100bn of green finance. We’re focusing on unlocking half-hourly emissions data to do this, before we add further use cases and data types.

This is how complex data infrastructures are built. As John Wilbanks of the Astera Institute recently described, “You build a complex data system by answering five questions at a time, using a standards based approach. And then when you’ve answered twenty, you’ll have a functioning complex data system”. We must design data infrastructures for specific primary use and general secondary use.

We recommend the National Data Library takes a similar approach. While original research is needed to clarify its intended users and their needs, we pointed to existing evidence on use cases that the Library could address. This includes: 52 societal challenges hindered by a lack of coordinated data (DARE UK); high level areas of research interest (HM Treasury); and high-value datasets for reuse (European Commission).

The Library can’t be a vehicle for everything

The ambiguous language used around the National Data Library has caused confusion. The UK Government’s AI Opportunities Action Plan has recently described the National Data Library as “an enormous opportunity”. It says that ‘alongside’ the National Data Library, the UK Government should:

• “Run open calls to receive proposals from researchers and industry to propose new data sets”.
• “Rapidly identify at least 5 high-impact public datasets it will seek to make available to AI researchers and innovators”.
• “Establish a copyright-cleared British media asset training data set”.
• “Finance the creation of new high-value datasets that meet public sector, academia and startup needs”.

These could well be useful interventions to make to support the UK’s AI sector. But in order to give the National Data Library a necessary focus, the UK Government should be clear about the wide set of interventions it plans to make for the data economy vs the subset that will be delivered by the Library itself. 

The limits of our recommendation

While it’d help improve data discovery, we’re conscious that our recommended execution of the National Data Library wouldn’t move the dial when it comes to streamlining access to research data drawn/linked from multiple public sector organisations.

We compiled evidence that this is the significant challenge holding researchers back from working with public sector data in the UK, including from Administrative Data Research UK, the Public Administration and Constitutional Affairs Committee and the Office for Statistics Regulation.

It’s a very difficult problem to address. We pointed to Ben Goldacre’s 2022 review of the UK’s health data ecosystem, which identified a wide range of barriers to more effective linkage or combination of data for research in health alone. It described how individual data holders operate in silo, developing their own, bespoke approvals processes that make secure data linkage and access neigh on impossible.

The review made 30 detailed recommendations. We suggested that a similarly broad and deep set of interventions will be required to harmonise access to data for research across the whole of the public sector. We think this work is broader and deeper than is possible for the National Data Library to deliver, and that progress on streamlining access to research data will instead be driven by other actors. At a recent event held by Health Data Research UK, Sir Robert Chote, Chair of the UK Statistics Authority, described progress being made on a number of fronts. This included: the Office for National Statistics’s Integrated Data Service; reexamining the Five Safes framework to consider accrediting Safe Programmes rather than only Safe Projects; and the work of the Pan UK Data Governance Group.

Our view and recommendation has also been shaped by budget considerations. We anticipate that investment in the National Data Library will be modest and less permanent in comparison with other public data infrastructure: the ESRC alone has spent more than £200m in data collection, creation, curation and delivery.

Moving forward

Our recommendations for delivering an effective National Data Library have been shaped by our use-case driven approach to delivering data infrastructure for net zero. As we present our paper this week, we hope the UK Government will consider our suggestions as they develop further plans.

As a potential solution to this, the report acknowledges the value of Perseus. Recognising the project’s value in offering a reliable model for data sharing, the report views Perseus as a potential bridge for the property sector’s data gap. What’s more, with the Perseus pilot fast approaching, the project serves as an inspiring example, holding the potential to accelerate decarbonisation in the property sector and beyond.

The energy data deficit

The property sector’s data deficit is nuanced. While owner-occupiers are able to access energy consumption data with relative ease, the same can not be said for tenant and landlord, where the sharing of data is far more restricted. According to the report, this is due to privacy laws that limit data-sharing capabilities and make it difficult for landlords to gather and track energy usage effectively. The issue becomes especially problematic when considering that over half of UK commercial real estate and one third of the residential sector are tenanted.

The report explains that “within the energy sector, Ofgem has identified licensees who are subject to data sharing regulations, which does not typically include property owners. This results in a situation where there is currently no legislation requiring the exchange of energy consumption data between property owners and occupiers.”

Perseus

With this in mind, the report indicates the need for a framework that facilitates data sharing, looking to Perseus as a prime example. If Perseus’s model of facilitating data sharing between SME and bank could be applied to the real estate sector, then data from tenants could be securely shared with landlords, ensuring energy consumption is monitored and decarbonisation efforts are tracked.

“(Perseus) demonstrates that setting up a governance framework and automated processes to build trust between stakeholders and to facilitate data sharing is possible from the perspective of the finance and banking industry. A similar framework could also be applied to the real estate sector to facilitate data sharing between tenants and their landlords, while ensuring that the data will be shared securely and used for the sole intended purpose of enabling the net zero transition.”

The review also looks at initiatives that are instrumental in our transition to net zero, with project Perseus being one of these. Perseus aims to lead the market down a path of enhanced data access, streamlined reporting and improved compliance with standards. By ensuring that quality data on SME emissions is readily available, Perseus seeks to address a critical gap and remove one of the many obstacles we face on the path to a low-carbon economy.

Financial flows

Spotlighted in the review is the UK’s strength in sustainable finance and its ability to accelerate our path towards a low-carbon economy. Despite this, the trajectory of transition finance towards our net zero future is being held back by high emitting sectors like heavy industry and agriculture, both of which face a significant investment gap.

And, while private capital is seen as a key catalyst for transition, real economic policy and public finance will be required, providing the market with confidence, mitigating risk, and allowing private finance to flow. As the report echoes, “Finance will ultimately flow to where markets believe there is future profitability”.

Data to unlock transition finance

Data – underpinned by assurance and verification – is set to play a central role in our transition plans, particularly when informing financial decisions. The report points out that the past five years have seen an increasing focus on bringing the right data and disclosure to the market. This, for the time being, has been centred around securing information from large companies. In the UK, for example, mandatory TCFD*-aligned disclosures apply primarily to listed companies and large private businesses.

And, while some smaller businesses have started to adopt disclosures voluntarily, a substantial data gap persists for SMEs, who often form the backbone of larger companies’ supply chains. The importance of measuring SME emissions should not be understated, considering they account for around half of the UK’s private sector turnover and business emissions. And yet, the review notes that a coherent, ambitious and forward-looking strategy for addressing issues related to SME data provision and access is currently lacking from the UK Government’s framework: “So far, there has been generally limited focus on how to support and encourage SMEs to develop transition strategies and disclose information in a structured way.”

Project Perseus

Icebreaker One has long recognised the burdensome process of carbon reporting that SMEs face as well as their integral role in our transition to net zero. Through one of our flagship projects, Perseus, we will help unlock access to finance that reduces emissions faster. We aim to do this by automating sustainability reporting for every SME business in the UK.

By automating primary data access we will ease the challenge small businesses face in finding and sharing the data they need to produce sustainability reports. This allows banks, for whom better data means lower risk, to offer SME clients favourable conditions, providing an incentive for SMEs to report their emissions.

“Perseus has the potential to provide personalised net zero recommendations for SMEs and match these recommendations with financial products to increase access to transition finance. SME participation is secured through engagement with existing suppliers (banks and accounting solutions), and data sharing permission is managed in a manner similar to open banking.”

*The Financial Stability Board has announced that the work of the TCFD has been completed, with the ISSB Standards marking the ‘culmination of the work of the TCFD’.

Building from principles tried and tested under Open Banking, the Roadmap identifies needs and opportunities to build a wide range of interoperable smart data schemes. It particularly highlights the banking, finance, energy and road fuels, telecommunications, transport, retail, and home-buying sectors for further development.

These changes represent a clear direction for the future of UK smart data, presenting a vision of decentralised, interoperable schemes based on common, open standards and supported by strong governance. These are changes that we strongly welcome and will continue to support.

Our work shows that smart data schemes which align with the principles of the Government’s Roadmap are both possible and successful. Our work also exemplifies why smart data schemes are vital in the transition to net zero (e.g. Perseus). 

Around half of UK business emissions are produced by 5 million SMEs. Perseus is creating new rules and processes to unlock access to green finance. It does this by making automated sustainability reporting possible for every SME business in the UK and, with trusted controls, empowering SMEs to share reporting data with banks. This will help unlock £billions that the banks are looking to deploy to enable a faster net zero transition, helping SMEs to action affordable, personalised, data-enabled insights on how best to improve efficiency and decarbonise their business. 

Perseus exemplifies how smart data schemes can support innovation and advance a progressive vision for financial services in the journey to net zero. Our work also highlights two opportunities for value creation which will multiply as the UK’s smart data economy grows. 

Firstly, Perseus creates both economic and environmental value for businesses by connecting smart data flows across sectors. We are starting with energy data and this  will expand as the project connects with other sectors holding key data such as water, transport, agriculture, and the built environment. 

Secondly, Perseus demonstrates the value of an interoperable approach to smart data sharing in the transition to net zero. Use cases which elevate the connected power of data to address environmental change cannot be neatly divided along sector lines. Without change, the potential of data and digitalisation to support positive environmental action will continue to hit arbitrary limits.

In both cases, value creation hinges on the instigation of decentralised, interoperable and trusted smart data schemes at scale across the economy. 

At IB1 we go one step further, choosing to operate with radical transparency about our own journey towards implementation. We openly publish and licence as many aspects of our process, research, and resources as possible (including success stories and the bumps in the road!). We further prioritise collaboration and knowledge exchange, encouraging other organisations to use our resources for collective benefit in developing the UK’s smart and environmental data sharing ecosystems. 

To get involved in Perseus, please contact us at perseus@ib1.org or join online today at ib1.org/join/perseus.

If you would like to discuss other programmes, sector or place-based work, please email partners@ib1.org or join as an IB1 member at ib1.org/join. 

Photo credit: Jerry Kavan on Unsplash

Background

Icebreaker One, in partnership with SSEN Transmission, IBM and Palantir, is working on the NIMBUS (Network Innovation and Meteorology to BUild for Sustainability) project, which aims to accelerate the transition to net zero by prolonging the life of energy assets through the introduction of granular weather data sources into the asset risk modelling methodologies. The NIMBUS project is funded by the Ofgem Strategic Innovation Fund (SIF), the SIF is delivered in partnership with Innovate UK.

In this consultation we present key findings from NIMBUS regarding the asset data sharing practices in the UK energy sector and the use of granular weather data as part of the CBRM (Condition Based Risk Management) methodology. 

We seek your input to ensure that we capture a wide range of expertise and views that reflect the data sharing landscape relevant to the NIMBUS use case, specifically pertaining to data on energy asset management. The proposals  presented in this consultation draw from prior NIMBUS project work incorporating desk research, Advisory Groups convening various members of TSO and DNO organisations, and stakeholder interviews to evaluate the current landscape of asset data sharing and the approaches to CBRM in the sector.

Important Details about the Consultation below:

• The consultation is open until 5pm on 12 March 2024. 
• Please submit your responses to: julija.hansena@ib1.org
• In your response , please indicate whether you are happy for your response to be published openly, attributed to you and/or your organisation. 
• If confidentiality is requested, your submission will be anonymised when Icebreaker One publishes their response. Please email Julija to discuss if you have specific anonymity requirements. 
• Comments are particularly encouraged from all transmission system operators (TSOs) and distribution network operators (DNOs) in the energy sector, however responses from utilities with similar assets such as water, rail and telecommunications are also welcomed.
  

Proposal

Through analysis of data gathered via Advisory Groups, stakeholder interviews and desk research, we have identified the following approaches to asset data sharing and the use of CBRM in the energy sector:

Statement 1

The sector is generally embracing the concepts of open data and data sharing, especially since the publication of Ofgem’s Data Best Practice Guidance which promotes data sharing and reuse. Many TSOs and DNOs have created their own open data portals (e.g. SSEN-T Open Data Portal, SSEN-D Data Portal), however there is still a noticeable lack of shared data standards, methodologies and tools that govern asset data sharing in this sector.

Questions about Statement 1: 

1. What is your organisation’s experience of sharing asset data?
2. Are there any data standards, methodologies or tools that are used in your organisation to facilitate data sharing and publication of this type of data? 

Statement 2

CBRM is a widely utilised methodology across the energy sector to facilitate asset risk management and reporting. However, asset managers report that it is somewhat limited in its approach and may well benefit from integrating weather data experienced locally by assets into the asset risk modelling process. Project NIMBUS aims to address this by assisting to integrate granular weather data into asset risk assessment methodologies, with the aim of optimising asset inspection schedules and subsequently reducing asset degradation and the associated costs.

Questions about Statement 2: 

3. In your organisation, are weather conditions experienced by assets currently integrated into the asset risk assessment methodologies?
4. If so, have you been able to measure and quantify the impact this has had on your asset inspection and maintenance cycles and the associated costs?

Statement 3

TSOs and DNOs handle large volumes of data of various levels of sensitivities, which need to be handled appropriately. In order to assist with data security, as well as enable data sharing between organisations, a framework like data sensitivity classes can be applied. For the NIMBUS use case, SSEN-T Open Data Portal makes use of a data triage process which is applied to all SSEN-T datasets prior to publication, with the results of the triage processes being published here. The data sensitivity classification framework that forms part of this triage process is closely aligned with IB1’s data sensitivity classes, whilst simultaneously adopting the classes to fit the needs of the business.

Questions about Statement 3: 

5. Are there any existing data triage processes or frameworks that aid in assessing data sensitivity levels of asset data in your organisation? 
6. Are they based on any existing processes or frameworks?

How can you help?

We are seeking responses to the 6 questions listed in the ‘Proposal’ section above. It should take no more than 30 minutes to respond. The consultation is open until 5pm UK on 12 March 2024 and responses are encouraged from all transmission system operators and distribution network operators in the energy sector, as well as adjacent organisations working on matters relevant to the NIMBUS use case.

To submit your responses, or if you have any queries please email: research@ib1.org

The report shares insights into market trends, costs, and the challenges SMEs face in carbon reporting. It aims to help businesses navigate carbon reporting and support their transition to net zero. It includes extensive research, covering over 270 carbon reporting solutions as well as in depth results of surveys and interviews of SMEs and solution providers. You can find the full long list of carbon reporting solutions here.

Access our comprehensive report here

The report features:

• Comprehensive list of solutions on offer: A detailed overview of 270 carbon reporting tools available, segmented into 10 solution types
• Guidance on choosing the right solution: Advice tailored to the specific needs and scale of a business, helping SMEs navigate through the plethora of options.
• Key considerations: Insights into what SMEs should look for in a provider, such as cost-effectiveness, ease of use, data accuracy, and scope of emission coverage.
• Learnings from other businesses: Real-world examples and insights directly from SMEs who have navigated and implemented carbon reporting solutions.
• Future trends and developments: Analysis on emerging trends in carbon reporting and how they might impact SMEs

How to get involved:

• Sign up to our newsletter to receive the report and updates from IB1
• Learn more about Perseus, the programme enabling automated emissions reporting for every SME in the UK.
• Get in touch to contribute and connect: research@ib1.org

Perseus home page

The reality, even in countries considered to be ‘water-rich’ – such as the UK – is that future supplies are by no means guaranteed. The impact of climate change on the world’s water must not be underestimated – whether that’s the effect of increasingly hot and dry spells on water reserves or the damage and pollution that’s the result of storm-driven flash flooding. 

The environmental impact of our reliance on fossil fuels to generate energy is increasingly well-understood by consumers and industry alike. However there is much more limited awareness of the links between water consumption and climate change. 

Lost water is also a huge issue. Ofwat estimates that, in the UK, a fifth of treated water running through pipes is currently lost to leakage. Substantial resources must be deployed in finding and stopping major leaks, and many smaller leaks may go undetected, not only wasting water but potentially adding to water customer bills.

Meanwhile, modern usage of the UK’s water pipes and other assets simply do not match the expectations that were in place at the time the infrastructure was built, thanks to increased pressure from population growth, changing customer expectations and, again, climate change. 

Simply trying to build our way out of the problem is not a viable solution. It’s just too costly – both from a financial and a carbon perspective – time-consuming, and resource-intensive. There aren’t enough people or material resources in the UK to replace the entire UK infrastructure at the pace required to keep up with changing demands. 

So what’s the solution?

Instead, we have to think innovatively. Another solution is to make better use of existing data. Getting as many eyes as possible on as much water data as possible can facilitate diversity of thought and enable the innovation required, at speed and at scale. Stakeholders that Icebreaker One has spoken to have highlighted a wide range of benefits that could be achieved with better access to data. To highlight just a few use cases, better access to water data could help:

• Identify the rivers most in need of clean-up initiatives
• Predict asset failure based on analysis of previous faults
• Drive awareness among consumers of the link between water consumption and the environment, and encourage behaviour change.

In some cases, data alone might do away with the need for pricier interventions. In others, it will help target resources to the projects and places that are most in need.  

What water data is already available?

There is no shortage of water data. A search for ‘water’ on Icebreaker One’s Open Net Zero data directory returns several thousand data sources linked to water. Data Owners include national and international regulatory bodies – including the UK’s Department for Environment, Food & Rural Affairs (DEFRA), academic and research organisations, and, of course, water companies. 

However, it’s fair to say that while water companies hold a cornucopia of data, the amount that we’ve so far been able to surface via Open Net Zero is fairly limited. As outlined in Ofwat’s 2023 report on Open data in the water industry, to date the majority of water companies have published only a small amount of open water data (Wessex Water’s ‘Water Marketplace’ being one notable exception). The data that is available is often in formats that restrict its use: for example, ‘data’ is often published in the form of web interfaces or maps, rather than the underlying data being published via an API (ideally) or .xls/.csv file, for example. And, even when all water companies publish the same data, users typically have to seek it out on multiple, individual water company websites. 

While there are good reasons for water companies to keep some water data behind closed doors – data protection or national security concerns, for example – there’s plenty of other data that could be made openly, and usefully, available for analysis by industry, academics, campaigners, innovators and more. 

What needs to happen?

While water companies have been exploring innovative uses of data for several years, it’s typically been in the form of hackathons. This approach is exciting and has delivered real benefits, but it doesn’t work at scale. What’s needed is a hub for open water industry data, published in a central location, backed by clear data standards and licences, and supported by a Trust Framework. 

Cue the Stream initiative, an Ofwat-backed collaboration between 15 water companies in England and Wales, alongside partner organisations including Aimii, Icebreaker One, the ODI, and Sia Partners, who have come together to unlock the potential of water industry data. 

As Melissa Tallack, Co-Lead for Stream at Northumbrian Water, puts it: “If we were to put a label on what Stream is, it’s probably a data institution, helping people publish and consume water data for the express purpose of improving transparency, trust and innovation to deliver benefit to customers, society, the environment and the economy. It’s part of the national data sharing infrastructure of the UK and that’s why we always have interoperability at front-of-mind.

“It’s not the path of least resistance to get there. Going together is not as easy as each company going alone. But the 15 Water Companies who are currently a part of Stream are not doing all this for their convenience now. They are doing this for the long-term gain, to create a future path of least resistance for publishers and users alike so that data challenges never stand in the way of building transparency, trust and innovation.”

Through collaborating to make water company data openly accessible in a coordinated, consistent, standardised manner, Stream aims to essentially enable an uber-hack, without time constraints, that allows anyone, anywhere, to use that data to help solve the challenges faced by the water sector, industry as a whole, society, and the environment. 

To add a final quote from Melissa: “Can you imagine the potential impact that kind of scale could achieve in finding new insights and new ways to solve problems and generate value?”

Stream phase one

Over the last year, Stream partners – including Icebreaker One – have been working together to launch the first few Stream datasets. You can see the early fruits of our labours in the Beta launch of the Stream platform, here. 

But this is only the beginning. Over the coming months, water companies involved in Stream will be working to get more datasets ready for publication, so you can expect bigger and better things in the new year and beyond. 

Get involved in Stream
The data published on Stream is only as valuable as the challenges it helps to solve. We rely on potential users of water company data to tell us what problems they are trying to solve, and how water data can help. You can follow Stream’s progress on its LinkedIn page, or help us to shape the next phases of Stream by joining one of our advisory groups. Find out more at ib1.org/stream. 

Please Share this page with your peers and networks, and join us in this collective effort to understand the breadth and depth of the carbon reporting solutions market.

Following our open consultation, we are launching a survey to help the British Business Bank (BBB) understand the landscape of carbon reporting (and, to some extent, ESG) solutions available to UK SMEs today. 

Carbon reporting is critical to any net-zero strategy, enabling organisations to measure, manage and monitor their carbon emissions effectively. With the array of reporting solutions available, it can be daunting for Small and Medium Enterprises (SMEs) to select the most suitable one for their unique needs. 

The survey aims to gather invaluable insights into the diverse landscape of carbon reporting solutions and gather feedback from SMEs regarding their experiences with these solutions. This will help the BBB in addressing and supporting market needs.

Participation in this survey is open to all UK SMEs who have searched for or used a carbon reporting solution and any provider of a carbon reporting solution. Your input will be vital in shaping the future of sustainable business practices. 

We encourage all interested to participate in the survey by 15th November 2023. All responses will be treated in the strictest confidence. We will publish aggregate results of this work openly for the benefit of the community. 

“We’d like to develop a standard for our data”

As always, the thing about standards is that there are so many to choose from. And, in the context of the work we do at Icebreaker One, these span domains and sectors. 

So, instead of developing standards, we help people come together (through our Icebreaking process) to choose what works for their use cases, then help align and implement rules that mandate their utilisation, as part of a Trust Framework. Based on patterns in those implementations we look for what is repeatable and can scale.

Below is a primer on some of the (many) ways to think about the words ‘data’ and ‘standards’.

• Classification Standards define a systematic way of categorizing data or objects based on predefined criteria, facilitating consistency and organization.
• Data File Format Standards specify the structure and encoding of files to ensure compatibility and interoperability between different software applications and platforms.
• Data Format Standards encompass guidelines for structuring data, including the arrangement of data elements and their representation, ensuring uniformity in data representation.
• Data Inflow Standards establish protocols for the reception and integration of data from external sources into an organization’s systems or databases.
• Data Management Standards provide best practices and guidelines for the effective handling, storage, and maintenance of data throughout its lifecycle.
• Data Organization Standards outline principles for arranging data elements, records, or datasets in a structured and logical manner for easy access and retrieval.
• Data Provider Standards set expectations and requirements for entities or systems that supply data to ensure data quality, consistency, and reliability.
• Data Sharing Standards define protocols and rules for securely exchanging data between organizations or systems, often emphasizing privacy, security, and consent.
• Geospatial Data Standards govern the format, content, and interoperability of geographic or spatial data, enabling consistent representation and analysis of location-based information.
• Governance Standards encompass a framework of policies, processes, and controls that guide and oversee data-related activities to ensure compliance, quality, and accountability.
• Linking and Matching Standards provide guidelines for identifying and connecting related data records or entities, often used in data integration and deduplication processes.
• Metadata Standards specify how descriptive information about data, such as data definitions, source details, and usage instructions, should be structured and formatted.
• Standardized Variable Standards define conventions for naming, measuring, and representing data variables to enhance consistency and comparability in data analysis and reporting.

As you can see, these embody many questions. It is easy to get lost in trying to define these in a generic manner (although there are often holistic models that can be utilised) and it can turn into quite an academic exercise. Instead, focusing on use cases helps drive towards what can be made consistent out of the chaos.

Here’s another useful view on creating consistency before standards: https://blog.ldodds.com/2023/09/18/consistency-before-standards/

And this ODI guide: https://standards.theodi.org/introduction/types-of-open-standards-for-data/

Full archive of videos: https://www.nasonline.org/programs/scientific-forum/researcher-access.html

Icebreaker One’s presentation begins at 46m 44s.

Session 4:  Privately Held Data: Opportunities, Challenges, and Lessons for Researchers from The National Academies on Vimeo.

Session 4:  Privately Held Data: Opportunities, Challenges, and Lessons for Researchers

This session will explore data collected by private companies, which often contain useful insights that can help alleviate major societal challenges including climate change, healthcare, food security, and disinformation. Accessing this data, however, can be costly, controversial, and unreliable. Solving these challenges could help unlock vast amounts of data for researchers, providing novel insights and better guidance for policymakers. With lessons from social media platforms, mobile health applications, and retailers, this session will highlight best practices for accessing data held by private companies and consider solutions to the challenges of commercial sensitivity, data protection, and emergency preparedness. Attendees can expect to leave with an understanding of the value to be gained from privately held data, topical data security challenges, and an overview of partnership-enhancing technologies.

• Gina Neff, Professor and Executive Director, Minderoo Centre for Technology & Democracy, University of Cambridge
• Henry T. (Hank) Greely, Professor, Stanford School of Medicine, Director, Center for Law and the Biosciences, Stanford University
• Cyndi Grossman, Senior Director, Biogen Digital Health
• Uyi Stewart, Chief Data and Technology Officer, Data.org
• Gavin Starks, Founder and CEO, Icebreaker One