Written by Technical Team | Last updated 23.07.2026 | 18 minute read
The UK energy system is becoming increasingly dependent on data that must move securely between organisations. Network operators need information from distributed energy resources. Flexibility providers need access to network requirements, dispatch instructions and settlement data. Asset owners need to exchange technical characteristics, availability and performance information. Retailers, market operators, aggregators, technology providers and public bodies all depend on data held beyond their organisational boundaries.
Historically, these exchanges have been delivered through bilateral integrations, bespoke file transfers, manually negotiated agreements and systems designed for individual programmes. This approach can work when a small number of established organisations exchange a limited set of predictable datasets. It becomes far less effective when thousands of participants, millions of connected devices and rapidly changing energy services must interact. Every new bilateral connection introduces another combination of technical interfaces, security controls, legal terms, data definitions and operational responsibilities.
The UK’s emerging Energy Data Sharing Infrastructure, or DSI, is intended to address this problem. It should not be understood as a central database into which the industry uploads all its data. Its more important function is to provide a common way for organisations to describe, discover, request, authorise and exchange data while retaining control of their own systems. This requires a federated technical architecture built around Data Preparation Nodes, shared trust services, machine-readable agreements and well-governed data products.
A centralised data platform can appear attractive because it promises one place to store, govern and analyse information. In practice, a national energy data lake would create substantial challenges. It would concentrate operational, commercial and security risk; require a central body to assume responsibility for data it did not create; duplicate large volumes of information already held in authoritative systems; and struggle to support the different latency, sovereignty, retention and assurance requirements found across the sector. Planning datasets, asset registers, half-hourly measurements, real-time telemetry and commercially sensitive forecasts cannot all be governed as though they were the same type of information.
The more scalable model separates the control plane from the data plane. The control plane provides common capabilities for identity, participant discovery, metadata, access requests, policy, assurance and agreement management. The data plane carries the actual information between authorised participants. Under this model, a central service may know that a particular data product exists, who owns it, what standards it follows and under what conditions it can be accessed, but the underlying data can remain within the provider’s environment until it is requested. When access is approved, it can move directly between participating organisations rather than being routed through a permanent central repository. This separation reduces unnecessary duplication, preserves accountability and allows each organisation to protect its operational boundaries.
The Data Preparation Node, or DPN, is the most important component at the participating organisation’s boundary. It is sometimes described as a gateway, but that term is too narrow. A well-designed DPN is simultaneously an integration layer, security enforcement point, data-product runtime, policy client and observability boundary. It connects the organisation’s internal systems to the wider DSI without requiring those systems to expose themselves directly.
This distinction is particularly important for energy infrastructure. Data may originate in SCADA platforms, advanced distribution management systems, meter data stores, energy trading systems, asset-management platforms, geographical information systems, forecasting services or third-party cloud applications. These systems have different security classifications, interface capabilities and operational lifecycles. Some were never designed for external connectivity. A DPN should therefore shield source systems from external consumers and provide a controlled anti-corruption layer between internal representations and shared industry products.
For critical operational environments, the DPN should normally sit outside the core operational technology zone. Data can be extracted through an integration boundary, demilitarised zone or staged publication service, depending on its sensitivity and required latency. Planning data may be replicated on a schedule, while near-real-time measurements may be published through event streams. Highly sensitive control-system data may require one-way transfer mechanisms, additional aggregation or strict temporal delays. Participation in the DSI should never require an external party to establish direct access to a SCADA network, historian or control-room system.
A modular DPN architecture allows an organisation to use common interfaces while selecting implementation components appropriate to its scale. A smaller flexibility provider may consume a managed DPN service. A network operator may integrate DPN functions into an existing enterprise integration platform. A technology provider may run a cloud-native deployment across several availability zones. The important requirement is not that every participant uses identical software, but that each node implements the required behaviours consistently.
A production-grade DPN is likely to require capabilities including:
The data pipeline inside the DPN should be designed as a repeatable product pipeline rather than a collection of one-off integration scripts. Extraction, mapping, validation, publication and delivery need clear contracts between stages. A producer pipeline may read from an internal source, convert records into an agreed schema, validate them, add provenance and publish them to an outbound broker. A consumer pipeline may receive an authorised stream, verify its schema and origin, apply local transformations and route it to a destination system. These stages should be independently testable and observable.
Current reference implementations demonstrate how container orchestration, event brokers, secrets management and certificate automation can be assembled to create such a node. Technologies such as Kubernetes, Kafka, Vault-style secret stores and automated certificate managers are useful implementation options, but they are not the architectural objective. The DSI should avoid becoming dependent on one cloud provider, deployment model or messaging product. A DPN implemented with alternative technologies should remain interoperable as long as it meets the shared protocols, security controls, metadata requirements and behavioural contracts.
Schema validation alone is not sufficient. A payload can be technically valid while being operationally misleading. A power measurement may conform to the expected numeric type but use the wrong unit. A location may be syntactically correct but refer to a retired asset. A forecast may arrive after the decision window in which it was useful. A meter reading may be complete but based on an estimation method unsuitable for settlement. Validation therefore needs several layers: structural conformance, reference-data integrity, semantic consistency, temporal relevance and use-case-specific quality thresholds.
Quality results should be treated as operational data in their own right. Rather than attaching a permanent badge declaring a product to be “high quality”, a DPN should generate measurable indicators over time. Completeness, timeliness, validity, duplication, consistency and availability can be tracked as time series. Consumers can then understand whether a product is currently meeting its stated service level, while producers can identify degradation before it creates downstream failures. The infrastructure may expose these indicators, but responsibility for correcting source data remains with the organisation that owns the underlying process.
Resilience must also be designed explicitly. Energy data flows will encounter expired certificates, unavailable catalogues, backpressure, duplicate events, late files, out-of-order telemetry, partial transfers, schema changes and temporary network failures. Pipelines should be idempotent where possible, preserve checkpoints, support replay and route unprocessable messages to controlled exception queues. Authorisation information should be cached safely so that an avoidable central-service interruption does not terminate every permitted exchange. At the same time, revocations and emergency policy changes must propagate rapidly enough to prevent continued access after trust has been withdrawn.
Technical connectivity does not create trust. Two organisations may be able to exchange data over an encrypted connection while still disagreeing about who is entitled to use it, for what purpose, for how long, under which licence and with what onward-sharing restrictions. The DSI therefore requires a trust framework that joins organisational governance to technical enforcement.
At the centre of this control plane is the Data Sharing Mechanism, or DSM. Its role is not to process every data packet. It provides shared services through which participants can be identified, products can be discovered, requests can be evaluated and agreements can be established. This may include a participant directory, metadata catalogue, certificate authority, standards registry, licence repository, product templates, assurance information and records of active sharing agreements. The DSM can coordinate trust without becoming the permanent route through which all data travels.
Onboarding into the trust framework should establish more than a username and password. An organisation must be verified as a legitimate participant, linked to accountable legal and operational contacts and assigned the appropriate roles. Its DPNs, services and workloads then require technical identities. Certificates can establish mutually authenticated connections between nodes, while signed tokens can communicate more granular permissions. Mutual TLS answers the question, “Which trusted system is connecting?” A properly scoped token can answer, “Which product, action, audience and permission is this system currently authorised to use?”
A Zero Trust design assumes that network location alone is not proof of legitimacy. Every connection should be authenticated, every request authorised and every permission constrained to the minimum necessary scope. Controls should consider participant identity, workload identity, product version, requested operation, token audience, agreement status and relevant contextual conditions. Certificate renewal, token expiry, key rotation and revocation must be automated. Administrative actions should require stronger controls than routine data exchange, and the compromise of one DPN component should not grant unrestricted access to every product the organisation publishes.
The most significant architectural step is to make sharing agreements machine readable. A traditional agreement may sit in a document-management system while engineers separately configure firewalls, API gateways, user accounts and message topics. This creates a gap between what the parties agreed and what the technology actually enforces. In a DSI, the approved terms should be translated into a Technical Sharing Agreement that can be consumed by DPN policy components.
A Technical Sharing Agreement should identify the producer, consumer, data product and permitted version. It should describe authorised operations, allowed purposes, geographic or asset scope, valid dates, request limits, delivery method, retention conditions, audit requirements and restrictions on onward sharing. It may also include redaction rules, quality expectations, service levels and termination conditions. The agreement should be signed or otherwise protected against tampering, distributed to the relevant nodes and versioned throughout its lifecycle.
This enables access policy to be evaluated consistently. A request can be checked against the active agreement rather than relying on a hard-coded rule written months earlier by a separate integration team. Where a product has standard access conditions, approval may be automated. Where commercial sensitivity, personal data or critical infrastructure risk is involved, the request can enter a negotiated workflow. In both cases, the final decision is converted into a policy artefact that the participating nodes can enforce.
Federated enforcement is essential to resilience. A DPN should periodically synchronise agreements, trust anchors, licences and policy updates from shared services, then make local decisions during data exchange. Requiring a real-time call to a central authorisation server for every message would introduce latency and a sector-wide point of failure. Local enforcement must nevertheless include controlled expiry and revocation mechanisms so that stale authorisation cannot persist indefinitely.
Audit information should record the decisions surrounding an exchange without creating another uncontrolled pool of sensitive data. Relevant events include product registration, access requests, approvals, policy updates, certificate issuance, successful connections, denied requests, significant validation failures and agreement revocations. Logs need consistent identifiers so an incident can be traced across organisational boundaries. They should also distinguish between metadata about an exchange and the data payload itself; centralised logging of complete payloads could undermine the federated privacy and sovereignty benefits the DSI is intended to provide.
The trust framework must also coexist with other forms of authority. Organisation-to-organisation access under the DSI is not the same as consumer consent. A company may be technically accredited to participate in the infrastructure but still require a consumer’s permission before accessing household-level information. Future interoperability with consumer consent services should therefore allow consent signals to become an additional policy condition rather than treating DSI membership as universal entitlement. Regulatory duties, commercial contracts, security classifications and individual consent may all contribute to the final authorisation decision.
The DSI will only create value if participants publish information as usable data products. A data product is not simply a table, file or API response. It is a governed package of data, meaning, delivery behaviour, quality information, ownership and support. It should be designed for a defined consumer need and operated through a lifecycle.
This product approach is necessary because discoverability alone does not produce interoperability. A catalogue containing thousands of vaguely named datasets may technically increase transparency while leaving consumers unable to judge whether any item is suitable. A useful catalogue entry must explain what the product represents, its spatial and temporal coverage, the assets or market processes it relates to, the method by which it was produced, the available delivery interfaces and the conditions under which it may be accessed.
Metadata should combine general catalogue standards with energy-specific semantics. Common fields such as title, description, publisher, licence and update frequency are important, but they do not fully describe an operational energy product. Consumers may need to know the measurement point, network level, aggregation method, unit, direction convention, settlement period, time zone, forecast horizon, confidence interval, asset identifier scheme and applicable industry profile. Without these details, two products that appear similar in a catalogue may behave very differently when integrated.
A robust data product contract should define:
The schema registry should be treated as an active engineering capability rather than a document archive. Schemas need immutable versions, compatibility rules and automated conformance testing. Producers should know whether a proposed change is backward compatible, and consumers should be able to test their integrations before a new version is activated. Breaking changes may require parallel operation, migration windows and explicit consumer acceptance. A field should not silently change meaning because its data type remains the same.
Semantic interoperability requires similar discipline. The energy sector already has standards covering network models, market messages, metering, asset information, device communication and operational control. The DSI should not attempt to replace every established domain standard with one universal model. Instead, it should support well-defined profiles, mappings and product-specific schemas that reuse existing standards where they are appropriate. A distribution-network planning product may use a different representation from a real-time flexibility dispatch product, provided each is clearly specified and the relationship between identifiers and concepts is understood.
The concept of a minimum operable standard is useful here. Requiring every participant to transform all information into a theoretically complete sector-wide model would raise the barrier to entry and delay adoption. Allowing every producer to publish an arbitrary structure would recreate the bilateral integration problem. The practical middle ground is to define the minimum semantic, structural, temporal, security and quality requirements needed for a specific use case to operate reliably. Those requirements can then evolve as adoption grows.
Data-product design should begin with a real operational decision. For example, a flexibility provider may need to determine whether an asset is eligible for a service, submit availability, receive a dispatch instruction and provide evidence for settlement. Each stage depends on particular products, identifiers, timing constraints and quality rules. Designing those products around the end-to-end workflow reveals requirements that would be missed by simply exposing existing database tables.
The producer should also consider how much information the consumer genuinely needs. Data minimisation is not only a privacy requirement; it reduces attack surface, bandwidth, processing cost and commercial risk. A DPN can generate different product views from the same source, applying aggregation, redaction or filtering according to the agreement. A planning user may need aggregated demand by area, while an authorised operational participant may require more granular data for a specific set of assets. Product variants should be intentional and governed, not improvised at the point of delivery.
Operational ownership is as important as technical publication. Every product needs an accountable owner who can approve changes, investigate quality issues and communicate incidents. A product that is available through an API but has no support model is not infrastructure-grade. Service levels should reflect the value and criticality of the use case. A monthly planning dataset can tolerate a different recovery time from a dispatch stream used in near-real-time operation.
Observability should connect the technical and product perspectives. Infrastructure teams need latency, CPU, memory, queue depth and error rates. Product owners need completeness, freshness, consumer usage, failed validations and service-level attainment. Security teams need denied access attempts, certificate anomalies and policy changes. These views should share consistent identifiers so that a delayed product can be traced from the consumer-facing symptom through the DPN pipeline to the underlying source.
Provenance provides the final layer of confidence. Consumers should be able to understand where the data originated, which transformations were applied, when it was processed and which product version produced it. For derived products, lineage may include multiple source systems, analytical models and reference datasets. This does not require exposing every internal implementation detail, but it should provide enough information to assess fitness for purpose, reproduce significant results and investigate disputed outcomes.
Organisations should not wait for every national governance question and technical component to be finalised before improving their readiness. The architecture is deliberately intended to evolve, and the most durable investments are capabilities that remain valuable regardless of implementation detail: explicit data ownership, secure integration boundaries, consistent metadata, automated validation, certificate management, policy enforcement and well-operated data products.
The first step should be selecting a specific cross-organisational use case rather than launching a general “DSI programme”. A suitable use case has identifiable producers and consumers, a measurable operational benefit and a genuine need for repeated data exchange. The organisation can then map the end-to-end journey, identify authoritative sources, define the minimum operable product standard and determine the trust conditions that must be enforced.
Architecture discovery should include both enterprise and operational technology. Teams need to understand where source data is created, how it crosses security zones, which identifiers are authoritative, where transformations currently occur and what failure modes are acceptable. Threat modelling should assess compromised credentials, malicious consumers, replayed messages, unauthorised onward sharing, schema poisoning, data exfiltration and denial-of-service scenarios. These risks should influence product granularity, network placement, authentication strength, monitoring and retention.
A DPN pilot should be built as a repeatable platform capability. It should demonstrate producer and consumer flows, local catalogue management, schema validation, secure node-to-node communication, policy synchronisation and operational monitoring. Automated deployment and testing are essential. Infrastructure configuration, product schemas, transformations and policy rules should be version controlled. Test environments should support representative data volumes, certificate rollover, breaking schema changes, access revocation and degraded connectivity.
Organisations should also decide which DPN capabilities they will own and which they may consume as managed services. Large regulated companies may require deep integration with existing identity, security operations and integration platforms. Smaller participants may need a low-friction hosted model. Both approaches can coexist if the external interfaces and assurance requirements remain consistent. Portability should be considered from the outset so that a managed service does not become an opaque dependency or restrict future participation.
The operating model must connect engineering, governance and product management. Security teams should not define controls without understanding the operational workflow. Data owners should not publish products without lifecycle and support commitments. Legal teams should not negotiate terms that cannot be represented in enforceable policy. Platform teams should not select technologies without considering participant accessibility and interoperability. The DSI is socio-technical infrastructure, so its success depends on these disciplines working as one delivery system.
Cost should also be treated as an architectural quality. High-volume telemetry, long retention periods and duplicated processing can make an apparently simple exchange expensive. Product teams should define the necessary granularity, frequency and retention based on the use case. Event-driven delivery may be preferable to repeated polling. Data should not be transformed multiple times when a governed reusable product can serve several consumers. Infrastructure metrics should be connected to product usage so that cost can be attributed and optimised without reducing service quality.
The long-term objective is not merely to connect more systems. It is to reduce the effort required to establish each new trusted relationship. A successful DSI should make the second and third data-sharing use cases materially easier than the first because identities, node capabilities, governance patterns, product templates and assurance controls can be reused. This is the difference between another integration programme and genuine shared infrastructure.
For UK energy organisations, the immediate engineering priority is therefore to create clean, secure and observable organisational boundaries. Internal complexity does not need to disappear, but it must be prevented from leaking into every external relationship. DPNs provide the mechanism for doing this. The DSM and trust framework provide common rules and verifiable identities. Data products provide stable units of meaning and operational value.
Together, these components can support an energy system in which information is exchanged more quickly without abandoning security, accountability or organisational control. The architecture will continue to mature, but its central direction is already clear: decentralised data exchange, shared trust, machine-enforceable governance and products designed for reuse. Organisations that build these capabilities now will be better placed to participate in flexibility markets, digital network operations, whole-system planning and the next generation of data-enabled energy services.
Is your team looking for help with UK energy technical architecture? Click the button below.
Get in touch