The Integration Gap

Last issue I ended with a promise: take the five properties a settlement record must have and hold them up against a real baseline methodology to see where they break.

I picked the UK Demand Flexibility Service. Two winters of live operation, well-documented by NESO, with a publicly acknowledged baseline problem that triggered a methodology change in November 2023. Documented, operational, known failure mode already in the record — a clean stress test.

I’m not a grid engineer. The DFS team built something that worked. This isn’t a critique — it’s an observation about what the record lacks, and a mapping to where those gaps have already been solved elsewhere.

What DFS is

Aggregators commit to demand reductions in specific 30-minute settlement periods. NESO (National Energy System Operator) dispatches the service and pays out based on delivered reduction against a baseline — what the participant would have consumed without the event. The baseline is calculated from the same settlement periods across the ten most recent comparable non-event weekdays. NESO runs the calculation. The aggregator receives a settlement statement.

Everything that follows is about the space between the calculation and the statement.

Property 1 — Provenance

NESO runs the baseline calculation in its settlement systems and sends aggregators a statement showing delivered MWh and payment. The aggregator receives the number, not the chain. Which ten days were selected? What adjustments applied? Against which version of metering data? That lineage lives inside NESO’s systems. An aggregator disputing their settlement has to ask the party whose number they’re disputing to show the working.

The engineering answer already exists: SLSA-style signed attestations — SLSA (Supply-chain Levels for Software Artifacts) is a framework from software supply chain security. A provenance record isn’t the artefact — it’s a signed statement of the artefact’s lineage, published alongside it. Anyone can inspect it without asking the builder. The energy equivalent: a signed attestation per settlement event, carrying the input chain as a first-class field. The number doesn’t change. What changes is whether anyone can verify where it came from.

Property 2 — Reproducibility

The baseline methodology is documented in NESO’s published service specifications. Any sufficiently motivated aggregator can read the rules. But reading the rules and reproducing the result are different things. The spec says “ten most recent comparable non-event weekdays” — but “comparable” and “non-event” both require interpretive choices that resolve in NESO’s implementation, not in the document. Two independent implementations of the same spec will converge on most cases and diverge on edge cases. When they diverge, you can’t tell whether the disagreement is in the inputs or the interpretation.

The principle applies beyond energy: a specification that can’t be implemented to convergence by independent parties isn’t a specification, it’s a description. For flexibility baselines, this means a machine-readable methodology definition where every tie-break is explicit, every adjustment deterministic, and the spec validated against a shared test set. Software build systems formalised this as the hermetic build pattern used in CI/CD (Continuous Integration/Continuous Delivery) pipelines: same inputs plus same environment equals same output, verifiable by any runner. Energy settlement needs the same discipline applied to methodology definitions. The gap isn’t technical. It’s a willingness to specify tightly enough to permit verification.

Property 3 — Ruleset versioning

In November 2023, NESO updated the baseline methodology to add a cap that limited the effect of participants over-consuming before events to inflate their baseline. The update was legitimate — a genuine gaming vector, addressed. But settlement records from October 2023 don’t carry a reference to which methodology version produced them. A dispute raised in early 2024 — after the update — involves two different sets of rules, but neither the settlement statement nor the dispute process names which was in effect. “Which baseline methodology was running on 14 October 2023?” is an answerable question that today requires archaeology, not a lookup.

Package management solved this years ago. In npm or Maven, `@1.2.3` refers to exactly one published artefact, permanently, regardless of what `@latest` has since become. The equivalent for settlement: each methodology version gets a stable identifier, settlement records carry it as a mandatory field, old versions remain permanently accessible. The November 2023 DFS update would have been a new version identifier. Pre-update settlements would carry the old one. Explicit, not inferred.

Property 4 — Tamper-evidence

The BSC (Balancing and Settlement Code) defines multiple settlement run windows — initial, interim, final. Corrections happen within that structure. When they do, corrected figures replace prior ones in the system of record. But none of those windows create a public, linked audit trail between the original figure and the corrected one. A corrected settlement and an original look identical from outside. Corrections are legitimate — errors happen. The question isn’t whether corrections should be allowed. It’s whether they should be silent and unlinkable.

Certificate Transparency, RFC 9162, solved the equivalent problem in internet security. Certificate Authorities can issue and update certificates — but every issuance is appended to a public, append-only Merkle tree. You can’t silently amend a prior entry. Corrections are new log entries that reference what they supersede. The DFS equivalent: settlement records written once to an append-only log; corrections are new entries with a reason, a reference, and a new signature. The original is never deleted — it’s made obsolete in a way the audit trail can see. Not novel infrastructure. The Merkle log pattern has been in production PKI (Public Key Infrastructure) systems for over a decade.

Property 5 — Independent verifiability

If an aggregator wants to verify their settlement, they ask NESO. “Ask us, we’ll check it for you” is middleware, not infrastructure. It works at low volume with high trust. Redispatch 2.0 and the ACER (Agency for the Cooperation of Energy Regulators) Demand Response Network Code are pointing toward a market that will involve millions of assets. Verification-by-asking doesn’t scale.

W3C Verifiable Credentials (VC) and selective disclosure are built for exactly this. A settlement record structured as a signed VC can be verified by any party with the public key and access to the referenced methodology version — no call to NESO required. Worth flagging: dispatch profiles are commercially sensitive. Selective disclosure — proving a specific field satisfies a constraint without revealing others — is designed into the VC model. A verification architecture that requires full disclosure won’t be adopted. One that doesn’t can be.

Looking at different markets

DFS is a UK mechanism, but the five gaps above aren’t a UK problem. They surface in any flexibility market built on a calculated baseline — which is most of them. Two markets show why the geography matters.

The Alpine cross-border case makes the provenance problem structural rather than incidental. Pumped hydro operators in Switzerland and Austria are among Europe’s largest flexibility providers, dispatched by grid operators across the CH-DE and AT-DE interconnectors. Settlement for a single event can involve Swissgrid, APG, and a German TSO — each running their own baseline calculation against their own metering data, under their own methodology version. The five gaps don’t just apply independently per jurisdiction. They compound. A provenance dispute that involves three separate systems of record, three methodologies with no shared versioning, and no common audit trail isn’t three times harder to resolve — it’s qualitatively different. Cross-border flexibility at scale requires the integration layer DFS hasn’t needed to solve yet.

The Gulf states present the opposite condition: not retrofit, but design choice. The UAE and its neighbours are building demand response and flexibility infrastructure now — as part of grid modernisation programmes that haven’t yet locked in their settlement architectures. Abu Dhabi and Dubai are expanding smart grid investment; the GCC interconnection project is adding cross-border balancing requirements. Markets at this stage have something DFS didn’t have in 2022: they can read the post-event reviews, observe where the gaps opened, and specify the verification layer before the first settlement dispute lands. That’s an advantage with a limited window. Once settlement contracts are signed and systems are procured, the retrofit problem arrives.

The integration gap is easier to close at the beginning than after the first contested baseline.

What this means for the market

These are not research problems. SLSA attestations ship in GitHub Actions today. Append-only Merkle logs run Certificate Transparency at global scale. W3C VC 2.0 is a published standard with multiple production implementations. The gap is an integration gap, not an invention gap.

My read: the commercial pressure will come from aggregators and software vendors tired of losing settlement disputes they should win — not from system operators, who have less incentive to move first. But either path leads to the same infrastructure. The same way VPP operators started asking for audit logs before regulators required them, the market will pull this into existence before the regulation mandates it.

What’s next

The next question this series has been circling without naming: if the primitives exist and the incentives are forming, what does a reference design actually look like? Not a product — a minimal architecture. The fewest moving parts that satisfy all five properties.

That’s Issue #007.

Sources and further reading

NESO Demand Flexibility Service — service specifications and post-event reviews: neso.energy/industry-information/balancing-services/demand-flexibility-service-dfs

UK Parliament POST Note 715 (February 2024) — DFS evaluation including baseline methodology update: researchbriefings.files.parliament.uk/documents/POST-PN-0715/POST-PN-0715.pdf

SLSA framework — provenance and attestation model: slsa.dev

Certificate Transparency, RFC 9162 — append-only Merkle log model: datatracker.ietf.org/doc/html/rfc9162

W3C Verifiable Credentials Data Model 2.0 — independent verifiability and selective disclosure: w3.org/TR/vc-data-model-2.0

Energy UK response to DFS Extended Balancing Reserve consultation (August 2024): energy-uk.org.uk

Grid Signal covers the intersection of grid modernisation, distributed trust, and digital infrastructure for energy. No hype — just structural insight for practitioners.

If a colleague would find this useful, forward it.