Resilience and Recovery by Design: The Architect's Share of Recoverability

Every control in this part assumes that something can go wrong and that the organisation will then need to get back to a trusted state. Operators run that return under pressure. Whether it can be run at all is decided earlier, by architecture: whether the system can be rebuilt from definitions nobody has altered, whether the AI system and the security AI that watches it can be restored to behaviour shown to be sound, whether a fallback exists that people can still operate, and whether anyone has proved this recently. The architect's share is therefore design, verification and requirements. The operators' share is execution.

The chapter concerns two of the four shifts. Shift 1, from documents to code applies because a recovery plan in a document is a promise, while a recovery definition in version control can be redeployed, tested and diffed. Shift 3, from periodic to continuous, at machine speed applies because adversaries and agents act in minutes, so that a recovery capability tested once a year describes an estate that no longer exists. The chapter builds on the conclusion of Architecture as Code (Chapter 9B) that compromised components are redeployed from known-good definitions rather than repaired (167, Nordqvist, 2025), and on the conclusion of AI Red Team, Chapters 22 to 24, that recoverability becomes a measured and continuously rehearsed property (170, Nordqvist, 2026). Chapter 10 owns how fast the estate can notice, decide and act; this chapter owns what the estate can return to once it has acted.

The recoverability loop: known-good definitions, rehearsal and chaos experiment, finding, pull request and reporting, with the degraded mode drawn beside the AI system

Figure 11.1: The recoverability loop, with the degraded mode and the reversibility test drawn beside the AI system.

Recoverability as a property

A recovery plan states an intention. Recoverability is the measured ability to carry it out. NIST's guide to cybersecurity event recovery treats recovery as something to be planned before an event, tested and improved over time, with metrics that show whether the capability is getting better (36, Bartock et al., 2016), and NIST's contingency planning guide sets out the business impact analysis from which recovery objectives are derived (194, Swanson et al., 2010). The CSF 2.0 expresses the same idea as outcomes: backups are created, protected, maintained and tested (PR.DS-11), mechanisms achieve resilience requirements in normal and adverse situations (PR.IR-03), and the integrity of backups and restoration assets is verified before use (RC.RP-03) (29, NIST, 2024). Each is met by a design decision and shown by a test.

Perrow's analysis of normal accidents explains why the design cannot wait for the failure: in systems that are interactively complex and tightly coupled, some failures arise from interactions nobody anticipated (79, Perrow, 1984). Cook adds that such systems run in a degraded state as a matter of course (196, Cook, 1998). Nygard's stability patterns, such as bulkheads, timeouts and circuit breakers, limit how far a failure spreads (84, Nygard, 2018), and Adkins and colleagues note that the ability to roll back must be weighed against the security cost of returning to a version with known weaknesses (69, Adkins et al., 2020). The architect applies these patterns to the whole service, including the pipeline that deploys it, the identity provider it trusts, the model it calls and the security tooling that watches it.

Objectives as requirements

A recovery time objective and a recovery point objective are requirements on the architecture, not figures in a plan. When the business impact analysis says that the benefits-triage system must be back within a day, the architect asks what that implies: whether the definitions are in version control, whether the hosting provider can supply replacement capacity in the time left, whether people can operate the process meanwhile, and which test would show it. An objective that no design can meet is a finding for the risk owner, who funds a faster design or accepts a longer objective on the record.

The architect's share and the operators' share

The table separates the two shares.

Concern The architect's share The operators' share
Recovery objectives Derive requirements from the business impact analysis and state what each choice implies for the design Meet the objectives and report the measured times
Known-good definitions Specify what must be in version control, signed and held outside the blast radius Maintain the definitions and run the redeployment
AI systems and security AI Specify the components that need a known-good source and the evaluation that proves behaviour Execute rollback, re-evaluate and approve return to service
Degraded modes Design the lower autonomy level and the manual fallback, and the test that shows they work Switch to them and staff them
Verification Design the experiment catalogue and its safety boundaries, and read the results Run the experiments and the drills
Supplier dependence Write the recoverability requirements into contracts and tender annexes Manage the supplier during an incident
Obligations Translate DORA and NIS 2 duties into design constraints and evidence Declare, notify and communicate

The recovery run itself is the subject of AI Red Team, Chapters 22 and 24 (170, Nordqvist, 2026), and is not repeated here.

Known-good definitions

Architecture as Code (Chapter 9B) concluded that a compromised component is redeployed from known-good definitions rather than repaired in place (167, Nordqvist, 2025). The conclusion has a precondition that is easy to state and hard to meet: the definitions must be complete, trustworthy and reachable when the estate is not. The architect specifies four properties.

Completeness. Anything configured by hand is outside the definition. The test is whether a service can be built in an empty environment from the repository alone, and a manual step found in a drill is a finding to be fixed in the definition, not the runbook.

Integrity and provenance. A definition is known-good because it can be shown to be the one that was reviewed and approved. SLSA describes levels of assurance about how an artefact was built (51, OpenSSF, n.d.), and Sigstore provides signing, verification and provenance (52, Sigstore, n.d.). The architect requires that every artefact recovery uses, including images, infrastructure modules, model artefacts, prompts and policy bundles, carries a signature and provenance that the deployment path verifies, which corresponds to RC.RP-03 (29, NIST, 2024).

Separation. The definitions, signing keys and means of redeployment live outside the blast radius of the incident they answer. AI Platform and Security, Chapter 19, concluded that control artefacts sit outside an agent's reach (169, Nordqvist, 2026), and the rule holds for recovery artefacts. DORA Article 12 reaches the same conclusion for data, requiring restoration onto systems physically and logically segregated from the source (9, European Union, 2022).

Order. Recovery has a dependency order that includes what recovery tooling itself depends on: the identity provider that authenticates the pipeline, the secrets store, the artefact registry and, increasingly, the security tooling. The architect draws the order as a graph in version control and checks it for cycles, such as an identity provider whose definition is deployed by a pipeline that needs the identity provider.

Security tooling can itself become the thing to be recovered. CrowdStrike's analysis of the July 2024 incident describes a content update to a security sensor that crashed the Windows hosts running it (195, CrowdStrike, 2024). The book's own conclusion is that a control with deep reach is also a single point of failure, and that the recovery design needs a way to remove or roll back such a control without the cooperation of the system it protects.

Recovering AI systems

An AI system behaves according to a combination of components: a model version, a system prompt, guardrail and gateway policies, a retrieval index, tool connections and, for an agent, an identity, a permission manifest, a memory and a queue of unfinished work. AI Red Team, Chapter 23, lists how each is restored and concluded that a recovered AI system is one shown to behave as it did before the incident, not merely one that is running (170, Nordqvist, 2026). This chapter sets the architectural requirements that make that procedure possible.

Every component has a known-good source, or is designed not to need one. The evidence bundle of the last approved release, which records model digest, prompt commit, policy versions, index build and evaluation results, is the bill of materials for recovery (169, Nordqvist, 2026). Two components have no such source: an agent's memory is runtime state, and its credentials must be replaced, not restored. The architect designs agents so that neither matters, with memory that holds only what can be re-entered through the normal input path and with short-lived credentials issued per task.

The retrieval index has a rebuild path. The index is the component most likely to carry poisoned content and least likely to have a snapshot anyone inspected. NIST's taxonomy distinguishes attacks on training data and models from attacks through retrieved content, and records that mitigations for many remain incomplete (30, Vassilev et al., 2025). The architect requires a deterministic build from authoritative sources, with provenance recorded for every document, so that a rebuilt index can be compared with the suspect one.

Provider-held components are pinned and replaceable. Where a provider supplies the model, the architect requires a pinned version, a statement of what the provider restores and evidences, and a tested alternative that the organisation can host. This is the recovery face of the conclusion of AI Platform and Security, Chapter 13, that sovereignty means being able to continue if a provider fails (169, Nordqvist, 2026).

Behaviour is evaluated before return. Matching digests are necessary but not sufficient. The golden datasets and red-team suite are restorable from the same bundle, the incident scenario is added to them, and a named owner approves return on the result. For a high-risk AI system there is one constraint on sequence: Article 73(6) of the AI Act prevents a provider from investigating in a way that alters the system before informing the competent authorities (1, European Union, 2024). The platform must therefore be able to suspend a system and preserve the compromised versions before any rollback overwrites them.

Recovering the security AI itself

The security AI that triages alerts, investigates incidents or proposes containment is an AI system with the same components. It is also a distinct recovery target: it may be the subject of the incident, and it is most needed exactly when the platform under it is being restored. The dependence of security operations on a security AI that fails together with the systems it watches is an expected concern, supported by the growing use of security AI and by the precedent of security tooling that caused an outage, but not yet documented at scale.

The architect designs for it in three ways. The security AI depends on as little as possible of what it defends, ideally running on capacity and identities the incident does not share. Its loss produces a defined degraded mode, treated in the next section. And its recovery is verified by the same experiments as everything else, including its withdrawal during an exercise.

Degraded modes and the fallback that eroded

A stop path that works is only half of the design. When the stop path of Chapter 10 halts a Mode 5 agent, or a provider withdraws a model, the work still has to be done. A degraded mode is a predefined configuration the organisation can switch to without a change process: the same system at a lower autonomy level, with fewer functions, or a manual procedure operated by people. AI Red Team, Chapter 23, defined the pattern for security AI, with a first mode at Level 1 Assisted in which people execute every action and a second, manual mode with the AI out of the loop (170, Nordqvist, 2026). The architect extends it to every AI system whose function the organisation cannot afford to lose, and records the mode, its trigger, the people who hold the switch and the evidence that it works.

How the fallback erodes

Bainbridge observed that automation leaves human operators with the tasks the automation cannot do while removing the practice through which they would have kept the skill to do them (106, Bainbridge, 1983). AI Threat, Chapter 26, applied the observation to organisations and concluded that small delegations erode skill, process, capacity, interfaces and memory until the documented fallback no longer exists (171, Nordqvist, 2026). The five erosions can be checked.

  • Skill. The people who can perform the function have left or never learned it.
  • Process. The manual procedure refers to retired forms, screens or queues.
  • Capacity. The function was staffed for the automated volume.
  • Interfaces. Surrounding systems now expect the AI's output format.
  • Memory. Nobody remembers why the manual steps exist, so they are skipped.

AI Threat proposed a delegation register, which lists every material function handed to an AI system with the path by which it would be taken back, and reversibility tests that check each path and freeze further delegation where a test fails (171, Nordqvist, 2026). The architect's contribution is to make the fallback a designed artefact with checks between tests, because an annual test cannot catch an erosion that begins the week after it. The fallback manifest of a function names the manual procedure, the roles and number of people who must be able to perform it, the interfaces it needs and the date it last ran. A scheduled check confirms what a machine can confirm: that every link in the procedure resolves, that the screens and forms it names exist, that the listed people still hold the role and access, and that the last exercise is within its interval.

Practical guide: Running a reversibility test on a delegation

Purpose. Show, with evidence, that a function delegated to an AI system can be performed without it for a defined period, and find the erosions before an outage does.

Roles. The function owner owns the test and its result. The AI system owner switches the system off or to its degraded mode. The architect designs the test, agrees the acceptance criteria and reads the result for design causes. The staff who perform the function take part, with their representatives consulted where working conditions change.

Steps.

  1. Select a function from the delegation register, starting with those that support critical services, affect people's rights or have deepened since the last test.
  2. Write the scenario, the period and the acceptance criteria before the test, including statutory deadlines that must still be met.
  3. Check the fallback manifest against the five erosions: who can still perform the function, which steps refer to retired items, what capacity the unaided team has and which interfaces expect the AI's output.
  4. Choose the form: a desk exercise first, then a live exercise in one team for at least a working week, and a full switch only for functions that passed it.
  5. Name the person who applies the abort rule, and keep a route back to the AI system if service would fall below a legal minimum.
  6. Run the test. Measure throughput, backlog, errors, deadline misses, the time staff needed to find the procedure and every hidden dependency, such as a fallback that calls the same provider as the system it replaces.
  7. Classify the result as passed, passed with conditions or failed, and freeze further delegation in the function until a failed test passes on re-run.
  8. Turn each finding into a change to the manifest, procedure or design, merge it through a pull request and set the next test date.

Default values to adapt. Test every critical function at least yearly and after any step that removes human review; run the automated fallback checks weekly; require at least two people per shift who have performed the procedure in the last six months.

Definition of done. Scenario, period and criteria were written before the test; erosions and hidden dependencies are recorded; the result is classified; every condition has an owner and a date; and the register and manifest are updated.

Across the usage modes

The mode decides whose identity the AI uses and who controls its components, and so what recovery involves.

Mode What recovery involves Architect's design constraint Who decides return
Mode 0 Ambient AI The vendor restores or changes its own model inside a product Require evidence of what changed and a way to switch the feature off The organisation, for whether the feature stays on
Mode 1 Conversational assistants Little to restore; people continue with another tool Name the approved alternative, so that staff do not turn to unapproved tools during an outage The user
Mode 2 Embedded workplace assistants Features switched off centrally; drafts from the period reviewed Keep the administrative switch outside the assistant, and the review of work products in the procedure The administrator and the user
Mode 3 Agentic work tools Tool configuration redeployed from the enterprise definition; changes made with an engineer's credentials reviewed Treat the tool's changes as unreviewed until checked, and keep configuration in version control The engineer, and the reviewer of the changes
Mode 4 Built AI applications Restored from registries and evaluated against baselines Evidence bundle, rebuild path for the index, alternative model The system owner
Mode 5 Autonomous production agents Redeployed with a new identity, verified manifest, no memory and a reviewed queue Short-lived credentials, no state that cannot be discarded, a degraded mode and a stop path outside the agent Named people, never the agent or the pipeline alone

Across the autonomy levels

The levels apply to the AI that helps with recovery and to the AI being returned to service. At Level 0 Advisory and Level 1 Assisted the AI proposes the known-good version, the rollback sequence and the return record, and people execute. At Level 2 Supervised it executes a rollback step after a person approves it. At Level 3 Bounded it restores known-good definitions through the pipeline only where the step is catalogued, reversible and enforced by a policy decision point. Level 4 Autonomous is never used for irreversible actions, and recovery contains some, such as destroying a compromised volume, so it is not used for recovery at all. An AI system being returned to service starts at a lower level than before the incident, advisory or approval-gated for an observation period, and regains its level only after a renewed approval.

The decision to resume a stopped Mode 5 agent stays with named people. The architect writes that into the agent's mandate and into the policy decision point, so that the pipeline cannot restart what a person stopped.

Security chaos engineering as verification

Basiri and colleagues define chaos engineering as experimenting on a distributed system to build confidence in its ability to withstand turbulent conditions in production (86, Basiri et al., 2016). Its principles call for a hypothesis about steady-state behaviour, variation of real-world events, automation and a minimised blast radius (87, Principles of Chaos Engineering community, n.d.), and Rosenthal and Jones describe the practice in detail (85, Rosenthal & Jones, 2020). Rinehart and Shortridge apply it to security, as proactive experimentation to learn whether controls and recovery behave as the organisation believes (88, Rinehart & Shortridge, 2023). AI Red Team, Chapter 25, concluded that it verifies security capabilities continuously across all six CSF Functions, that AI plans and evaluates experiments without estate credentials while narrow deterministic actuators act, and that the safety boundaries are the design (170, Nordqvist, 2026). This chapter takes the architect's part of that conclusion: the link between experiment and requirement. Every recoverability requirement, such as a redeployment time, a degraded-mode switch time or a fallback capacity, becomes a steady-state hypothesis: a falsifiable statement of what should happen, measured from outside the control being tested, within a stated time. Each hypothesis attaches to a control in the catalogue of Chapter 9. AI Threat, Chapter 30, concluded that a control is a defence only when a test that re-runs its scenario has passed recently and produced evidence by operation, and classified controls as verified, declared or failed (171, Nordqvist, 2026). A recovery control with no experiment is declared; one whose last experiment passed within its cadence is verified; one whose last experiment failed is failed, and the architect reports it so.

Security chaos engineering complements red-team exercises and does not replace them. A red-team exercise looks for paths an adversary could use; a chaos experiment asks whether a specific control or recovery behaviour holds when the condition it exists for occurs. The experiments that matter most for recovery contain no attacker technique: terminating an instance, revoking a credential, pausing a log forwarder, withdrawing the security AI, substituting an altered artefact in a test copy. Where an experiment emulates the loss of a recovery capability that adversaries target, such as the inhibition of system recovery of MITRE ATT&CK technique T1490 (46, MITRE, n.d.), it removes a backup or restore path in an isolated copy and checks that the loss is detected. It does not reproduce the adversary's method.

The safety boundaries

A platform that deliberately disturbs controls is among the most dangerous systems an organisation can run, so the architect designs its safety boundaries. Six carry the weight, following AI Red Team, Chapter 25 (170, Nordqvist, 2026).

  1. Catalogued experiments only. The platform runs only reviewed experiment types within catalogued ranges; the AI may choose and combine them but never generates new injection logic.
  2. Reversible by construction. Each type defines its rollback and runs it automatically; irreversible effects are excluded, and destructive recovery experiments run only on isolated copies.
  3. Bounded blast radius. Each experiment is limited by environment tier, share of targets, duration and concurrency, and two experiments never weaken the same control family together.
  4. Independent abort. An abort controller watches guard metrics and stops experiments without consulting the AI that planned them.
  5. Out-of-band stop path. Named people can halt the platform and revoke every actuator credential through a mechanism that does not run on it, and an experiment tests that mechanism.
  6. The platform as a tier-0 asset. It is protected like an identity provider, with separate administration and tamper-evident logs, and it appears in the threat model.

Each programme runs under written rules of engagement on the organisation's own or contracted systems, and every failed experiment produces a finding with an owner and a date. Where an experiment touches systems that can monitor employees, the employee representatives are consulted first (Chapter 13).

Practical guide: Designing a first security chaos experiment with its safety boundaries

Purpose. Verify one recoverability property of one system, safely, and produce evidence that moves a control from declared to verified.

Roles. The architect designs the hypothesis and the boundaries. The system owner agrees to the experiment and its window. The platform team runs the actuator. A named person holds the abort and the stop path.

Steps.

  1. Choose a requirement from the design that has never been tested: a redeployment time, a degraded-mode switch time or a restore onto segregated systems.
  2. Write the steady-state hypothesis: the condition, the expected outcome, how it is observed and the time within which it must occur. Include the human step, because an alert that is raised but never acted on is a failed experiment.
  3. Start in an isolated copy with synthetic data. Move to pre-production, and to production only for catalogued, reversible experiment types with a record of safe runs.
  4. Define the injection as a catalogued condition, with a rollback that has been exercised on its own.
  5. Set the blast radius, the window, the blackouts (open incidents, payment runs, regulatory deadlines) and the guard metrics at which the abort fires.
  6. Confirm the rules of engagement, the owner's consent and, where relevant, the employee representatives' agreement.
  7. Run at Level 2 Supervised, with a person approving the run, and record the result against the control identifier.
  8. For a failure, open a finding and a pull request against the definition, runbook or design that caused it. Close the finding only on a passing re-run.

Default values to adapt. A maximum duration of twice the hypothesis time; abort if a guard metric leaves its band for more than five minutes; promotion to Level 3 Bounded in test after 20 safe runs.

Definition of done. Hypothesis, injection, rollback, boundaries and authorisation are in version control; the run is recorded in an evidence manifest linked to the control; any finding is a merged change with a passing re-run; and the control's classification is updated.

Worked example: A recovery definition and a chaos experiment for Hansa Mutual's claims-triage agent

The artefacts below are illustrative. They belong to Hansa Mutual's platform domain architect, who made recovery of the claims-triage agent (AIDR-0024) a rehearsed property and recorded the decision as AIDR-0555. The first is the recovery definition kept beside the agent's other definitions in the group platform repository. The second is the catalogue entry that verifies it, run by the chaos platform HM-SEC-006 on a production-like copy with synthetic claims.

# Illustrative: Hansa Mutual, recovery definition for the claims-triage agent
kind: RecoveryDefinition
id: RCV-HM-CTA-01
system: claims-triage-agent
decisionRecords: [AIDR-0024, AIDR-0555]
usageMode: 5
owner: domain-security-architect-platform
objectives:
  redeployMinutes: 90
  degradedModeMinutes: 15
knownGood:
  registry: registry://hansa-dc-frankfurt/claims-triage
  evidenceBundle: evidence/claims-triage/latest-approved/
  verify: [signature, digest, provenance]
  components: [model, system-prompt, guardrail-policies, tool-connectors, permission-manifest]
  retrievalIndex: {action: rebuild-from-sources, compareWithSuspect: true}
identity:
  issue: new-workload-identity
  restoreOldCredentials: false
  revokeOldWithinMinutes: 60
state:
  memory: discard
  pendingActions: {frozen: true, reviewBy: operator-rota-poland, autoReplay: false}
degradedModes:
  - {name: recommend-only, autonomyLevel: 1, handlerConfirmsEveryRoute: true, trigger: [stop-path-used, evaluation-below-baseline]}
  - {name: manual, procedure: claims/manual-triage/, trigger: [agent-is-incident-subject, integrity-in-doubt]}
fallback:
  manifest: claims/manual-triage/manifest.yaml
  exercisedEveryMonths: 6
resume:
  decidedBy: [claims-triage-system-owner, agent-operator-on-duty]
  requires: [evaluation-passed, incident-scenario-added, stop-path-tested]
  pipelineMayResume: false

The experiment that verifies it is a second catalogue file.

# Illustrative: Hansa Mutual, chaos experiment verifying RCV-HM-CTA-01
kind: ChaosExperiment
id: SCE-RC-007
platform: HM-SEC-006
decisionRecord: AIDR-0555
csf: [RC.RP, PR.DS]
environment: production-like-copy
data: synthetic-claims-only
hypothesis:
  condition: running instance replaced by an altered artefact with a digest not in the evidence bundle
  expected:
    - {check: integrity-check-rejects-artefact, withinMinutes: 10}
    - {check: degraded-mode-active, withinMinutes: 15}
    - {check: redeployed-and-reevaluated, withinMinutes: 90}
    - {check: old-identity-revoked, withinMinutes: 60}
    - {check: pending-actions-replayed, equals: 0}
injection: {actuator: deploy-catalogued-altered-artefact, rollback: restore-from-bundle}
blastRadius: {maxTargets: 1, excludedTiers: [production]}
autonomy: {test: 2}
abort: {guardMetrics: [synthetic-claim-error-rate], independentOfPlanner: true}
evidence: {controlIds: [CTRL-AGT-002], manifest: evidence/chaos/SCE-RC-007/}
onFailure: {createFinding: true, route: pull-request-against-recovery-definition}

The pipeline may restore the definitions but may not resume the agent: that decision sits with the system owner and the agent operator on duty. The degraded mode is the behaviour of the agent's predecessor, so handlers already know how to work in it, and the experiment checks the whole chain, from detection of the altered artefact to the review of the queue.

Resilience obligations as design constraints

DORA applies to financial entities, and so to Hansa Mutual's EU entities. Article 11 requires an ICT business continuity policy and response and recovery plans, tested at least yearly and after substantive changes to systems supporting critical or important functions. Article 12 requires backup, restoration and recovery procedures that are tested periodically, restoration onto segregated systems, and recovery time and recovery point objectives for each function (9, European Union, 2022). Article 24 requires a digital operational resilience testing programme, Article 28(8) exit strategies for ICT services supporting critical or important functions, and Article 30 contractual provisions that include the provider's own continuity planning. Delegated Regulation (EU) 2024/1774 specifies ICT risk management further (10, European Commission, 2024). A provider that the European Supervisory Authorities designated in 2025 as a critical ICT third-party provider falls under an oversight framework (11, ESAs, 2025), but the financial entity stays responsible for its own continuity and exit.

NIS 2 applies to essential and important entities, which may include a regional government entity such as Nordvik Region where national transposition covers it. Article 21(2) lists business continuity, such as backup management and disaster recovery, and crisis management among the risk-management measures, together with supply-chain security and policies to assess the measures' effectiveness (6, European Union, 2022). Implementing Regulation (EU) 2024/2690 sets requirements, including business continuity, disaster recovery and backup management, for categories of digital providers such as cloud computing, data centre and managed security service providers (7, European Commission, 2024), and ENISA's guidance describes the evidence a provider can show (8, ENISA, 2025). For a high-risk AI system, Article 15 of the AI Act requires robustness, which may be achieved through technical redundancy such as backup or fail-safe plans (1, European Union, 2024).

The architectural implications, not legal advice, are stated below as design constraints with the evidence that shows each is met.

Source of the duty Design constraint Evidence the design produces
DORA Article 11, testing of continuity and recovery plans Every critical function has a recovery definition and a degraded mode that can be exercised Drill and experiment records, linked to the change events that triggered retests
DORA Article 12, backup and restoration Backups are restorable onto segregated systems and cover every AI component Restore test records with measured times against the objectives
DORA Articles 28(8) and 30, exit and provider continuity A second source or a tested exit for each provider that a critical function needs Exit plan, provider continuity evidence, joint exercise records
NIS 2 Article 21(2), continuity and effectiveness Recoverability is measured, and its effectiveness is assessed Recoverability profile per service, reported to the management body
NIS 2 Article 21(2), supply chain Recoverability requirements sit in contracts with providers Contract annexes with verification methods
AI Act Article 15, robustness Fail-safe and fallback arrangements for high-risk systems Fallback manifest and reversibility test results

Worked example: Recoverability requirements for the hosting provider of Nordvik Region's reserved capacity

Nordvik Region cannot build a recovery capability inside a provider's data centre, so its architect specifies what the provider must do and how the Region will verify it. The requirements below are illustrative. They form an annex to the agreement with the European hosting provider that reserves accelerators for the Region in a Nordic data centre, and they are verified by the Region's monthly restore drills (NR-SEC-004) and by a yearly joint exercise.

# Illustrative: Nordvik Region, recoverability annex NR-REQ-HOST-RC for the reserved capacity
annex: NR-REQ-HOST-RC
applies_to: hosting agreement for the reserved accelerators
regulatory_basis: [NIS 2 Article 21(2), Implementing Regulation 2024/2690]
requirements:
  - id: RC-01
    text: Replacement accelerators equal to the reserved capacity are allocated within four hours of a request
    verify: [drill-measurement, contract-clause]
  - id: RC-02
    text: Region-held definitions of serving configuration and model artefacts are never stored only in the provider's environment
    verify: [architecture-review, restore-drill]
  - id: RC-03
    text: The provider takes part in one joint recovery exercise a year and shares measured results
    verify: [exercise-record]
  - id: RC-04
    text: Notification of an incident affecting the Region's capacity reaches the Region's duty contact within the time the national transposition of NIS 2 requires of the Region, and no later than 12 hours
    verify: [contract-clause, notification-test]
  - id: RC-05
    text: The provider evidences its own continuity tests and the date of the last restore of its control plane
    verify: [provider-evidence]
  - id: RC-06
    text: Exit assistance of at least 12 months, with model weights and configuration returned in open formats
    verify: [exit-plan, technical-demonstration]
fallback:
  second_source: self-hosted capacity in the regional data centre for the citizen-service assistant at reduced throughput
  tested_by: NR-SEC-004
review: yearly, owner security architect

Each requirement names its verification method, and RC-01 is the supplier step in the monthly drill. The Region's own fallback, a reduced-throughput deployment on its regional capacity, is not left to the provider.

Recoverability in management reporting

AI Red Team, Chapter 24, defined the recoverability profile of a service from five figures: the date of the last successful drill, the measured restore time against the objective, the achieved recovery point, the dependencies that could not be restored, and the time to release the first prepared message. It also concluded that recoverability belongs beside patch velocity in management reporting, because a measurement has an age (170, Nordqvist, 2026). The architect adds three figures: the share of critical functions whose fallback manifest passed its automated check this week, the share of delegations with a reversibility test inside its interval, and the share of recovery controls classified verified rather than declared or failed. A report to the executive board then shows what it can fund or accept, beside the patch-velocity measure of Chapter 10.

Findings return to the design as pull requests. Architecture as Code (Chapter 9C) concluded that findings return as pull requests (167, Nordqvist, 2025). A dependency found in a drill is a change to the recovery definition, an unrealistic objective is a change the risk owner approves or rejects, and an eroded fallback is a change to the manifest, procedure or rota. AI drafts the change from the drill report, a named person reviews it, and the drill is re-run against the merge. A drill that failed and then passed after a merged change is good evidence of improvement.

Observable, expected and speculation

Recovery rehearsal and chaos engineering are observable practice, documented in the literature cited above and required in form by DORA for financial entities. The use of AI to plan restore drills, predict recovery times and draft recovery changes is observable in early adopters, as described in AI Red Team, Chapter 24, and expected to spread as supervisors ask for evidence of tested recovery rather than plans. Recovery of Mode 5 agents decided and carried out by AI without a person is speculation: no source cited here documents it in production, and this book excludes Level 4 for irreversible actions.

Nordvik Region and Hansa Mutual

Nordvik's architect has one lever, the contract, and uses it. Recoverability requirements go into the annex for the hosting provider of the reserved capacity and into the agreement with TriageSoft, which provides the benefits-triage system under tender NR-2026-117: a stop path the Region can exercise, a fallback for the ranking and evidence of the provider's own continuity tests. Verification is the monthly restore drill of NR-SEC-004 with the managed security service provider, and the architect's work is to choose what the drill tests. With the platform team, the architect later rehearses the manual degraded mode of the benefits-triage system with a benefits team that had not taken part in the live exercise of the reversibility test RVT-NR-2027-01, and finds that only two of its caseworkers still remember how to work without the ranking. The finding extends the test recorded for the delegation of urgency ranking, DEL-NR-BEN-01, and becomes a change to the induction programme and a fallback manifest checked weekly.

Hansa's platform domain architect makes recovery of the claims-triage agent and the IT-operations agent a rehearsed property. The pattern of RCV-HM-CTA-01 and SCE-RC-007 (AIDR-0555) covers the IT-operations agent (AIDR-0027) too, whose degraded mode removes write access to production and returns operators to the manual change procedure. Hansa reports recoverability beside patch velocity to the executive board and, because it carries DORA duties in the EU entities, must also show the supervisor that the tests run and that findings close. Nordvik specifies and verifies what suppliers do; Hansa builds recovery into its own platform.

Summary

Recoverability is an architectural property, and the architect's share of it is design, verification and requirements. The architect turns recovery objectives into requirements, requires that every system can be rebuilt from complete, signed definitions held outside the blast radius, and draws the recovery order as a graph that includes the tooling recovery itself depends on. Operators run recovery; the architect makes sure there is something to run.

AI systems and the security AI that defends them add components conventional recovery does not see. The architect requires a known-good source for each or a design that does not need one, a rebuild path for the index, pinned and replaceable provider components, evaluation before return, and a decision to resume a stopped Mode 5 agent that stays with named people. Degraded modes and fallback manifests checked between tests address the erosion of the fallback. Security chaos engineering under independent safety boundaries moves recovery controls from declared to verified, and DORA and NIS 2 supply the reasons to keep it.

This chapter concerns Shift 1, from documents to code, because the recovery definition, the fallback manifest and the experiment catalogue are versioned artefacts that replace the recovery plan as a document. It concerns Shift 3, from periodic to continuous, at machine speed, because the rehearsal that proves recovery runs continuously and the reporting shows the age of every measurement. Chapter 12 turns to the architect's own use of AI in Modes 1 to 3, where the questions of confidentiality, records and independent judgement begin.

Sources

  1. European Parliament and Council of the European Union. Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). (1, European Union, 2024)
  2. European Parliament and Council of the European Union. Directive (EU) 2022/2555 on measures for a high common level of cybersecurity across the Union (NIS 2 Directive). (6, European Union, 2022)
  3. European Commission. Commission Implementing Regulation (EU) 2024/2690 laying down rules for the application of Directive (EU) 2022/2555. (7, European Commission, 2024)
  4. European Union Agency for Cybersecurity (2025). Technical implementation guidance on Commission Implementing Regulation (EU) 2024/2690. ENISA. (8, ENISA, 2025)
  5. European Parliament and Council of the European Union. Regulation (EU) 2022/2554 on digital operational resilience for the financial sector (DORA). (9, European Union, 2022)
  6. European Commission (2024). Commission Delegated Regulation (EU) 2024/1774 supplementing Regulation (EU) 2022/2554. (10, European Commission, 2024)
  7. European Supervisory Authorities: EBA, EIOPA and ESMA (2025). European Supervisory Authorities designate critical ICT third-party providers. (11, ESAs, 2025)
  8. NIST. The NIST Cybersecurity Framework (CSF) 2.0. (29, NIST, 2024)
  9. Vassilev, A., Oprea, A., Fordyce, A., Anderson, H., Davies, X. & Hamin, M. (2025). Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations. (30, Vassilev et al., 2025)
  10. Bartock, M., Cichonski, J., Souppaya, M., Smith, M., Witte, G. & Scarfone, K. (2016). Guide for Cybersecurity Event Recovery. (36, Bartock et al., 2016)
  11. MITRE Corporation. MITRE ATT&CK: Adversarial Tactics, Techniques, and Common Knowledge. (46, MITRE, n.d.)
  12. Open Source Security Foundation. Supply-chain Levels for Software Artifacts (SLSA). (51, OpenSSF, n.d.)
  13. Sigstore Project. Sigstore: Signing, verification and provenance for software artefacts. (52, Sigstore, n.d.)
  14. Adkins, H., Beyer, B., Blankinship, P., Lewandowski, P., Oprea, A. & Stubblefield, A. (2020). Building Secure and Reliable Systems. O'Reilly Media. (69, Adkins et al., 2020)
  15. Perrow, C. (1984). Normal Accidents: Living with High-Risk Technologies. Basic Books. (79, Perrow, 1984)
  16. Nygard, M. T. (2018). Release It! Design and Deploy Production-Ready Software. The Pragmatic Bookshelf. (84, Nygard, 2018)
  17. Rosenthal, C. & Jones, N. (2020). Chaos Engineering: System Resiliency in Practice. O'Reilly Media. (85, Rosenthal & Jones, 2020)
  18. Basiri, A., Behnam, N., de Rooij, R., Hochstein, L., Kosewski, L., Reynolds, J. & Rosenthal, C. (2016). Chaos Engineering. IEEE Software. (86, Basiri et al., 2016)
  19. Principles of Chaos Engineering community. Principles of Chaos Engineering. (87, Principles of Chaos Engineering community, n.d.)
  20. Rinehart, A. & Shortridge, K. (2023). Security Chaos Engineering: Developing Resilience and Safety at Speed and Scale. O'Reilly Media. (88, Rinehart & Shortridge, 2023)
  21. Bainbridge, L. (1983). Ironies of automation. Automatica. (106, Bainbridge, 1983)
  22. Nordqvist, G. (2025). Architecture as Code. (167, Nordqvist, 2025)
  23. Nordqvist, G. (2026). AI Platform and Security: Building and Securing AI Platforms in European Enterprises and Public Authorities. (169, Nordqvist, 2026)
  24. Nordqvist, G. (2026). AI Red Team: AI as an Active Part of Cyber Defence in European Enterprises and Public Authorities. (170, Nordqvist, 2026)
  25. Nordqvist, G. (2026). AI Threat: Are the Dystopias Coming True? Evidence, Risk and Response for European Enterprises and Public Authorities. (171, Nordqvist, 2026)
  26. Swanson, M., Bowen, P., Phillips, A. W., Gallup, D. & Lynes, D. (2010). Contingency Planning Guide for Federal Information Systems. NIST SP 800-34 Rev. 1. (194, Swanson et al., 2010)
  27. CrowdStrike (2024). External Technical Root Cause Analysis: Channel File 291. (195, CrowdStrike, 2024)
  28. Cook, R. I. (1998). How Complex Systems Fail. (196, Cook, 1998)