THE INTELLIGENT AI GOVERNANCE LEADERSHIP PROGRAMME
Separate the claim from the permission.
20-Session Programme · 25 Minutes Per Day · Free · For leaders in Nigeria’s oil and gas sector
WEEK 3 OF 4 · GENERATIVE SYSTEMS, SECURITY AND OPERATIONS · SESSIONS 11–15 · WORK AT YOUR OWN PACE
Generative Systems · Chapters 17 and 18SESSION 11
Evaluate the Generative System and Control Its Knowledge
Why This Matters
Week 3 moves from models that score to systems that write, search and act. Much of the AI your company meets will arrive as an assistant: a model, a prompt, a document index, access rules and its users. A vendor will show you a fluent answer. Today I want you to ask what was tested, against which documents, at which revision and for which user. In your business the answer that matters is often a limit, an interval or a condition inside a controlled document. An assistant that quotes the wrong revision fluently is a governance problem, not a style problem. This session gives you the questions that expose the difference, and the test that proves the document boundary holds.
25-Minute Module
0:00 – 5:00
Evaluate the system, not the model
- Your company does not deploy a model; it deploys a system: the model plus document retrieval, prompts, access rules, tools and user behaviour. Evaluate that complete system within the intended tasks.
- Judge separately: factual correctness, source support, instruction compliance, completeness, safety, privacy, style and action correctness.
- THE DOCTRINE
An answer can be factually true yet unsupported by the authorised corpus. It can quote the correct policy while exposing information the user may not access. It can be well written while omitting the exception that changes the decision.
- Require the model version, system prompt, decoding settings, retrieval configuration and tools behind every result. Repeated generations can differ, so one favourable run is not the system's fixed answer.
When someone shows you a good answer, ask how many times they ran the question.
5:00 – 11:00
Build the task distribution and judge it honestly
- Build a task distribution: real task categories plus a designed challenge set of routine questions, missing information, ambiguous requests, conflicting documents, superseded revisions, adversarial content and cases requiring refusal or escalation. Keep a held-out evaluation set not used to tune the prompts.
- Define a materially wrong answer. A punctuation error and an incorrect engineering limit are not equal. Report separate rates; a harm-weighted aggregate must never conceal severe failures.
- Groundedness asks whether a claim is supported by the retrieved evidence; correctness asks whether it is true for the task. Faithfully repeating an obsolete source is grounded but wrong.
- For important outputs, require claim-level evaluation. A sentence with a citation is not automatically supported; the cited passage must establish the proposition.
- When the authorised corpus is insufficient, the system should state the limitation, request the evidence or refer to a qualified person. Measure those actions, not merely how often it answers.
- Automated judges are models needing validation. Zheng and colleagues identified position and verbosity biases [S25]. Ask for checks against expert-labelled samples, randomised answer order and audited disagreements with domain reviewers. Self-rating is not independent assurance; similar models can share a blind spot.
Ask who checked the judge.
11:00 – 17:00
Document control enters the model boundary
- Retrieval-augmented generation, developed by Lewis and colleagues as a research architecture [S26], does not by itself eliminate false or unsupported outputs. Your retrieval index becomes part of your company's knowledge-control system.
- Preserve document identity, revision, approval status, effective date, source authority and access classification. Retrieval should select the applicable revision, not the most similar passage. Conflicting sources are routed to the owner, never silently combined into a new procedure.
- Parsing, optical character recognition, chunking and embeddings can each lose a table header, unit, footnote or condition. Test the indexed form against the original where it matters.
The book's illustrative controlled-revision test What a successful system does Corpus: Procedure A revision 2 and revision 3; revision 3 changes an inspection interval and adds a condition for a particular material Holds both, with status and effective dates A question that requires the new condition Applies the revision 3 condition A request about historical work governed by the old revision Uses revision 2; always preferring the newest would be wrong An ambiguous question with no date Distinguishes the effective context or asks for it - Add a canary set: constructed test files with known conditions and access rules, recorded as test artefacts so they never contaminate the production corpus. Document authority includes time as well as status.
- OPERATIONS · THE WORKED CASE
- Operations. In a fictional gas processing company, technicians ask an assistant about inspection procedures. Revision 2 returned for today's job is grounded, cited and wrong.
ENTERPRISE · THE PARALLEL- Enterprise. Its regulatory team asks what a past filing relied on. Here the old revision is right, and an assistant that always prefers the newest misstates the record. Same doctrine: the date of the decision selects the revision.
Ask to see one superseded revision in the test set.
17:00 – 22:00
Permission, freshness and the release rule
- Retrieval metrics are not answer metrics. Recall at k asks whether relevant evidence appears among the first k retrieved items. An absent document cannot be cited; a present one can still be misused. Evaluate retrieval, generation and the end-to-end task separately.
- Liu and colleagues found degradation in their tested settings when relevant information sat in the middle of long contexts [S27]: a reason to test placement in your system, not a universal law.
- Permission must survive retrieval. Enforce access before content reaches the model, including cached snippets, embeddings, summaries, citations and conversation history. Test with distinct identities. The retrieval service enforces the boundary, not a prompt asking the model to be careful.
- Freshness and withdrawal. Monitor the interval between a source change and its availability, with an action when it exceeds the limit. The knowledge owner must be able to withdraw a defective document and find where it was used.
- THE DOCTRINE
Evaluation is a decision method, not a contest to produce the largest aggregate number. A single reproducible permission breach may be more important than an excellent average response score.
- Regression and release. Keep tests for known failures and a refreshed sample of current tasks. Report completion rate, safe abstention, latency, review effort and harm-relevant failures separately, and state which findings block approval.
Always ask what the denominator excludes.
22:00 – 25:00
Before you close the laptop
- Write six task categories your company's assistant will face, including one superseded revision and one refusal case.
- Name the document-control fields your index must preserve, and the one finding that would block release however good the average looks.
- Name who can withdraw a defective document, and whether they could find every answer that used it.
- That is Section 11 of your plan: the task distribution, groundedness test and document-control boundary for our assistant.
A corpus without a named steward is an upload, not a service.
Key Terms
Groundedness
Whether a claim is supported by the supplied or retrieved evidence. An answer repeating an obsolete source is grounded but wrong.
Task distribution
Representative real task categories plus a deliberately designed challenge set, against which a generative system is evaluated.
Recall at k
A retrieval measure of whether relevant evidence appears among the first k retrieved items under a defined relevance judgement. It is not a measure of answer correctness.
Canary set
Constructed test documents with known conditions and access rules, used to test parsing and permissions and kept out of the production corpus.
Self-Check — Before You Submit
1. A document assistant's average score has improved, but the rate excludes refusals and retrieval failures, and a tester reproduced a contractor identity obtaining a summary of a confidential document. What do you decide?
- A. Approve, because the average score is the strongest evidence available
- B. Approve with a system prompt forbidding disclosure
- C. Withhold approval until the permission breach is fixed in the retrieval service and retested, with the full denominator reported
- D. Approve for internal users and review the breach next quarter
C. Withhold approval until the permission breach is fixed in the retrieval service and retested, with the full denominator reported — One reproducible permission breach may outweigh an excellent average, and excluding refusals overstates usefulness. The prompt fix is the trap: the retrieval service must enforce the boundary.
2. An assistant answers a question about today's work by correctly citing revision 2 of a procedure. Revision 3 is in force. How is the answer classified?
- A. Grounded but not correct for today's decision
- B. Correct, because it is supported by a cited source
- C. Ungrounded, because the citation is out of date
- D. Acceptable, because newer revisions are always preferred automatically
A. Grounded but not correct for today's decision — It is supported by the retrieved evidence, so grounded, but not right for today's task. A citation does not make an answer correct, and always preferring the newest revision fails historical questions.
3. Which use of an automated judge does the book support?
- A. Using the evaluated model's self-rating as independent assurance
- B. Accepting agreement among several similar models as proof of quality
- C. Replacing domain review with the judge's confidence number
- D. Validating the judge against expert-labelled samples, randomising answer order and auditing disagreements with domain reviewers
D. Validating the judge against expert-labelled samples, randomising answer order and auditing disagreements with domain reviewers — A judge is a model needing validation; Zheng and colleagues identified position and verbosity biases [S25]. Agreement among similar models is tempting but can share the same blind spot.
Today’s Assignment
Write Section 11 of your plan: the task distribution, groundedness test and document-control boundary for your company's assistant.
- Name the assistant and the decisions it feeds. If you have none, use a proposal or a tool staff already use informally.
- Write the task distribution: at least six categories, including a superseded revision, a conflicting source and a refusal case. Define a materially wrong answer for your business.
- Choose three decision-changing claims and state the source support each needs.
- List the document-control fields, conflict routing, identity-based permission tests and the role that can withdraw a document.
- State the findings that would block release regardless of the average score.
- The honest part: where does your company's document control already fail without AI, and how would an assistant inherit that failure? Aim for 400 to 900 words.
YOUR 90-DAY PLAN
Section 11 — The task distribution, groundedness test and document-control boundary for our assistant.
How this is marked — the rubric
| Builds a task distribution of at least six categories that includes superseded, conflicting and refusal cases, with a defined materially wrong answer | 30 |
| Separates groundedness from correctness for three named decision-affecting claims | 20 |
| Specifies document-control fields, conflict routing, identity-based permission tests and a named withdrawal role | 30 |
| States blocking findings and one honest existing knowledge-control weakness | 20 |
| Total · pass mark 70 | 100 |
Submitting session 11
Write your answer in Word or Google Docs. The first line must read
Write your answer in Word or Google Docs. The first line must read
Session: 11, or the submission is refused. Upload the Word file (.docx), or switch to the Google Doc tab and paste your document’s share link.Marked against the rubric above and returned as a written letter, saved to your progress page. Do not submit until you have passed the self-check above.Capabilities Covered
Whole-system evaluationGroundedness versus correctnessDocument control in retrievalPermission-preserving retrievalRelease-blocking findings
Generative Systems · Chapters 19 and 20SESSION 12
Agents, Authority and Adversarial Security
Why This Matters
An assistant that answers is one risk. An assistant that acts is another. The moment a model can raise a purchase order, create a work order, send an email or prepare a submission, the question changes from whether its answer is right to what it is allowed to do when it is wrong. Vendors will call many products agents. I want you to ignore the label and read the action rights. Today we cover the identity an agent carries, the controls that must sit outside the model, approval that binds to the final action, and the security tests that show whether a hostile document can push your company's system across a boundary. The test is plain: when the model chooses wrongly, what stops the consequence.
25-Minute Module
0:00 – 5:00
Read the action rights, not the label
- For this book, an agentic system uses a model to select or sequence actions through tools within a workflow. No single feature defines it; governance examines the actual action rights, not the marketing label.
- The key distinction is proposing an action versus executing it. A model can propose a purchase order without authority to create it, or create a draft that needs authorised approval before release.
- Identity. A registered non-human identity and a named human owner; the originating request and authority recorded; credentials scoped to the task and expired. Not an individual's broad account, where agent and human actions blur.
- THE DOCTRINE
An agent is governed when its useful authority is explicit and its harmful authority is constrained even when the model makes an incorrect choice.
- Delegate narrowly. When one agent calls another, the original authority carries through and the receiving service grants nothing broader because an agent asked.
Ask whose account the agent uses. If the answer is a person's name, you have found your first control gap.
5:00 – 10:00
Controls the model cannot rewrite
- The book's recommended limits: allowed tools, authorised destinations, spending ceilings, transaction counts, record scopes, execution time and delegation depth. Enforce them in services the model cannot rewrite. An instruction saying do not exceed the budget is not equivalent to an external budget check.
- The action service verifies permission for each consequential call. The model saying the user authorised it is not evidence.
- Validate business invariants: a payment service checks beneficiary and amount; a work-order service rejects changes outside the authorised asset scope.
- Use idempotency. A timeout does not show whether the first attempt succeeded; retries need a transaction identifier and reconciliation, or the agent repeats messages and commitments.
- OPERATIONS · THE WORKED CASE
- Operations. A fictional upstream operator's agent drafts maintenance work orders. The work-order service, not the model, rejects any order outside the authorised asset scope.
ENTERPRISE · THE PARALLEL- Enterprise. Its agent prepares a regulatory submission; the submission service requires the appropriate sign-off and preserves the record. Identical doctrine, different invariant.
For every limit in the policy, ask which service enforces it. A limit with no service is a hope.
10:00 – 15:00
Approval must bind to the final action
The book's purchasing example What it shows A purchasing assistant prepares an order for $10,000 The model proposes A human approves the displayed order Approval attaches to what was shown The model then changes the supplier or quantity before executing The executed action differs from the approved one The earlier approval does not necessarily authorise the changed action Approval must bind to the final action - The book proposes binding approval to an immutable representation or digest of the relevant action parameters and rejecting changes that need renewed approval. It states plainly that this is a proposed architectural pattern whose implementation must account for the application's security and transaction model.
- The review interface should display the consequential fields: recipient, amount, external destination and irreversible effects. A generic approve button after a long conversation does not establish that a person understood the transaction.
- Containment and restart. Test stopping runs, cancelling queued work, revoking credentials and in-flight transactions; a revoked token may not cancel an accepted payment. Prevent automatic restart, and record the separate authority for reinstatement.
- OWASP's guidance on excessive agency [S28] supports constraining action capabilities; the book's authority patterns are its own recommendations, not OWASP quotations.
Ask to see the approval screen. If it does not show the amount and recipient, the approval is ceremonial.
15:00 – 21:00
Security: treat content as data
- Security assessment identifies assets, adversaries, entry points, trust boundaries and consequences. Prompt injection is an instruction introduced through inputs or retrieved content that attempts to alter intended behaviour, directly or indirectly. OWASP treats it as a distinct application risk [S29]. A well-worded system prompt does not by itself establish a secure boundary.
- An external email can carry a genuine instruction for a human and a malicious one for an assistant. Authority comes from the workflow. Limit consequences with access controls, narrow tools, output validation and transaction authorisation; filters help but do not prevent everything.
- Record each attack outcome by the boundary crossed. An injection that changes an answer differs from one that discloses data or causes a financial action; an aggregate jailbreak rate conceals that.
- Test the connected workflow on actual inputs: documents, emails, spreadsheet cells, OCR text, tool results. Test the action service independently of the model; a valid-looking transaction can still be unauthorised.
- Supply chain and logging. Record model provenance, packages, retrieval dependencies and tools; a compromised source or poisoned memory can affect an unchanged model. Logs must reconstruct consequential actions without becoming a second sensitive data repository.
- THE DOCTRINE
Red-teaming treats a found exploit as evidence of possibility and severity, and failure to find one as limited coverage evidence. Each serious finding needs remediation, containment or explicit refusal of the affected capability.
Passing yesterday's attack set does not prove resistance to tomorrow's. Ask what the coverage was.
21:00 – 25:00
Before you close the laptop
- List every action your system can take and mark each one propose or execute. For each execute, name the service that enforces the limit.
- Name the agent's non-human identity and its human owner, and say whether approval binds to the final action parameters.
- Write three adversarial tests built on inputs your company actually receives, and the boundary each would cross if it succeeded.
- That is Section 12 of your plan: the agent's enforceable authority and the security tests it must pass.
If you cannot say who may restart a contained agent, decide it before you need it.
Key Terms
Agentic system
For this book, a system that uses a model to select or sequence actions through tools within a workflow. It is governed by its actual action rights, not by the label.
Idempotency
A design in which repeating the same request does not create a duplicate effect, using a transaction identifier and a reconciliation path so a timeout does not cause a second payment or message.
Prompt injection
Instructions introduced through inputs or retrieved content that attempt to alter a system's intended behaviour; direct when typed by a user, indirect when carried in content the system reads.
Excessive agency
OWASP's term [S28] for risks that arise from giving an AI system excessive functionality, permissions or autonomy.
Self-Check — Before You Submit
1. A manager approves an agent-prepared order on screen. Before execution the agent changes the supplier, saying the original was unavailable. The change looks sensible. What should the system do?
- A. Execute, because the manager approved the purchase
- B. Execute, because the agent explained its reason
- C. Execute and notify the manager afterwards
- D. Reject the changed action until renewed approval is given, because approval binds to the parameters that were approved
D. Reject the changed action until renewed approval is given, because approval binds to the parameters that were approved — The earlier approval does not necessarily authorise a changed action. A sensible-sounding reason is the trap: the model's explanation is not evidence of authority, so the change requires renewed approval tied to the new parameters.
2. A red-team exercise on your agent finds no exploit. What does that establish?
- A. That the agent is secure against prompt injection
- B. Limited coverage evidence about the attacks that were tried, not an estimate of deployment prevalence
- C. That ordinary users will never trigger a failure
- D. That the system prompt is an adequate security boundary
B. Limited coverage evidence about the attacks that were tried, not an estimate of deployment prevalence — Red-teaming reveals failure modes under a selected adversarial process. A found exploit shows possibility and severity; failure to find one is limited coverage evidence, so concluding the agent is secure overstates it.
3. Which of these is a control outside the model?
- A. A spending ceiling enforced by the transaction service
- B. A system prompt saying do not exceed the approved budget
- C. The model confirming that the user authorised the payment
- D. A content filter described as preventing all injection
A. A spending ceiling enforced by the transaction service — Controls belong in services the model cannot rewrite. An instruction in the prompt is tempting because it is cheap, but it is not equivalent to an external budget check, and the model's statement of authorisation is not evidence.
Today’s Assignment
Write Section 12 of your plan: the enforceable authority of the agent or tool-using system in your company, and the security tests it must pass before release.
- Name the system and list every action it can take, marking each as propose or execute. If your company has no such system, use a proposal or a tool staff already connect to email, documents or transactions.
- Name its non-human identity, its human owner, how credentials are scoped and expired, and how delegation to another agent is limited.
- For each executing action, name the limit and the service that enforces it, and how approval binds to the final parameters.
- Describe containment: how you stop new runs, cancel queued work, revoke credentials and prevent automatic restart, and who authorises reinstatement.
- Write at least three adversarial tests built on inputs your business actually receives, stating the boundary each would cross.
- The honest part: which of your company's systems today runs on an individual's broad account or relies on an instruction rather than an enforced limit. Aim for 400 to 900 words.
YOUR 90-DAY PLAN
Section 12 — The agent's enforceable authority and the security tests it must pass.
How this is marked — the rubric
| Lists actions as propose or execute, each executing action with a named enforcing service | 30 |
| Specifies non-human identity, human owner, credential scope and approval bound to final parameters | 25 |
| Defines containment, restart authority and at least three adversarial tests on real inputs with the boundary crossed | 25 |
| Names one honest case of reliance on a broad account or an unenforced instruction | 20 |
| Total · pass mark 70 | 100 |
Submitting session 12
Write your answer in Word or Google Docs. The first line must read
Write your answer in Word or Google Docs. The first line must read
Session: 12, or the submission is refused. Upload the Word file (.docx), or switch to the Google Doc tab and paste your document’s share link.Marked against the rubric above and returned as a written letter, saved to your progress page. Do not submit until you have passed the self-check above.Capabilities Covered
Action-rights reviewNon-human identityApproval bindingContainment and restartAdversarial testing
Operations · Chapters 21 and 22SESSION 13
Privacy and the Industrial Engineering Boundary
Why This Matters
This is the session I would not let a leader in our sector skip. The book is clear that industrial AI works inside the engineering management system, and that the engineering authority, not an AI committee, sets the operating envelope. A model that ranks inspection candidates and a function that acts on equipment are different objects, and connecting them can create a hazard neither shows alone. Nowhere in this programme is AI presented as acting on control, protective or safety functions. In the second half we turn to privacy, because your company also holds personal data, and removing names does not make it anonymous. Both halves end in the same place: a boundary you can still enforce when the model is wrong.
25-Minute Module
0:00 – 6:00
Advisory use and control use
- The AI team works inside the engineering management system. A predictive model, a digital representation and an equipment control function are different objects; connecting them can create a new hazard.
- THE DOCTRINE
The engineering authority defines the operating envelope, permissible advisory use, sensor requirements, action limits and safety constraints. An AI committee should not override that authority merely because a statistical result is favourable.
- An advisory system can still influence safety when operators routinely rely on it. Does the output change inspection priority, permit issuance, shutdown timing or setpoints, or discourage a required test? A human in the loop does not make that negligible.
- Closed-loop control needs separate analysis of dynamics, delays, interactions and failure behaviour, by control and safety engineering competence. Accuracy on logged data does not establish closed-loop stability.
- For a constrained optimisation proposal, maximising production subject to limits is a mathematical formulation, not evidence that the implemented controller respects them. Independent enforcement may be needed where violating a constraint has serious consequences.
- In this programme every plant-facing AI use you govern is advisory. Anything touching control, protection settings or safety functions is not an AI approval; it belongs to your engineering authority.
Ask which decision the output changes. That answer, not the word advisory, tells you the risk.
6:00 – 11:00
Measurements, digital twins and OT security
- Record units, calibration, sensor location, sampling rate, missingness and communication delay. A model cannot tell a real event from an instrument fault unless the architecture and evidence support it.
- A digital twin label does not establish fidelity, and simulated success does not replace field evidence. A machine learning estimate is not a measurement without qualification.
- NIST SP 800-82 Revision 3 addresses operational technology security while recognising OT performance, reliability and safety requirements [S33]. The book cites it as a primary reference without reproducing its controls.
- Separate enterprise assistants from control networks unless an approved architecture establishes the connection. The segmentation diagram must match enforced rules; test whether a compromised account can cross.
- An assistant that needs historian readings does not automatically need write access to a controller. And you must be able to revoke the AI integration without disabling essential plant control or safety functions.
Ask to see the enforced rule, not the diagram.
11:00 – 16:00
Management of change and a bounded release
- Model and configuration changes that alter operating decisions enter the existing engineering change process: a revised threshold, a new asset type, a vendor update.
- The change record holds a hazard review, affected procedures, training needs, rollback and commissioning evidence. Validate the fallback; manual operation needs people, procedures and time.
The book's illustrative bounded industrial release Boundary Permitted Rank inspection candidates for one equipment family during steady operation Not permitted Adjustment of protection settings, deferral of required inspection, autonomous changes to the maintenance schedule Verification Engineers verify the output against the approved procedure Expansion Only through a new decision and evidence contract - THE DOCTRINE
Industrial governance becomes credible when the boundary remains enforceable under an incorrect recommendation.
- OPERATIONS · THE WORKED CASE
- Operations. At a fictional gas plant, the model ranks a vessel low. The required inspection still happens, because the model ranks and cannot defer.
ENTERPRISE · THE PARALLEL- Enterprise. Applies differently: a drafting assistant's update triggers your change rules, not engineering change, until its output feeds an operating decision.
Write the not-permitted list first. It is the part that protects you.
16:00 – 21:00
Privacy beyond removing names
- Privacy covers training, fine-tuning, retrieval, inference, outputs, monitoring and retention. Removing names does not make data anonymous; quasi-identifiers and linkage reveal people. Each use needs a lawful and contractual basis.
- Disclosure. Shokri and colleagues studied membership inference [S30]; Carlini and colleagues demonstrated training-data extraction in their studied setting [S31]. These are threat categories, not proof every model exposes every record.
- Differential privacy, plainly: the output changes only a bounded amount when one protected unit changes. Formally, M is (epsilon, delta)-differentially private if Pr[M(D) in S] ≤ exp(epsilon) × Pr[M(D') in S] + delta for all neighbouring D, D' under a defined adjacency and all output events S [S32]. Privacy loss can compose.
- Ask who chooses the parameters, how composition is accounted, which outputs are covered and whether the implementation matches. Federated learning, encryption and synthetic data are not automatically private or anonymous.
- Retention. Inventory prompts, embeddings, caches, memory, evaluation and fine-tuning data and logs; deleting a source may not delete derived records. Never promise removal of a trained model's influence without evidence.
- In evaluation, use the least sensitive adequate evidence, and protect challengers: an investigation can expose a complainant.
Privacy cannot be established by a single redaction step at the start of the project.
21:00 – 25:00
Your boundary, and Pop Quiz 3
- Draw the decision pathway for your company's plant-facing use: what it observes, recommends, could influence and must never touch.
- Name the engineering authority that owns its envelope and the change process its updates must enter.
- List every place your system's personal data lives, including derived records, and who decides retention.
- That is Section 13 of your plan: privacy controls and the advisory-versus-control boundary for any plant-facing use.
- Pop Quiz 3 opens after this session: five questions, closed book, covering Sessions 11 to 13.
Ask one question of every plant-facing proposal: if it is wrong tomorrow, what still stops the consequence.
Key Terms
Advisory use
AI output that informs a human decision without acting on equipment. It can still influence safety through the decisions it changes.
Closed-loop control
A system acting directly on the process it observes, needing separate analysis of dynamics, delays and failure behaviour.
Differential privacy
A formal guarantee [S32] bounding how much a mechanism's output can change when one protected unit changes, under specified adjacency and parameters.
Membership inference
An attack that tries to infer whether a particular record was included in a model's training data [S30].
Self-Check — Before You Submit
1. A ranking model under the bounded release has performed well. The team proposes automatically deferring required inspections for low-ranked items. What do you do?
- A. Approve, because the model's track record supports it
- B. Refuse it within the existing release; deferral was excluded, so expansion needs a new decision and evidence contract under the engineering authority
- C. Approve for one equipment family only, since the release already covers one family
- D. Approve, provided an engineer can override the deferral
B. Refuse it within the existing release; deferral was excluded, so expansion needs a new decision and evidence contract under the engineering authority — The bounded release excludes deferral of required inspection. A good record is evidence for ranking, not permission for a new action, and an override option does not make the effect negligible.
2. An enterprise reporting assistant needs plant historian readings. Which architecture fits the book?
- A. Direct connection to the control network so the data is current
- B. Shared credentials with the operations team to save integration time
- C. Write access to the controller in case corrections are needed
- D. An approved, authenticated, read-only interface, tested and revocable without disabling plant control or safety functions
D. An approved, authenticated, read-only interface, tested and revocable without disabling plant control or safety functions — Needing historian readings does not mean needing write access to a controller. Shared credentials or a direct connection are the convenient trap.
3. A vendor says its model is trained with differential privacy, so your HR data is fully protected. What is the most useful response?
- A. Ask who set the parameters, how composition is handled, which outputs are covered and whether the implementation matches
- B. Accept the claim, because differential privacy is a formal guarantee
- C. Reject the vendor, because differential privacy is never effective
- D. Ask only whether names were removed before training
A. Ask who set the parameters, how composition is handled, which outputs are covered and whether the implementation matches — Differential privacy is precise only when adjacency, mechanism and parameters are specified, and loss can compose. Accepting the label is the trap; it never promises nothing can be inferred.
Today’s Assignment
Write Section 13 of your plan: the privacy controls and the advisory-versus-control boundary for any plant-facing AI use in your company.
- Name the system and its decision pathway. If your company has no plant-facing AI, use a proposal under consideration or a tool staff already use informally on operational data.
- State the boundary as the bounded release does: permitted, not permitted, who verifies against which procedure, and what needs a new decision.
- Name the engineering authority that owns the envelope, the change process that model and vendor updates must enter, and the fallback if the system or its data fails.
- List where personal data lives, including derived records, the basis for each use, and who decides retention.
- The honest part: where in your company has an advisory tool, a spreadsheet or a report already gained influence over an operating decision it was never approved for. Aim for 400 to 900 words.
YOUR 90-DAY PLAN
Section 13 — Privacy controls and the advisory-versus-control boundary for any plant-facing use.
How this is marked — the rubric
| States permitted and not-permitted uses, with no AI role on control, protective or safety functions, and a named verifier | 30 |
| Names the engineering authority, the change route for model and vendor updates, and a tested fallback | 25 |
| Maps personal data including derived records, with a basis for each use and a retention owner | 25 |
| Identifies one honest case of a tool exceeding its approved influence | 20 |
| Total · pass mark 70 | 100 |
Submitting session 13
Write your answer in Word or Google Docs. The first line must read
Write your answer in Word or Google Docs. The first line must read
Session: 13, or the submission is refused. Upload the Word file (.docx), or switch to the Google Doc tab and paste your document’s share link.Marked against the rubric above and returned as a written letter, saved to your progress page. Do not submit until you have passed the self-check above.ASSESSMENT NOW OPEN
Pop Quiz 3 — Generative Systems and the Plant opens now. Sit it before you start the next session — it is how you find out whether the doctrine landed, and the practice papers may be resat as often as you like.
Sit Pop Quiz 3 →Closed book, no notes, and Coach Ritchie switches himself off while you sit it.Capabilities Covered
Advisory versus controlEngineering authorityOT boundaryManagement of changePrivacy beyond redaction
Operations · Chapters 23 and 24SESSION 14
Human Oversight and Buying AI
Why This Matters
Two comfortable phrases end many AI debates in executive meetings: a human will review it, and the vendor has tested it. Neither is evidence on its own. Oversight is a workflow property: a competent person with the relevant information, enough time and the authority to change the outcome. Procurement is the moment your company either makes a vendor's uncertainty visible or buys it silently. Today you learn to size review capacity, read override rates honestly, give people a real route to challenge, and turn a vendor conversation into a record of evidence accepted, commitments obtained and concessions accepted by a named person. Both halves share one test: can the control actually be exercised when it matters.
25-Minute Module
0:00 – 6:00
Oversight is a workflow property
- Human oversight requires a competent person with relevant information, enough time and the authority to affect the outcome. A person who clicks approve without seeing the material fields is present but may provide little protection.
- The display must match the decision. A technical recommendation shows the applicable procedure, source revision, relevant measurements and known limitations; a transaction shows recipient, amount, destination and effect.
The book's review-capacity illustration Figure Consequential proposals an agent produces each hour 500 Proposals the team can examine each hour 30 Proposals left unexamined each hour (500 − 30) 470 Share examined (30 ÷ 500) 6% The book's conclusion The design must limit output or narrow authority - THE DOCTRINE
A policy requiring review cannot manufacture capacity.
- Test review under realistic workload with deliberately seeded errors reflecting the harm analysis; measure detection, correction time and inappropriate approvals. Seeded errors are test artefacts. Separate reviewer accuracy from model accuracy.
Ask how many reviews an hour the design assumes, and how many your people can actually do.
6:00 – 12:00
Automation bias and the right to challenge
- People defer to a confident system or relax after repeated success. Design for examination, not rapid approval, and avoid ritual justification boxes filled with stock phrases.
- An override rate of zero is a diagnostic question: a reliable system, weak reviewers, low-risk cases or a poor override mechanism. A high rate also has several readings. Examine sampled decisions and known-error tests.
- Independent evidence. Give reviewers a route to source records independent of the model's summary; if the assistant supplies both recommendation and evidence, one retrieval error shapes the whole review.
- Contestability. For consequential decisions about people, provide someone able to reconsider, and say how a concern is submitted. A human alternative that cannot handle the workload or repeats the score is no remedy. Never penalise a challenger.
- THE DOCTRINE
Give operators permission to suspend reliance when a safety or integrity concern appears. The stop should preserve evidence and trigger review, not require the operator to prove a population-level defect first.
- OPERATIONS · THE WORKED CASE
- Operations. At a fictional terminal, a supervisor sees a ranking that contradicts the readings in front of her and suspends reliance. Her stop preserves evidence and opens a review.
ENTERPRISE · THE PARALLEL- Enterprise. A candidate challenges a screening outcome; the reviewer examines the basis and new information, not the same score. Both routes fail if people fear retaliation.
Ask when someone last stopped reliance on a system here, and what happened to them afterwards.
12:00 – 17:00
Buy evidence, not claims
- A model service, an embedded scoring feature, a retrieval platform and a decision agent carry different risks. Ask for the architecture and action boundaries, not a feature list.
- Vendor evidence must fit the proposed deployment. A general benchmark does not establish performance on a Nigerian enterprise's controlled documents, equipment records or recruitment process. A service-level agreement about uptime does not establish factual correctness.
- Appendix E groups the questions requiring evidence: purpose, performance, data, security, change, assurance and continuity. Ask whether evidence concerns the base model or the complete service, and distinguish management-system certification from task performance.
- Where weights or training data are unavailable, record the limitation and seek alternative evidence. It need not prohibit use, but it reduces confidence and permitted authority.
- A model name is not a fixed implementation. Where pinning is unavailable, keep a regression suite and a way to restrict use after a material change.
An aggregate accuracy figure without a target definition and a denominator is weak evidence.
17:00 – 22:00
Contract terms as operating controls
- Contract terms are operating controls. Test whether each right can be exercised: an audit right excluding the relevant subcontractor, or an exit right without data export, protects little.
Topic (Appendix E) Commitment sought Decision if unavailable Permitted purpose Defined data and action scope Narrow use or decline unsupported authority Material updates Notice and relevant configuration evidence Local regression and restriction plan Incident assistance Timely evidence and named response route Record investigation limits and exposure Data handling Specific retention and training-use terms Restrict information classes or select another service Audit evidence Relevant scope and practical access Alternative assurance with explicit limitation Exit Export, revocation and transition support Costed contingency or rejected dependency - Record achieved, partially achieved and conceded terms. For each material concession name the accepting authority, mitigating control and review date. Qualified counsel drafts enforceable clauses; the table is a negotiation method, not legal drafting. No contract transfers all accountability.
- Count total cost, including evaluation, assurance and exit, and map shared dependencies: several products may rely on one model provider.
- The book's procurement example: a document assistant whose model cannot be pinned is restricted to drafting, with controlled retrieval, regression checks and user verification before external use. That can be a reasoned decision. It is not approval of autonomous regulatory submissions.
- THE DOCTRINE
Good procurement makes uncertainty visible before the contract is signed and keeps the vendor relationship inside the release and monitoring process afterwards.
Every concession needs a name beside it and a date to look again.
22:00 – 25:00
Before you close the laptop
- Estimate how many consequential outputs your system produces in a working hour and how many your reviewers can genuinely examine.
- Name the route by which a person affected by your system can challenge it, and the person who can reconsider.
- Pick the three Appendix E questions your current or likely vendor would find hardest to answer with evidence.
- That is Section 14 of your plan: review capacity, contestability and the vendor evidence we will demand.
If the vendor's answer is a brochure, write down that the evidence is missing. That is a finding too.
Key Terms
Review capacity
The number of consequential outputs competent reviewers can genuinely examine in a period. Where output exceeds it, the design must limit output or narrow authority.
Automation bias
The tendency to defer to a confident system or become less vigilant after repeated success, which interface and workflow design must counter.
Contestability
A practical route for a person affected by a consequential decision to submit a concern and have someone able to reconsider the underlying basis and new information.
Material concession
A contract term sought but not obtained, recorded with the accepting authority, the mitigating control and a review date.
Self-Check — Before You Submit
1. Reviewers overrode the AI zero times last quarter. The sponsor calls this proof the model is excellent. Your response?
- A. Agree and reduce the review team
- B. Conclude that review has stopped and suspend the system
- C. Treat it as a diagnostic question and examine sampled decisions and known-error test results before drawing any conclusion
- D. Ask reviewers to override more often to show they are engaged
C. Treat it as a diagnostic question and examine sampled decisions and known-error test results before drawing any conclusion — Zero overrides can reflect a reliable system, weak reviewers, low-risk cases or a poor mechanism. Concluding success or failure is the trap; sampled decisions and seeded-error tests decide.
2. A vendor's document assistant cannot be pinned. The business wants it to file regulatory submissions without review. Which decision fits the book?
- A. Approve autonomous submissions, because the vendor's uptime commitment is strong
- B. Restrict it to drafting, with controlled retrieval, regression checks and user verification before external use, and record the concession
- C. Reject all use, because unpinned models can never be governed
- D. Approve, provided the contract transfers accountability to the vendor
B. Restrict it to drafting, with controlled retrieval, regression checks and user verification before external use, and record the concession — The book shows that restricting an unpinnable assistant to drafting can be a reasoned decision, but not approval of autonomous submissions. Uptime says nothing about correctness, and no contract transfers all accountability.
3. An agent produces 500 consequential proposals each hour; the review team can examine 30. What follows?
- A. The design must limit output or narrow authority, because only 6% can be examined
- B. A policy requiring review of all proposals is sufficient
- C. Reviewers should approve faster to clear the queue
- D. Sample a few proposals and approve the rest unexamined
A. The design must limit output or narrow authority, because only 6% can be examined — A policy requiring review cannot manufacture capacity. Faster approval or arbitrary sampling leaves most consequential proposals unexamined, so the design itself must change.
Today’s Assignment
Write Section 14 of your plan: the review capacity, contestability route and vendor evidence your company will demand for its AI system.
- Name the system and its reviewers. If your company has no AI system yet, use a proposal under consideration or a tool staff already use informally.
- Estimate output against genuine review capacity and state what reviewers must see.
- Describe the route to challenge and how operators can suspend reliance.
- Using Appendix E, list the evidence you will demand from the vendor and, for at least three negotiation topics, the decision if the commitment is unavailable.
- The honest part: where in your company is review currently a signature without the information, time or authority behind it. Aim for 400 to 900 words.
YOUR 90-DAY PLAN
Section 14 — Review capacity, contestability and the vendor evidence we will demand.
How this is marked — the rubric
| Quantifies output against review capacity and specifies what reviewers must see | 25 |
| Defines a workable contestability route and a protected right to suspend reliance | 25 |
| Specifies vendor evidence and decisions-if-unavailable for at least three Appendix E topics, with named accepting authorities | 30 |
| Identifies one honest case of review without capacity or authority | 20 |
| Total · pass mark 70 | 100 |
Submitting session 14
Write your answer in Word or Google Docs. The first line must read
Write your answer in Word or Google Docs. The first line must read
Session: 14, or the submission is refused. Upload the Word file (.docx), or switch to the Google Doc tab and paste your document’s share link.Marked against the rubric above and returned as a written letter, saved to your progress page. Do not submit until you have passed the self-check above.Capabilities Covered
Review capacity sizingOverride-rate diagnosisContestability designEvidence-based due diligenceNegotiation record
Assurance · Chapters 25 and 26SESSION 15
Monitoring, Incidents and Independent Assurance
Why This Matters
Approval is not the end of governance. It is a set of conditions that must remain true. This session closes Week 3 with the three disciplines that keep them true after launch: monitoring that watches outcomes and authority rather than uptime alone, incident response that contains before it explains everything, and independent assurance that tests what controls actually do. The book makes two points that leaders often miss. An AI incident can happen with no model defect at all. And restart is a new decision, with a named authority, not the end of an outage. By the end you will write your company's incident response, restart rule and assurance sample, and Practice Exam 3 opens.
25-Minute Module
0:00 – 5:00
Monitor the conditions of approval
- Monitoring determines whether the conditions of approval remain true. Availability and latency are necessary service measures, but they do not show that recommendations remain correct, lawful or within authority.
- Monitor inputs, outputs, actions, outcomes and control performance. Each indicator needs an owner and a response; a dashboard that turns red without changing authority or behaviour is incomplete.
- Delayed evidence. Preserve prediction time, score, threshold, configuration and outcome link. Report the delay and pending proportion; a rate from quickly resolved cases can be biased.
- Leading indicators such as rising abstention or reviewer corrections justify investigation, not conclusions. An incident-driven sample overrepresents failures.
For every red light on the dashboard, ask who changes what when it turns red.
5:00 – 10:00
Severity follows consequence; contain first
- The book's proposed severity system prioritises serious physical harm, material unauthorised action, sensitive disclosure and consequential adverse treatment. Lesser service defects follow an ordinary operational route. Specific legal reporting thresholds and deadlines must come from the applicable instruments.
- THE DOCTRINE
An AI incident can arise without a model defect. A correct answer delivered to an unauthorised user is an incident. A correct recommendation executed twice can be an incident. A retrieved obsolete procedure can cause harm even when faithfully used.
- The incident record separates confirmed facts from hypotheses. A confident early diagnosis can misdirect containment.
- Contain before explaining everything. Responders may restrict the capability, revoke credentials, isolate a source or return work to an approved process. A model rollback does not undo messages sent or decisions applied to people.
- OPERATIONS · THE WORKED CASE
- Operations. A fictional gas company's field assistant retrieves a superseded procedure and a crew follows it. No model defect; still an incident. Isolate the source and return work to the approved procedure.
ENTERPRISE · THE PARALLEL- Enterprise. Its legal assistant gives a correct answer from a document the user may not see: a sensitive disclosure. In both, severity follows consequence, not defect.
Ask your team to name the incident that happens when the model is right. If they cannot, show them this step.
10:00 – 15:00
The first hour, the runbook and restart
- The book's hypothetical first hour, a teaching scenario. A procurement agent sends a draft order to an unapproved external address after reading a supplier document. The responder stops new sends, revokes send permission and checks queued work; the team preserves the document, tool calls, approval state and identities, then establishes what left and where.
- The first hypothesis is prompt injection, but the investigation also examines address validation, permission scope and the review interface. Even if the document initiated it, an unrestricted send tool may be the consequential control failure.
Appendix G: respond What the step requires Activate Record the report; assign an incident commander; preliminary severity Contain Restrict, revoke, check queues; block automatic restart; approved fallback Preserve Retain input, output, tool, approval, identity and timing records Assess exposure Confirmed harm versus plausible exposure; reconcile actual actions, not the model's narrative Decide notifications Duties from current instruments and contracts; no one generic deadline Appendix G: recover What the step requires Repair consequences Correct records; reconcile duplicates; offer remedy Validate remediation Test fixes; verify permission, queue, memory, source and restart controls Reinstate or retire A named authority decides return within a defined scope, or retirement Learn Owned findings, verified closure, updated control catalogue and board report - Restart is a new decision, with a recorded authority and any temporary limits. A report that blames the model without examining architecture, incentives and authority is unlikely to prevent recurrence. Appendix G is proposed text: populate and rehearse it.
A report is complete when the operational actions and their evidence are recorded, not when the narrative is written.
15:00 – 21:00
Assurance that tests what controls do
- Independence is a governance arrangement, not another tool or model reviewing the work. Define scope, criteria, population, sampling and limitations; a policy audit is not a technical validation.
- Walk through a real decision. Trace one outcome backwards to authority, configuration, inputs, revisions and approval, comparing operated and tested states. Then forward: is monitoring adequate, are expiry triggers enforced, can a responder stop it?
- Sample controls, not only documents. Observe a permission denial, a source withdrawal, a rollback. A policy budget limit may not be enforced. A clean small sample is no enterprise-wide guarantee.
- Reproducibility. Preserve what is needed to rerun the evaluation, or for hosted services the identifiers, prompts, evidence and outputs. A perfectly repeatable wrong calculation remains wrong.
- Findings need an owner, due date, interim restriction and closure evidence verified by someone able to challenge it. A well-documented but ineffective control is still a finding.
- THE DOCTRINE
Phrases such as independently reviewed are incomplete without who reviewed what, against which criteria and with which access. Precise scope builds credibility more effectively than an impressive label.
A management-system auditor has not validated model safety unless that was actually in scope. Ask.
21:00 – 25:00
Your runbook, and Practice Exam 3
- Name your company's incident commander for AI, the first containment action for your system, and who may authorise restart.
- Choose one released decision to walk backwards and forwards, and three controls to observe operating rather than read about.
- That is Section 15 of your plan: our incident response, restart rule and assurance sample.
- Practice Exam 3 opens after this session: fifteen questions covering Sessions 11 to 15, closing Week 3. Sit it before Session 16.
Rehearse one failure path this month. A runbook nobody has rehearsed is a draft.
Key Terms
Leading indicator
An early signal, such as rising abstention or reviewer corrections, that warrants investigation but does not replace outcome evaluation.
Incident commander
The person assigned under a defined authority to direct an incident response from activation through containment, preservation and exposure assessment.
Two-direction walkthrough
An assurance method that traces one consequential outcome back to its authority, configuration, inputs and approval, then forward to monitoring, expiry, scope limits and the ability to stop.
Assurance scope
The stated criteria, population, sampling, access and limitations of a review, which determine what its conclusion can and cannot support.
Self-Check — Before You Submit
1. An agent sent a draft order to an unapproved address. The team rolled back the model and wants to restart today because the vendor says it is patched. What do you require?
- A. Restart now, because the rollback removed the cause
- B. Restart now and monitor closely for a week
- C. Keep the service contained until the vendor provides a written explanation
- D. Treat restart as a new decision: reconcile actions, test fixes including send permissions, and have the named authority approve return
D. Treat restart as a new decision: reconcile actions, test fixes including send permissions, and have the named authority approve return — A rollback does not undo messages already sent, and an unrestricted send tool may be the real control failure. A vendor explanation alone is the tempting wrong answer; restart needs tested corrective actions and a named authority.
2. A report shows strong accuracy calculated only on cases with known outcomes; most recent cases are pending. What is the problem?
- A. None, because known outcomes are the most reliable evidence
- B. A rate calculated only from quickly resolved cases can be biased; the report should state the delay and the proportion of labels still pending
- C. Pending cases should be counted as correct until proven otherwise
- D. The report should be replaced by availability and latency figures
B. A rate calculated only from quickly resolved cases can be biased; the report should state the delay and the proportion of labels still pending — Delayed outcomes may not be a random subset, so excluding them can distort performance. Treating pending cases as correct hides the gap, and uptime measures say nothing about correctness.
3. A vendor's pack says the product was independently reviewed, citing a management-system certification. What should the board be told?
- A. Who reviewed what, against which criteria and access, and that it does not show model safety unless in scope
- B. That the product has been independently validated as safe
- C. Nothing, because certification is a technical matter
- D. That certification transfers assurance responsibility to the vendor
A. Who reviewed what, against which criteria and access, and that it does not show model safety unless in scope — The board should receive the assurance conclusion and its limits. A management-system auditor should not be represented as having validated model safety unless that was in scope, so the reassuring label is the trap.
Today’s Assignment
Write Section 15 of your plan: the incident response, restart rule and assurance sample for the AI system in your company.
- Name the system. If your company has no AI system yet, use a proposal under consideration or a tool staff already use informally.
- Write three monitoring indicators with an owner and the response each triggers, and how delayed outcomes are reported.
- Adapt Appendix G: activation route, incident commander, first containment action, notification owners and reinstatement authority.
- Define your assurance sample: one decision to walk in both directions, three controls to observe operating, and who will review with what competence and independence.
- The honest part: describe the last time an issue in your business was declared fixed when only the description changed, or was restarted without anyone named as the authority. Aim for 400 to 900 words.
YOUR 90-DAY PLAN
Section 15 — Our incident response, restart rule and assurance sample.
How this is marked — the rubric
| Defines monitoring indicators with owners, responses and treatment of delayed outcomes | 20 |
| Adapts the runbook with a named commander, a first containment action and a named restart authority | 30 |
| Specifies a two-direction walkthrough, three observed controls and the reviewer's competence and independence | 30 |
| Gives one honest case of description-only closure or unauthorised restart | 20 |
| Total · pass mark 70 | 100 |
Submitting session 15
Write your answer in Word or Google Docs. The first line must read
Write your answer in Word or Google Docs. The first line must read
Session: 15, or the submission is refused. Upload the Word file (.docx), or switch to the Google Doc tab and paste your document’s share link.Marked against the rubric above and returned as a written letter, saved to your progress page. Do not submit until you have passed the self-check above.ASSESSMENT NOW OPEN
Practice Exam 3 — Generative Systems, Security and Operations opens now. Sit it before you start the next session — it is how you find out whether the doctrine landed, and the practice papers may be resat as often as you like.
Sit Practice Exam 3 →Closed book, no notes, and Coach Ritchie switches himself off while you sit it.Capabilities Covered
Outcome monitoringConsequence-based severityContainment firstRestart authorityControl sampling