Every chapter of the DMBOK2, summarised with worked examples, sample data and diagrams. Pick a chapter from the selector on the left, or select an area on the DAMA wheel.
01The DAMA wheel and your performance
90%+ ✓80–89%70–79%Under 70%
Data Governance sits at the centre because it guides and oversees every other area. The ten knowledge areas around it are the functions that manage data through its lifecycle.
Select any segment to open that chapter. Chapters 1 and 2 set the foundations, and chapters 14 to 17 cover topics that cut across the wheel.
After each chapter quiz, that chapter's segment on the wheel and its bar on the chart take the colour of your latest score. Use the chapter menu on the left to jump to any chapter.
02Exam weighting and your scores
The CDMP Fundamentals exam has 100 questions in 90 minutes, weighted by knowledge area. Pass marks are 60% for Associate, 70% for Practitioner and 80% for Master. Each chapter's latest and best quiz scores are saved here after every completed quiz.
03Your mistakes bank
Every question you answer wrongly in any quiz is kept here until you get it right. Practise them in exam mode or practice mode, for all chapters or one at a time.
04Chapter index
CDMP exam weighting:11%10%6%2%Not weighted
Weights are the CDMP Fundamentals exam percentages as commonly published by DAMA for DMBOK2, across 14 areas. Check DAMA's current exam guidance before relying on them.
Foundations
Knowledge areas
Beyond the wheel
05The running example
Worked examples throughout the guide use the same small, fictional business so that each chapter builds on the last.
Customers
ID
Name
Country
C1
Northwind Ltd
GB
C2
Contoso SA
FR
Products
ID
Name
Price
P10
Laptop
899.00
P20
Mouse
19.99
P30
Monitor
249.00
Orders
Order
Customer
Lines
1001
C1
Laptop ×1, Mouse ×2
1002
C2
Monitor ×1
1003
C1
Mouse ×5
In June 2026, Northwind Ltd renames itself Northwind Group Ltd. Several chapters use this change to show how history, quality and governance work. The data consultancy staff Asha, Ben and Chloe also appear in examples.
06How to use this guide
Chapter selector: the button at the top left. Open it and type to filter, use the arrow keys, then press Enter.
Section menu: under the selector, the current chapter's sections. Arrows open and close each group; the menu follows you as you scroll.
Previous and Next: step through chapters in order, from the sidebar or the bottom of each chapter.
Labels: material marked Beyond DMBOK is supplementary context, such as newer practices, rather than a summary of the book.
Links to RDM and DQ: each chapter ends with how it connects to reference data management and data quality.
Chapter 1 · Foundations
Data Management
What data management is, why data is treated as an asset, the principles and challenges involved, and the frameworks DAMA uses to organise the whole discipline.
01Introduction
Definition
Data management is the development, execution and supervision of plans, policies, programmes and practices that deliver, control, protect and enhance the value of data and information assets throughout their lifecycles.
Almost every business process depends on data. Organisations that manage data well can use it to compete, make decisions and meet regulatory duties. Those that don't carry hidden costs: rework, duplicated effort, poor decisions and compliance risk.
Goals of data management
Understand and support the information needs of the enterprise and its stakeholders
Capture, store, protect and ensure the integrity of data assets
Ensure the quality of data and information
Ensure the privacy and confidentiality of stakeholder data
Prevent unauthorised or inappropriate access, manipulation or use of data
Ensure data can be used effectively to add value to the enterprise
Data as an asset
An asset is an economic resource that can be owned or controlled and that holds or produces value. Data meets that test, but it behaves differently from physical assets:
Property
Physical asset (a delivery van)
Data asset (a customer list)
Consumption
Used up or worn out by use
Not consumed by use; can be used by many people at once
Copying
Can't be copied
Copied easily and cheaply, which makes it hard to control
Location
In one place at a time
Can exist in many places at once, often inconsistently
Value over time
Depreciates on a schedule
Can gain value when combined with other data, or lose it quickly when out of date
On the balance sheet
Yes
Rarely, which is why its value is often invisible
02Essential concepts
Data, information and knowledge
The DMBOK cautions against treating these as a strict pyramid, but the distinction is still useful. Data is a representation of facts. Information is data in context, with meaning. Knowledge is understanding built from information, which lets people act.
Data
2026-03-02, C1, P10, 1
Values with no context.
Information
Northwind bought one laptop on 2 March
The same values, with definitions and context.
Knowledge
Northwind buys a laptop each quarter
A pattern people can act on, such as a timely sales call.
Data management principles
The DMBOK sets out a series of principles. In summary:
Data is an asset with unique properties
It isn't consumed by use and is easily copied, so it needs its own management approach.
Its value can be expressed in economic terms
Even approximate valuations help prioritise investment.
Managing data means managing quality
Data has to be fit for purpose, or it has no value.
It takes metadata to manage data
You can't manage what you can't describe.
It takes planning
Data flows across many processes and systems, so it must be designed, not left to chance.
It is cross-functional
It needs business and technical skills working together.
It requires an enterprise perspective
Local optimisation creates silos and duplication.
It must account for many perspectives
Different users need the same data in different ways.
It is lifecycle management
Data must be managed from creation to disposal.
Different data has different lifecycles
Reference, master and transactional data age differently.
It includes managing risk
Data can be lost, stolen or misused, and it carries legal duties.
Requirements drive technology
Business needs for data should shape IT decisions, not the other way round.
It requires leadership commitment
It needs sustained sponsorship to succeed.
Challenges
Data is hard to value because it doesn't appear on balance sheets and its value depends on use.
Data quality problems are often invisible until something fails.
Data is spread across many systems and copies drift apart.
Ownership is unclear, because data crosses departmental boundaries.
Business and IT see data differently, and both views are needed.
Metadata is often missing, so data can't be found or understood.
Risks are real: breaches, regulatory fines and reputational damage.
Worked example: the cost of poor data
The 1-10-100 rule (Labovitz and Chang) is often quoted: it costs about 1 unit to prevent a data error at entry, 10 to correct it later, and 100 if it is never fixed and causes failure downstream.
Northwind's address errors, illustrative figures
Approach
Records affected
Cost per record (£)
Total (£)
Prevent: validate postcode at entry
2,000
1
2,000
Correct: clean up in a quarterly batch
2,000
10
20,000
Fail: returned deliveries and lost orders
2,000
100
200,000
The ratios are a rule of thumb, not a measurement. The point is that fixing data at source is far cheaper than fixing its consequences.
Ways to value data
Approach
The question it asks
Example
Replacement cost
What would it cost to recreate the data if it were lost?
Re-keying 50,000 customer records at £2 each = £100,000
Market value
What would someone pay for it?
The price of a comparable purchased contact list
Identified opportunities
What revenue or savings does it enable?
A cross-sell campaign that depends on clean purchase history
Selling data
What income could it earn as a product?
Anonymised market data sold to partners
Risk cost
What penalties or losses would poor or missing data cause?
Regulatory fines for inaccurate reporting
The data lifecycle
Data moves through a lifecycle. Each stage has its own management needs, and data can be used, enhanced and shared many times before it is disposed of.
Plan→Design and enable→Create or obtain→Store and maintain→Use→Enhance→Dispose
Worked example: Northwind's customer record through its lifecycle
Stage
What happens
Data management concern
Plan
Sales need a single customer view
Requirements, ownership
Design and enable
Customer entity modelled; CRM configured
Modelling, architecture, standards
Create or obtain
C1 "Northwind Ltd" entered in CRM, March 2026
Quality at entry, reference data for country
Store and maintain
Held in CRM, copied to billing and the warehouse
Integration, security, backup
Use
Invoices, sales reports, marketing
Access control, metadata, quality
Enhance
Renamed to Northwind Group Ltd; credit rating added
History, master data, stewardship
Dispose
Archived or deleted after the retention period ends
Retention rules, privacy law
Types of data
Data can be classified by its role, and each type needs different handling.
Type
Description
Running example
Reference data
Code lists used to classify other data; small and slow-changing
Country codes GB, FR
Master data
Core business entities shared across processes
Customer C1, Product P10
Transactional data
Business events
Order 1001
Metadata
Data that describes data
"CustomerName: legal trading name, max 200 characters, owner Sales Ops"
Unstructured data
Documents, images, email and other content without a fixed model
Signed contract PDF for Northwind
03Data management frameworks
Frameworks help people see how the parts of data management fit together. The DMBOK describes several.
Strategic Alignment Model
Henderson and Venkatraman's model aligns four domains: business strategy, IT strategy, organisational infrastructure and processes, and IT infrastructure and processes. In data terms, it argues that data and information sit between business and IT, and must be aligned with both.
Amsterdam Information Model
The Amsterdam Information Model (Abcouwer, Maes and Truijens) takes the same idea further. It adds a middle layer for information management between business and technology, organised into strategy, structure and operations. Data management sits in that middle column.
The DAMA wheel
The DAMA-DMBOK framework places Data Governance at the centre, surrounded by ten knowledge areas: Data Architecture, Data Modeling and Design, Data Storage and Operations, Data Security, Data Integration and Interoperability, Document and Content Management, Reference and Master Data, Data Warehousing and Business Intelligence, Metadata, and Data Quality. The wheel shows that all areas are needed and that governance holds them together. It doesn't imply an order of implementation. See the home page for an interactive version.
The environmental factors hexagon
Every knowledge area is shaped by the same seven factors. Goals and principles sit at the centre; the other six surround them.
Environmental factors
The factors fall into three groups: people (roles and responsibilities; organisation and culture), process (activities; deliverables) and technology (practices and techniques; technology).
Knowledge area context diagrams
Each chapter of the DMBOK has a context diagram summarising the area in the same shape: definition and goals at the top, then inputs, activities and deliverables, then suppliers, participants and consumers, then techniques, tools and metrics. It works much like a SIPOC diagram (suppliers, inputs, process, outputs, customers). Every chapter in this guide ends with a context summary in that format.
The DMBOK pyramid (Aiken)
Peter Aiken's view describes how organisations typically mature, building from the bottom up:
Phase
What organisations do
Knowledge areas involved
Phase 1
Buy or build applications that use databases
Modeling and design, storage and operations, security
Phase 2
Discover quality problems as data is integrated and reused
Data quality, integration, metadata, architecture
Phase 3
Recognise the need for formal oversight
Data governance
Phase 4
Exploit well-managed data for advanced uses
Reference and master data, warehousing and BI, document management, big data
Aiken's point is that governance should come earlier than it usually does. Organisations often only discover they need it after quality problems hurt.
Functional area dependencies
The DMBOK also shows how areas depend on each other. Governance enables everything. Architecture, modelling, storage, security and integration are foundations. Quality, metadata and master data are activities that make data fit for use. Warehousing, BI, document management and big data are the uses that depend on those foundations.
04Data management strategy
A data management strategy is a plan for maintaining and improving the quality, integrity, security and access of data. It should support the business strategy.
Components of a strategy
A compelling vision for data management
A summary business case, with selected examples
Guiding principles, values and management perspectives
The mission and long-term goals
Proposed measures of success
Short-term objectives (12 to 24 months) that are specific, measurable, actionable, realistic and time-bound
Descriptions of roles and organisations, with responsibilities and decision rights
Descriptions of the programme's components and initiatives
A prioritised work programme with scope boundaries
A draft implementation roadmap with projects and actions
Typical deliverables
Charter
Data management charter
Vision, business case, goals, principles, measures of success, critical success factors and recognised risks.
Scope
Scope statement
Goals and objectives for a planning horizon, roles and organisations, and the leaders accountable.
Roadmap
Implementation roadmap
Specific programmes, projects, task assignments and delivery milestones.
Worked example: a one-page strategy
Northwind's data strategy on a page (illustrative)
Element
Content
Vision
Every customer and product decision uses one trusted version of the data.
Business case
Invoice disputes cost £180,000 a year; 40% trace back to inconsistent customer and country data.
Principles
Fix data at source. One owner per data domain. Reference data is managed centrally.
12-month objectives
Governed country and product code lists; customer duplicates below 2%; disputes down 25%.
Measures
Duplicate rate, validity score for critical data elements, dispute volume.
Data is an asset with unusual properties, and it must be managed across its whole lifecycle.
Key idea
Data management is cross-functional and needs both business and technical perspectives.
Key idea
The DAMA wheel and the environmental factors hexagon give a common structure for every knowledge area.
Key idea
A data management strategy should be driven by business value and expressed as a roadmap.
06Links to reference data and data quality
"Managing data means managing quality" is one of the core principles, so data quality isn't an optional extra.
"Different types of data have different lifecycles" is the DMBOK's own case for treating reference data separately: it is small, shared and slow-changing, but errors in it spread everywhere.
The 1-10-100 rule is a simple way to make the business case for fixing data at the point of entry, often by validating against governed reference data.
Chapter 2 · Foundations
Data Handling Ethics
How to acquire, store, manage, use and dispose of data in ways that are fair, respectful and lawful, and the practical risks that arise when organisations get it wrong.
01Introduction
Definition
Data handling ethics is concerned with how to procure, store, manage, use and dispose of data in ways that are aligned with ethical principles, including community responsibility.
Ethics is about doing the right thing even when the law doesn't require it, or hasn't caught up. Data can affect people's lives: credit decisions, job applications, medical care, prices and what news they see. Mishandled data can cause real harm, even without bad intent.
Business drivers
Trust: customers, employees and partners share data only with organisations they trust.
Reputation: ethical failures make headlines and destroy brand value fast.
Regulation: privacy laws impose large fines, and regulators look at intent as well as compliance.
Better decisions: ethical practice, such as challenging bias, also produces more accurate analysis.
Goals
Define ethical handling of data in the organisation
Educate staff on the organisation's risks if data is handled unethically
Change or instil preferred culture and behaviours
Monitor regulatory environment, measure, monitor and adjust organisational approaches
02Essential concepts
Ethical principles for data
The DMBOK draws on the Belmont Principles, originally written for research on human subjects, and the Menlo Report, which adapted them for information and communications technology research.
Principle
Meaning
Applied to data
Respect for persons
Treat people as autonomous; protect those with reduced autonomy
Informed consent; clear choices; extra care with children's data
Beneficence
Do no harm; maximise benefits and minimise harms
Assess the potential harm of a new use before starting it
Justice
Distribute benefits and burdens fairly
Check that models don't disadvantage particular groups
Respect for law and public interest
Added by the Menlo Report
Be transparent and accountable; comply with law
Principles behind privacy law
Many privacy laws share roots in the Fair Information Practice Principles (FIPPs) and the OECD privacy guidelines. The EU and UK General Data Protection Regulation (GDPR) sets out seven principles:
GDPR principle
What it requires
Northwind example
Lawfulness, fairness and transparency
Have a legal basis; be open about processing
The privacy notice explains how order data is used
Purpose limitation
Collect for specified purposes; don't reuse incompatibly
Delivery addresses aren't reused for unrelated profiling
Data minimisation
Collect only what is needed
No date of birth for business customers
Accuracy
Keep data accurate and up to date
Contact details are checked yearly
Storage limitation
Keep data no longer than necessary
Lapsed contacts are deleted after 3 years
Integrity and confidentiality
Protect against loss and unauthorised access
Encryption and role-based access
Accountability
Be able to demonstrate compliance
Records of processing; data protection impact assessments
GDPR also requires a lawful basis for each processing activity: consent, contract, legal obligation, vital interests, public task or legitimate interests. Other regimes covered in the DMBOK include Canada's PIPEDA and US sector laws and FTC guidance.
Risks of unethical data practice
The DMBOK lists ways data can mislead or harm, often without anyone intending it.
Timing
Manipulating timing
Releasing or omitting data at moments chosen to distort perception, such as burying bad results.
Visualisation
Misleading charts
Truncated axes, distorted scales or cherry-picked ranges.
Definitions
Unclear definitions
Comparing figures calculated differently, as if they matched.
Bias
Bias
Conclusions shaped by how data was collected, selected or analysed.
Integration
Transforming and integrating data
Combining datasets in ways that reveal more than any one dataset should.
Privacy
Weak obfuscation
Masking or anonymising badly, so people can be re-identified.
Worked example: a misleading chart
Customer complaints rose from 96 in Q1 to 104 in Q2, an 8% increase. The same two numbers look very different depending on where the axis starts.
Axis starts at 90: looks like complaints doubledAxis starts at 0: an honest 8% rise
Neither chart contains false numbers. The left one still misleads, which is why the DMBOK treats presentation as an ethical issue.
Types of bias
Bias
What happens
Example
Data collection for pre-defined result
Analysis is designed to reach an answer already decided
A study commissioned to "prove" a campaign worked
Biased use of data collected
Data is manipulated or selected to fit a conclusion
Dropping awkward months from a trend
Hunch and search
An analyst searches the data until something confirms their hunch
Testing 50 segments and reporting the one that "works"
Biased sampling
The sample doesn't represent the population
Surveying only customers who reply to email
Context and culture
Assumptions built into the data reflect one group's view
A gender field with only two options
Worked example: re-identification and k-anonymity
Removing names isn't enough. Combinations of ordinary fields, called quasi-identifiers, can single people out. Postcode, date of birth and gender together identify a large share of individuals.
"Anonymised" extract: each row is unique, so anyone who knows these facts can identify the person
Postcode
Birth date
Gender
Spend band
M1 4BT
1985-04-12
F
High
M1 4BT
1986-11-03
F
Low
M1 7ED
1990-02-28
M
Medium
M1 7HU
1991-07-15
M
High
Generalise the quasi-identifiers until every combination appears at least k times. Here k = 2:
2-anonymous extract: every person is indistinguishable from at least one other
Postcode district
Birth years
Gender
Spend band
M1
1985–1989
F
High
M1
1985–1989
F
Low
M1
1990–1994
M
Medium
M1
1990–1994
M
High
k-anonymity has limits. If both people in a group had the same spend band, knowing someone is in that group would still reveal it. Techniques such as l-diversity and differential privacy address this. Beyond DMBOK
Obfuscation and redaction techniques
Masking a Northwind contact record
Technique
Original
Masked
Notes
Character masking
4929 1234 5678 9012
**** **** **** 9012
Keeps the last digits for recognition
Substitution
Sarah Jones
Emma Clarke
Realistic but fictional values
Shuffling
Salaries in a column
Same values, reassigned to other rows
Keeps totals, breaks the link to people
Nulling
07700 900123
(null)
Removes the value entirely
Generalisation
1985-04-12
1985
Reduces precision
Tokenisation
sarah@example.com
tok_8f3a2c
Reversible only through a secure token vault
See Chapter 7 (Data Security) for persistent versus dynamic masking.
Ethical risk model for sampling and analysis
The DMBOK suggests checking a project against risk factors before starting. Each "yes" raises the ethical risk and calls for closer review.
Worked example: a churn-prediction project
Question
Answer
Risk
Does it use personal or sensitive data?
Yes, purchase and contact history
Raised
Could the result affect people's treatment?
Yes, offers and prices may differ
Raised
Is the data used for a new purpose?
Partly; order data is reused for prediction
Check lawful basis
Could the sample under-represent a group?
Possibly; few small-business customers
Test for bias
Is the logic explainable to those affected?
Yes, simple scorecard model
Lower
Is there human review of decisions?
Yes, account managers approve offers
Lower
03Activities
Review current-state data handling practices. Understand how data is collected, used and shared today, and where the risks are.
Identify principles, practices and risk factors. Agree the organisation's ethical principles and what they mean in practice.
Create an ethical data handling strategy and roadmap. Include values, a code of conduct, compliance framework, risk assessment, training and monitoring.
Address practices and gaps. Change processes that don't meet the principles.
Communicate and educate staff. Make expectations clear and give people the means to raise concerns.
Adopt a socially responsible approach to data, including data science and machine learning.
Worked example: an ethical data code
Northwind data principles
1. We use customer data only for purposes customers would reasonably expect, or have agreed to.
2. We collect the minimum data we need and delete it when we no longer need it.
3. We test models and reports for unfair outcomes before using them to make decisions about people.
4. A person reviews any automated decision that significantly affects a customer or employee.
5. Anyone can raise a concern about data use without fear, through the data ethics inbox.
04Ethics, AI and automated decisions Beyond DMBOK
Since the DMBOK2 was published, machine learning and AI agents have raised the stakes. Systems can now act on data at speed and scale, so ethical weaknesses spread faster.
Explainability: people affected by a decision should be able to understand it. GDPR gives rights around solely automated decisions with legal or similarly significant effects.
Bias amplification: models trained on historical decisions can reproduce past unfairness.
Human oversight: scale it to impact and reversibility. High-impact, irreversible decisions need meaningful human review, not rubber-stamping.
Regulation: the EU AI Act classifies some uses, such as credit scoring and recruitment, as high risk, with extra obligations.
05Context summary
Inputs
Existing and emerging regulation, business strategy, organisational values, data handling practices.
Deliverables
Ethical principles and code of conduct, ethical handling strategy, risk assessments, training, monitoring.
Participants
Executives, data governance, legal and compliance, data protection officer, data scientists, stewards.
Metrics
Training completion, number of reported issues and how they were resolved, impact assessments completed, compliance audit findings.
06Links to reference data and data quality
Accuracy is a legal duty under GDPR, not just a quality target. Poor data quality can be an ethical and regulatory failure.
Reference data carries assumptions. A gender code list with two values, or a nationality list that omits some states, embeds a worldview into every system that uses it. Code list design is an ethical choice.
Unclear definitions are listed as an ethical risk. Governed business glossaries and reference data are the remedy.
Chapter 3 · Knowledge area (centre of the wheel)
Data Governance
The exercise of authority and control over data: who decides, who is accountable, which rules apply, and how issues get resolved. It sits at the centre of the DAMA wheel because it guides every other area.
01Introduction
Definition
Data governance is the exercise of authority, control and shared decision-making (planning, monitoring and enforcement) over the management of data assets.
Governance versus management
The DMBOK separates the two. Governance makes sure data is managed properly; data management does the managing. A common analogy is finance: accountants manage the money, and auditors oversee that it is managed correctly.
Data governance
Data management
Oversight: "doing the right things"
Execution: "doing things right"
Sets policy, standards and decision rights
Follows them in daily work
Resolves issues and escalations
Raises issues and implements fixes
Measures compliance and value
Delivers data services and improvements
Business drivers
Reducing risk: general risk management, data security, privacy and regulatory compliance.
Improving processes: data quality, metadata management, project efficiency and vendor management.
Regulatory compliance is often the first driver, but value creation should follow.
Goals and principles
Data governance should be sustainable, embedded in normal processes, and measured. The DMBOK's guiding principles include:
Leadership and strategy
Successful governance starts with committed leadership and a data strategy.
Business-driven
It's a business programme that governs IT decisions about data, not an IT project.
Shared responsibility
Business stewards and technical data professionals share accountability.
Multi-layered
It happens at enterprise and local levels, and usually in between.
Framework-based
An operating framework defines roles and how they interact.
Principle-based
Principles guide decisions where policies don't cover a situation.
02Essential concepts
Governance organisation
A governance programme needs bodies at different levels. Typical ones:
A typical governance operating model
Body
Who
Responsibilities
Data Governance Steering Committee
Senior executives
Highest authority; approves strategy and funding; oversees the council
Data Governance Council
Business and IT leaders, data owners
Manages initiatives, policies and metrics; resolves escalated issues
Data Governance Office (DGO)
Small permanent team
Runs the programme day to day: standards, glossary, communications, metrics
Data Stewardship Teams
Business and technical stewards for a domain
Define data, set rules, approve changes, resolve domain issues
Local Data Governance Committee
Division or department leads
Governance within a unit, aligned to enterprise policy
Operating model types
Model
How it works
Suits
Watch-out
Centralised
One organisation governs data across all business units
Smaller or highly regulated organisations
Can be slow and remote from local needs
Replicated
Each business unit runs the same model and standards independently
Autonomous units with similar needs
Inconsistency creeps in across units
Federated
A central body sets enterprise policy; units govern locally within it
Large, diverse enterprises
Needs clear decision rights between levels
Data stewardship
Stewardship is the accountability and responsibility for data and processes that ensure effective control and use of data assets. Stewards are usually business people who do this as part of their job.
Role
Focus
Northwind example
Chief Data Steward
May chair governance bodies or act as the CDO's deputy
Head of Data Governance
Executive Data Steward
Senior manager who sits on the council
Sales Director, accountable for customer data
Enterprise Data Steward
Oversees a domain across business functions
Customer Master Data Lead
Business Data Steward
Subject expert accountable for a subset of data
Asha, for customer addresses and country codes
Data Owner
Business data steward with approval authority for decisions about data in their domain
Finance Controller, owner of the currency code list
Technical Data Steward
IT professional in a knowledge area, such as a DBA or integration specialist
Ben, CRM data administrator
Coordinating Data Steward
Leads and represents teams of stewards in cross-domain discussions
Chloe, coordinating customer and finance stewards
Worked example: a RACI for customer data
R = responsible (does the work), A = accountable (one per row), C = consulted, I = informed.
Decision or task
Sales Director (owner)
Business steward
Technical steward
DGO
Council
Define "active customer"
A
R
C
C
I
Approve a new customer segment code
A
R
C
I
I
Set data quality thresholds
A
R
C
C
I
Fix a duplicate in CRM
I
A
R
I
–
Resolve a cross-domain definition dispute
C
C
I
R
A
Publish the customer data policy
C
C
I
R
A
Data policies
Policies are short statements of principle and management intent. They say what must be done and by whom, not exactly how. Standards and procedures supply the detail.
Example policy: reference data
Purpose. To make sure shared code lists are consistent across all systems.
Policy. Each enterprise reference data set (such as countries, currencies and product categories) has one named owner and one authoritative source. Systems must take reference data from the authoritative source and must not maintain local copies with different values. Changes are requested through the reference data change process and approved by the owner.
Applies to. All business units and systems that store or exchange coded values.
Owner. Data Governance Council. Reviewed annually.
Business glossary
A business glossary is a governed list of business terms with agreed definitions. It is one of the most visible governance deliverables.
Glossary entry
Field
Value
Term
Active Customer
Definition
A customer with at least one invoiced order in the last 12 months.
Not to be confused with
Registered Customer (any customer with an account, active or not)
Calculation
COUNT(DISTINCT CustomerID) where InvoiceDate ≥ today minus 12 months
Owner
Sales Director
Steward
Asha
Status
Approved, 2026-04-10
Used in
Monthly sales dashboard; churn model
Issue management and escalation
Governance provides a route for data issues that can't be solved locally. Most should be settled at the lowest level possible.
Raised by any user→Stewardship team→Governance Council→Steering Committee
Issue log extract
ID
Issue
Type
Owner
Status
DG-014
Sales and Finance count customers differently
Definition
Council
Escalated
DG-015
CRM uses "UK", billing uses "GB"
Reference data
Asha
Resolved
DG-016
Who can approve new product categories?
Decision rights
DGO
Open
DG-017
Marketing wants to reuse support tickets for profiling
Ethics and privacy
Council
Open
Typical issue types: authority, change management and escalation, compliance, conflicts, conformance to policy and standards, contracts, data security and identity, and data quality.
Data asset valuation
Governance should help the organisation understand the value of its data, using the approaches in Chapter 1. Even rough valuations help prioritise which data to govern first.
03Activities
1. Define data governance for the organisation
Develop the data governance strategy, aligned to business strategy.
Perform a readiness assessment: data management maturity, capacity to change, collaborative readiness, and business alignment.
Perform discovery and business alignment: find the business problems that governance can help solve, such as audit findings or reporting disputes.
Develop organisational touch points with procurement, finance, the PMO, audit and SDLC processes, so governance is built into existing ways of working.
2. Develop the data governance strategy
Deliverables include a charter, an operating framework with accountabilities, an implementation roadmap and a plan for operational success.
3. Implement data governance
Sponsor data governance: secure an executive sponsor who can remove obstacles.
Define the operating framework: bodies, roles and decision rights.
Develop goals, principles and policies.
Underwrite data management projects: make sure projects follow standards.
Engage change management: communication, training and support.
Engage in issue management.
Assess regulatory compliance requirements.
Implement governance: start with a pilot domain, then expand.
Sponsor data standards and procedures.
Develop a business glossary.
Coordinate with architecture groups.
Sponsor data asset valuation.
4. Embed data governance
The aim is for governance to become part of how the organisation works, not a separate project. Signs of success: stewards are named in job descriptions, projects consult the DGO automatically, and metrics are reviewed routinely.
04Tools and techniques
Online presence and websites
A governance site with policies, standards, the glossary and contacts.
Business glossary
Often part of a data catalogue or metadata tool.
Workflow tools
For change requests, approvals and issue tracking.
Document management
For policies, minutes and decisions.
Governance scorecards
Dashboards of the metrics below.
05Metrics
The DMBOK groups governance metrics into three areas:
Area
Example metrics
Value
Contributions to business objectives; reduction of risk; improved efficiency in operations
Effectiveness
Achievement of goals and objectives; extent to which stewards use the tools; effectiveness of communication, education and training; adoption of change
Sustainability
Performance of policies and processes; conformance to standards and procedures
Business strategy and goals, IT strategy and goals, data management and data strategies, organisation policies and standards, business culture assessment, data maturity assessment, IT practices, regulatory requirements.
Deliverables
Data governance strategy, data strategy, business and data governance strategy roadmap, data principles, policies, processes, operating framework, roadmap, implementation strategy, operations plan, business glossary, governance scorecard, website, communications plan, recognised data value, maturing practices.
Participants
Steering committees, CIO, CDO and chief data stewards, executive, coordinating, business and technical stewards, governance bodies, compliance team, DM executives, change managers, enterprise data architects, project management office, governance bodies, audit, data professionals.
Consumers
Data governance bodies, project managers, compliance team, DM communities of interest, DM team, business management, architecture groups, partner organisations.
07Links to reference data and data quality
Reference data is a governance test case. It is small, shared and visible, so it is an ideal pilot domain: one owner per code list, one authoritative source, and a change process.
Many governance issues are reference data issues, as the issue log shows ("UK" vs "GB"). Governance gives them a route to resolution.
Data quality thresholds and rules need an owner to approve them. Governance supplies the decision rights.
This links to the workshop question on strategies that stall: governance without an operating model and named owners is usually where delivery breaks down.
Chapter 4 · Knowledge area
Data Architecture
The blueprint for an organisation's data: what data it has, how it's structured at enterprise level, how it flows between systems, and how that design is kept aligned with business strategy.
01Introduction
Definition
Data architecture is about identifying the data needs of the enterprise (regardless of structure), and designing and maintaining the master blueprints to meet those needs. Those blueprints guide data integration, control data assets and align data investments with business strategy.
Data architecture is most valuable when it fully supports the needs of the whole enterprise. It acts as a bridge between business strategy and technology execution.
Business drivers
Strategically prepare organisations to evolve their products, services and data quickly, taking advantage of business opportunities in emerging technologies
Translate business needs into data and system requirements, so processes consistently have the data they need
Manage complex data and information delivery throughout the enterprise
Facilitate alignment between business and IT
Act as agents for change, transformation and agility
Outcomes
Data storage and processing requirements
Designs of structures and plans that meet current and long-term enterprise data requirements
02Essential concepts
Enterprise architecture domains
Data architecture is one of several architecture domains. They depend on each other.
Domain
Purpose
Elements
Roles
Enterprise business architecture
Identify how an enterprise creates value for customers and other stakeholders
Business models, processes, capabilities, services, events, strategies, vocabulary
Business architects and analysts, business data stewards
Enterprise data architecture
Describe how data should be organised and managed
Data models, data definitions, data mapping specifications, data flows, structured data APIs
Data architects and modellers, data stewards
Enterprise applications architecture
Describe the structure and functionality of applications
Business systems, software packages, databases
Applications architects
Enterprise technology architecture
Describe the physical technology needed to enable systems to function and deliver value
John Zachman's framework is an ontology for describing an enterprise. Its columns ask six questions; its rows are six perspectives, from executive to technician. Data architecture lives mostly in the What column (inventory).
Perspective
What (inventory)
How (process)
Where (network)
Who (organisation)
When (timing)
Why (motivation)
Executive (scope)
List of things important to the business: Customer, Product, Order
Core processes
Locations
Organisations
Events and cycles
Goals
Business management
Conceptual model: subject areas and relationships
Process models
Logistics network
Workflow
Master schedule
Business plan
Architect
Logical data model
Application architecture
Distributed system architecture
Human interface
Processing structure
Business rules
Engineer
Physical data model
System design
Technology architecture
Presentation
Control structure
Rule design
Technician
Data definitions (DDL)
Program code
Network architecture
Security architecture
Timing definitions
Rule specification
Enterprise
Actual data
Working functions
Usable network
Functioning organisation
Implemented schedule
Working strategy
The highlighted column maps straight onto Chapter 5's levels: conceptual, logical, physical, then the real data.
Other frameworks include TOGAF (The Open Group Architecture Framework), with its Architecture Development Method, and industry-specific reference architectures.
Enterprise data model
An enterprise data model (EDM) is a holistic, enterprise-level, implementation-independent model of the data the organisation uses. It's usually built in layers:
Layer 1
Subject area model
The major subject areas and how they relate. One page.
Layer 2
Conceptual model
Key entities and relationships for each subject area.
Layer 3
Logical models
Detailed attributes, keys and rules for each area.
Worked example: Northwind subject area model
Data flow design
Data flows show which processes and systems create and use data, and how it moves between them. They are documented in two main ways: as matrices and as diagrams.
Worked example: a CRUD matrix
C = create, R = read, U = update, D = delete. The matrix shows where each entity is mastered, and where there are conflicts.
Business process
Customer
Product
Order
Invoice
Country (ref)
Onboard customer
C U
R
Maintain catalogue
C U D
Take order
R
R
C U
R
Bill customer
R U
R
R
C
R
Manage code lists
C U D
Finding: both onboarding and billing update Customer. Two processes updating the same master data is a classic cause of inconsistency. The architecture should choose one system of record for customer details.
Worked example: a data flow diagram
Northwind target data flows
Reference data feeds the hub so codes are validated once. The dashed line shows billing reading customer data from the hub instead of keeping its own copy, which resolves the CRUD conflict above.
Architecture principles
Example data architecture principles
Principle
Rationale
Implication
Data has one system of record
Multiple masters drift apart
Every entity in the CRUD matrix has exactly one creating system
Reference data is shared, not copied
Local code lists diverge
Systems consume code lists from the reference data service
Integrate through managed interfaces
Point-to-point links multiply cost
New integrations use the hub or event platform
Design for history
Regulators and analysts need "as it was"
Changes are captured with effective dates
Modern architecture patterns Beyond DMBOK
Pattern
Idea
Data management implication
Data lakehouse
One platform combining lake storage with warehouse-style tables and transactions
Governance and quality apply to files as well as tables
Data mesh
Domain teams own and publish data as products, on a self-serve platform, with federated governance (Zhamak Dehghani)
Needs shared standards for identifiers and reference data, or products won't join up
Data fabric
A layer of active metadata and automation that connects data across platforms
Depends on rich, trustworthy metadata (see Chapter 12)
Data products and marketplaces
Curated, owned datasets with contracts and SLAs, discoverable in a catalogue
Mike Ferguson's "build once, reuse everywhere" theme
03Activities
Establish data architecture practice
Evaluate existing data architecture specifications. Find what documentation exists and how accurate it is.
Develop a roadmap. Describe the target architecture and the steps to get there, based on business priorities and dependencies.
Manage enterprise requirements within projects. Make sure each project fits the enterprise model and doesn't create new silos.
Integrate with enterprise architecture
Work with business, application and technology architects so data designs fit the wider architecture, and so architecture governance reviews include data.
Worked example: a roadmap
Phase
Scope
Depends on
Business outcome
1. Foundations
Subject area model; reference data service for country and currency
Governance owners named
Consistent codes in new projects
2. Customer
Customer master hub; billing reads from hub
Phase 1
Fewer invoice disputes
3. Analytics
Warehouse sourced from hub and ERP
Phase 2
Trusted customer reporting
4. Products
Product master and published data products
Phases 1–3
Reuse across teams
04Tools and techniques
Data modelling tools
For enterprise and project models.
Asset management software
Inventories of systems and data stores.
Graphical design applications
For architecture diagrams.
Lifecycle projections
Planning when technologies are introduced, current or retired.
Diagramming clarity
Clear legends, consistent alignment and minimal crossing lines.
05Governance and metrics
Architecture standards compliance rate: how many projects comply with the data architecture.
Implementation trends: how far the actual architecture moves toward the target.
Business value: for example reduced integration cost, faster delivery of new data needs, fewer redundant data stores.
Architecture governance typically reviews project designs at defined checkpoints, approves exceptions, and keeps the enterprise model current.
06Context summary
Inputs
Enterprise architecture, business architecture, IT standards and goals, data strategies.
Deliverables
Data architecture design, implementation roadmap, enterprise data model, data flow designs, data value chains, information architecture.
Participants
Enterprise data architects, data modelling specialists, data stewards, enterprise architects, solution architects.
Consumers
Database administrators, software developers, project managers, support teams.
07Links to reference data and data quality
Reference data is a subject area in its own right and should appear on the enterprise model, not be buried inside applications.
The CRUD matrix is a quality tool: multiple creators or updaters of the same entity predict consistency problems before they happen.
Architecture principles such as "one system of record" and "reference data is shared, not copied" turn RDM and DQ goals into design rules projects must follow.
For the workshop's "where to start" question: architecture tells you which entities are shared most widely, which helps pick the critical data elements to fix first.
Chapter 5 · Knowledge area
Data Modeling and Design
What the DMBOK says about discovering, analysing and representing data requirements, from conceptual models through to physical database design.
01Introduction
Definition
Data modeling is the process of discovering, analysing and scoping data requirements, then representing and communicating them precisely in a data model.
Business drivers
A common vocabulary for data
Capturing and documenting knowledge about systems and data
A primary communication tool during projects
A starting point for customising, integrating or replacing an application
Goal
Confirm and document an understanding of different perspectives. The result is applications that match current and future business requirements, and a foundation for broader work such as master data management and data governance.
02Essential concepts
Types of data that get modelled
Reference data
Category information
Data used to classify and assign types, such as customers by market or products by category.
Master data
Resource information
Profiles of resources needed to run operations: products, customers, suppliers, sites.
Transactional
Business event information
Data created during operations, such as orders and payments.
High volume
Detail transaction information
Often from point-of-sale systems, social media, sensors and similar sources.
Components of a data model
Entity. A thing an organisation collects information about: the who, what, when, where, why and how of the business.
Relationship. An association between entities. Its properties include:
Cardinality: one-to-one, one-to-many or many-to-many
Arity: unary (recursive), binary or ternary
Foreign keys, and identifying vs non-identifying relationships
Cardinality in Information Engineering (crow's foot) notation:
Exactly oneZero or oneOne or manyZero or many
Attribute. A property that describes an entity. Keys are a special case:
By construction: simple, compound, composite and surrogate keys
By function: candidate, primary and alternate keys
Types of keys
The DMBOK groups keys two ways: by how they are built (construction) and by the job they do (function). One sample table shows most of them at once.
A system-generated identifier, usually an integer, with no business meaning
ConsultantKey = 1, 2, 3
Outside the DMBOK, many people use "compound" and "composite" to mean the same thing: any key with more than one column.
By function
Key
What it is
Example
Super key
Any set of attributes that uniquely identifies a row, even with extra columns it doesn't need
StaffNo + Name
Candidate
A minimal super key: remove any attribute and it stops being unique
ConsultantKey, StaffNo, Email
Primary
The candidate key chosen as the main identifier
ConsultantKey
Alternate
A candidate key not chosen as primary, usually enforced with a unique constraint
StaffNo, Email
Natural / business
A candidate key with real-world meaning that the business recognises
StaffNo, a VAT number, an ISO country code
Foreign
An attribute that refers to the primary key of another entity
ServiceCode → Services
Related terms
Identifying relationship: the parent's key becomes part of the child's primary key. OrderID is part of OrderLines' key.
Non-identifying relationship: the foreign key is not part of the child's primary key. CountryCode in Customers is an example.
Smart (intelligent) key: a key that encodes meaning, such as UK-LDN-0042. Usually avoided, because the key has to change when the meaning changes.
Hash key: a key calculated by hashing the business key. It's used in Data Vault 2.0 (see below).
Keys working together
Primary, foreign, composite and surrogate keys across one small warehouse load. A product arrives from the source system with its natural key, gets a surrogate key in the warehouse, and the fact table combines several foreign keys into a composite primary key.
Product (source system)
ProductID
ProductName
P10
Laptop
P20
Mouse
P30
Monitor
DimProduct (warehouse)
ProductKey
ProductID
ProductName
1
P10
Laptop
2
P20
Mouse
3
P30
Monitor
FactSales: composite primary key of four foreign keys
DateKey
ProductKey
StoreKey
CustomerKey
Qty
SalesAmount
20260302
1
5
1001
1
899.00
20260302
2
5
1001
2
39.98
20260305
3
7
1002
1
249.00
Key
Where
What it does here
Primary key (natural)
Product.ProductID
Identifies each product in the source, using a code the business knows
Surrogate key
DimProduct.ProductKey
A meaningless integer the warehouse controls. It becomes the dimension's primary key.
Alternate / business key
DimProduct.ProductID
Kept so every warehouse row can be traced back to the source
Uniquely identifies a sale at the declared grain: one product, one store, one customer, one day
Why use a surrogate key
Protection from source changes: if the source renumbers P10 to LAP-001, only one DimProduct row changes, not millions of fact rows.
Integration: CRM calls a customer C1 and billing calls it 7781. One surrogate key can represent both.
History: SCD Type 2 needs a new key for each version of the same business key.
Performance: small integer keys join faster than long text codes.
Link to RDM and DQ: natural keys are where quality problems hurt most. Duplicate or inconsistent business keys create duplicate customers, and surrogate keys can hide that problem rather than fix it.
Domain. The complete set of valid values for an attribute. A domain can be defined by data type, data format, a list (a code list), a range or a rule.
Link to RDM and DQ: domains are where reference data and validity rules sit in the model. A governed list of country codes is a domain.
Data modelling schemes
Scheme
Typical use
Notations and ideas
Key originators
Relational
Operational systems, normalised design
Information Engineering (IE), IDEF1X, Barker, Chen
E.F. Codd (relational model, 1970). Notations: Peter Chen (1976), Clive Finkelstein and James Martin (IE), Richard Barker (Barker), US Air Force ICAM programme (IDEF1X)
Dimensional
Data warehouses and BI
Star and snowflake schemas; facts, dimensions and grain; conformed dimensions and facts
Ralph Kimball (with Margy Ross)
Object-oriented
Application design
UML class diagrams
Grady Booch, James Rumbaugh, Ivar Jacobson
Fact-based
Capturing business semantics
ORM, FCO-IM
Terry Halpin (ORM, building on Sjir Nijssen's NIAM); Guido Bakema, Jan Pieter Zwart and Harm van der Lek (FCO-IM)
Time-based
Historised, auditable integration layers
Data Vault (hubs, links, satellites), Anchor Modeling
Dan Linstedt (Data Vault); Lars Rönnbäck and Olle Regardt (Anchor Modeling)
NoSQL
Non-relational stores
Document, key-value, column-oriented, graph
No single originator
Worked examples of every scheme follow. Each one uses the same small set of customers, orders and products, so you can compare how the schemes organise identical data.
Relational: entity-relationship model
A relational model organises data into tables of rows and columns, linked by keys. Normalised to 3NF, it suits operational systems where data is written often and must stay consistent. This is the 3NF orders model from the normal forms section, drawn in Information Engineering (crow's foot) notation.
Orders ER diagram
How to read it: each customer is in exactly one country, and a country has zero or many customers. Each customer places zero or many orders. Each order has one or many lines. Each product appears on zero or many order lines.
The same model as SQL table definitions:
CREATE TABLE Country (
CountryCode CHAR(2) PRIMARY KEY,
CountryName VARCHAR(100) NOT NULL
);
CREATE TABLE Customer (
CustomerID VARCHAR(10) PRIMARY KEY,
CustomerName VARCHAR(200) NOT NULL,
CountryCode CHAR(2) NOT NULL REFERENCES Country (CountryCode)
);
CREATE TABLE "Order" ( -- quoted because ORDER is a reserved word
OrderID INTEGER PRIMARY KEY,
OrderDate DATE NOT NULL,
CustomerID VARCHAR(10) NOT NULL REFERENCES Customer (CustomerID)
);
CREATE TABLE Product (
ProductID VARCHAR(10) PRIMARY KEY,
ProductName VARCHAR(200) NOT NULL
);
CREATE TABLE OrderLine (
OrderID INTEGER REFERENCES "Order" (OrderID),
ProductID VARCHAR(10) REFERENCES Product (ProductID),
Qty INTEGER NOT NULL CHECK (Qty > 0),
PRIMARY KEY (OrderID, ProductID)
);
Link to RDM and DQ: the constraints are data quality rules built into the design. NOT NULL enforces completeness, CHECK enforces validity, and the foreign key to Country stops anyone entering a country code that isn't in the governed reference list.
Dimensional: star vs snowflake schemas
Both are dimensional designs. A central fact table holds the measures (quantity, sales amount) at a declared grain, such as one row per product per store per day. It is surrounded by dimension tables that hold the descriptive context used to slice and filter.
Star schemaSnowflake schema
Sample fact rows (the same in both designs):
DateKey
ProductKey
StoreKey
CustomerKey
Qty
SalesAmount
20260302
1
5
1001
1
899.00
20260302
2
5
1001
2
39.98
20260305
3
7
1002
1
249.00
The product dimension in each design. The star keeps the whole hierarchy in one wide, denormalised table. The snowflake normalises it into separate tables.
Star: DimProduct
ProductKey
ProductName
Category
Department
1
Laptop
Computers
Electronics
2
Mouse
Accessories
Electronics
3
Monitor
Computers
Electronics
Snowflake: DimProduct
ProductKey
ProductName
CategoryKey
1
Laptop
10
2
Mouse
20
3
Monitor
10
DimCategory
CategoryKey
Category
DeptKey
10
Computers
100
20
Accessories
100
DimDepartment
DeptKey
Department
100
Electronics
Underlined = primary keyItalic = foreign keyRepeated value
Aspect
Star
Snowflake
Dimension structure
One denormalised table per dimension
Dimensions normalised into hierarchies of tables
Joins per query
Few: fact to each dimension
More: fact to dimension to sub-dimensions
Query speed and simplicity
Usually faster and easier for users and BI tools
Slower and more complex to write
Redundancy
Hierarchy values repeat on every row
Each value stored once
Maintenance
Renaming a category touches many rows
Renaming a category touches one row
Best fit
The default for BI and reporting
Very large dimensions, or hierarchies shared across dimensions and managed centrally
Who is behind these designs
Star schema
Ralph Kimball
Popularised dimensional modelling and the star schema in The Data Warehouse Toolkit (1996; later editions with Margy Ross). Generally advises against snowflaking, apart from limited "outrigger" tables.
Snowflake schema
No single author
A normalised variant of the star, named after the shape its diagram makes. It grew out of applying normalisation to dimension tables.
The counterpoint
Bill Inmon
Often called the father of the data warehouse (Building the Data Warehouse, 1992). Argued for a normalised (3NF) enterprise warehouse that feeds dimensional data marts. This is the classic Inmon vs Kimball debate.
A third way
Dan Linstedt
Created Data Vault, a model for historised integration layers that sits between source systems and dimensional marts.
Link to RDM: the tables a snowflake splits out (category, department, region, city) are usually reference data. Managing those hierarchies centrally is what makes conformed dimensions possible across marts.
Querying the star. A typical BI question, sales by month and category, needs only one join per dimension:
SELECT d.MonthName, p.Category, SUM(f.SalesAmount) AS Sales
FROM FactSales f
JOIN DimDate d ON d.DateKey = f.DateKey
JOIN DimProduct p ON p.ProductKey = f.ProductKey
GROUP BY d.MonthName, p.Category;
In the snowflake version, the same query needs two more joins: DimProduct to DimCategory, and DimCategory to DimDepartment if you report by department.
Dimensional: slowly changing dimensions
Dimension attributes change over time. Kimball and Ross defined standard ways to handle this, called slowly changing dimension (SCD) types. You choose a type per attribute, not per table.
Type
Technique
History kept
Typical use
Type 0
Retain the original value; never change it
Original only
Date of first order, original credit score
Type 1
Overwrite the old value
None
Correcting spelling mistakes
Type 2
Add a new row with a new surrogate key
Full
Name, address, segment: anything reported "as it was"
Type 3
Add a column for the previous value
One previous value
Before-and-after views of a reorganisation
Type 4
Move fast-changing attributes into a separate mini-dimension
Full, in the mini-dimension
Credit band, payment terms
Type 5
Type 4 plus a Type 1 link from the main dimension to the current mini-dimension row
Full, plus the current profile
Showing a customer's current band next to history
Type 6
Type 2 rows plus a Type 1 "current value" column (1 + 2 + 3 = 6)
Full, plus the current value on every row
Reporting history under either the old or the current name
Type 7
The fact holds both the surrogate key and the durable business key
Full, queried as-was or as-is
Letting users choose either view at query time
Worked examples of the most common types, using Northwind's rename in June:
Type 0: retain the original. FirstOrderDate is set once and never updated, whatever else changes.
DimCustomer (Type 0 attribute)
CustomerKey
CustomerID
CustomerName
FirstOrderDate
1001
C1
Northwind Ltd
2026-03-02
Type 1: overwrite. The old value is lost. Past sales now report under the new name.
DimCustomer (Type 1)
CustomerKey
CustomerID
CustomerName
1001
C1
Northwind Group Ltd
Type 2: add a new row. Full history. Each version gets its own surrogate key, so March sales stay linked to key 1001 and report under the old name.
DimCustomer (Type 2)
CustomerKey
CustomerID
CustomerName
ValidFrom
ValidTo
IsCurrent
1001
C1
Northwind Ltd
2026-03-01
2026-06-14
N
1047
C1
Northwind Group Ltd
2026-06-15
9999-12-31
Y
Type 3: add a column. Keeps only the previous value, which is useful for "before and after" reporting.
DimCustomer (Type 3)
CustomerKey
CustomerID
CurrentName
PreviousName
1001
C1
Northwind Group Ltd
Northwind Ltd
Type 4: mini-dimension. Attributes that change often, like credit band and payment terms, move to a small separate dimension. The fact table carries both keys, so the main customer dimension doesn't grow a new row every time a credit limit changes.
DimCustomerProfile (mini-dimension)
ProfileKey
CreditBand
PaymentTerms
1
Up to 24,999
30 days
2
25,000–49,999
30 days
3
Up to 24,999
45 days
4
50,000 and over
30 days
FactSales (extract)
DateKey
CustomerKey
ProfileKey
SalesAmount
20260305
1002
3
249.00
20260512
1002
2
310.00
Contoso's March sale was made on profile 3 (20,000 limit, 45 days). After its terms changed in April, its May sale points to profile 2. The customer row itself didn't change. Some teams use "Type 4" to mean a separate history table instead; Kimball's definition is the mini-dimension.
Type 6: hybrid of 1, 2 and 3. Type 2 rows keep the historic name, and a CurrentName column is overwritten on every row, so reports can use either.
DimCustomer (Type 6)
CustomerKey
CustomerID
HistoricName
CurrentName
ValidFrom
ValidTo
IsCurrent
1001
C1
Northwind Ltd
Northwind Group Ltd
2026-03-01
2026-06-14
N
1047
C1
Northwind Group Ltd
Northwind Group Ltd
2026-06-15
9999-12-31
Y
Type 2 is the reason dimensional models use surrogate keys: the business key C1 alone can't tell the two versions apart.
Object-oriented: UML class diagram
An object-oriented model describes classes of objects, with their attributes (data) and operations (behaviour). It's used for application design. To store the objects, the model has to be mapped to tables, a step called object-relational mapping.
Orders class diagram
UML element
Meaning
In the example
Class
A type of object, with attributes and operations
Customer, Order, Product
Abstract class
A class with no direct instances, only subclasses. Its name is shown in italics.
Party
Generalisation
"Is a kind of", meaning inheritance. Drawn with a hollow triangle.
Customer and Supplier are kinds of Party
Association
A relationship between classes
Customer places Order
Composition
Strong ownership: the parts can't exist without the whole. Drawn with a filled diamond.
An OrderLine belongs to exactly one Order
Multiplicity
How many objects take part
1, 0..*, 1..*
Operation
Behaviour the class provides
total() adds up the order lines
Mapping inheritance to tables has three common options: one table for the whole hierarchy with a type column, one table per class, or one table per concrete class. Operations like total() aren't stored; they become application code or calculated columns.
Fact-based: ORM and FCO-IM
Fact-based modelling starts from facts written as plain sentences, checked by business experts against real example data. It doesn't decide early what is an entity and what is an attribute. Tables are derived later from the fact types and their constraints. ORM (Object-Role Modeling) and FCO-IM (Fully Communication Oriented Information Modeling) both work this way.
Step 1: verbalise elementary facts, with examples
Fact type
Example fact
Constraint agreed with the business
Customer has CustomerName
Customer C1 has name "Northwind Ltd"
Each Customer has exactly one name
Customer is located in Country
Customer C1 is located in Country GB
Each Customer is in exactly one Country
Customer places Order
Customer C1 places Order 1001
Each Order is placed by exactly one Customer
Order includes Product in Quantity
Order 1001 includes Product P10 in quantity 1
Each Order and Product pair has at most one quantity
Step 2: draw the fact types. In ORM, entity types are rounded shapes and each fact type is a row of role boxes. A bar over a role is a uniqueness constraint, and a dot is a mandatory constraint.
ORM: Customer places Order
Step 3: derive the tables. The uniqueness bar says each Order plays its role only once, so Order is the key and "placed by" becomes a CustomerID column in the Order table. The three-role fact type "Order includes Product in Quantity" has a uniqueness constraint spanning Order and Product, which becomes OrderLine's compound key. Applying the rules to all the fact types gives exactly the 3NF tables from the relational example.
FCO-IM, developed in the Netherlands, puts even more weight on how domain experts actually phrase facts. It records those phrasings as an "information grammar" and generates the data model from it.
Time-based: Data Vault hubs, links and satellites
Data Vault was created by Dan Linstedt in the 1990s. Data Vault 2.0 is described in Building a Scalable Data Warehouse with Data Vault 2.0 (Linstedt and Michael Olschimke, 2015). It's a modelling approach for the integration layer of a warehouse. It keeps full history, records where every row came from, and absorbs new or changing sources without redesign.
It separates three things that other models mix together:
Hub
What the thing is
The unique list of business keys for a core business concept, such as Customer or Order.
Link
How things relate
A relationship or transaction between two or more hubs, such as Customer placed Order.
Satellite
What it looks like over time
The descriptive attributes of a hub or link, with a new row every time they change.
Example Data Vault model
Table
Holds
Typical columns
Rules
Hub
Unique business keys for one concept
Hash key, business key, load date, record source
No descriptive attributes. A business key is loaded once and never changes.
Link
Relationships and transactions between hubs
Link hash key, the hub hash keys it connects, load date, record source
Always modelled as many-to-many, so a change in a business rule doesn't force a redesign. No descriptive attributes.
Satellite
Descriptive attributes and their history
Parent hash key + load date (together the key), hash diff, attributes, record source
Insert-only: a change adds a new row. Usually split by source system and by how often the attributes change.
Sample data, continuing the orders example. Hash keys are shortened here.
Hub_Customer
CustomerHK
CustomerBK
LoadDate
RecordSource
9f2c…a1
C1
2026-03-01
CRM
4b7e…d3
C2
2026-03-04
CRM
Hub_Order
OrderHK
OrderBK
LoadDate
RecordSource
1a9d…e0
1001
2026-03-02
ERP
7c3f…b2
1002
2026-03-05
ERP
e5b1…44
1003
2026-03-09
ERP
Link_CustomerOrder
LinkHK
CustomerHK
OrderHK
LoadDate
RecordSource
2d8a…19
9f2c…a1
1a9d…e0
2026-03-02
ERP
6e04…c7
4b7e…d3
7c3f…b2
2026-03-05
ERP
b3f9…5a
9f2c…a1
e5b1…44
2026-03-09
ERP
Sat_Customer_CRM
CustomerHK
LoadDate
HashDiff
CustomerName
CountryCode
RecordSource
9f2c…a1
2026-03-01
51ac…
Northwind Ltd
GB
CRM
9f2c…a1
2026-06-15
8d02…
Northwind Group Ltd
GB
CRM
4b7e…d3
2026-03-04
c7e9…
Contoso SA
FR
CRM
History in action: when Northwind renamed itself in June, the satellite gained a new row instead of overwriting the old one. Both versions are kept, with the date and the source system.
Key terms
Hash key: a hash of the business key (for example MD5 or SHA). Every table can calculate it independently, so hubs, links and satellites load in parallel without looking up surrogate keys.
Hash diff: a hash of all the attributes in a satellite row. If the incoming hash diff matches the latest row, nothing changed and nothing is inserted.
Load date and record source: on every table, giving a full audit trail of when data arrived and where from.
Raw vault and business vault: the raw vault stores source data as received. The business vault adds derived data and business rules.
Point-in-time (PIT) and bridge tables: helper tables that make the vault faster to query. They're covered in detail below.
Reference tables: code lists such as countries or currencies, kept as simple reference tables rather than hubs.
Satellites in depth
Satellites hold everything descriptive, and all the history. A hub or link can have several satellites. Design choices about how to split them matter more than any other part of a vault.
Why split satellites
By source system: CRM and billing each describe the customer differently, so each gets its own satellite. Sources never overwrite each other.
By rate of change: attributes that change daily, like credit balance, go apart from ones that rarely change, like name. Otherwise every small change copies the whole row.
By sensitivity: personal data in its own satellite is easier to secure, and easier to erase for a GDPR request.
The second customer satellite from the diagram, from the billing system:
Sat_Customer_Billing
CustomerHK
LoadDate
HashDiff
PaymentTerms
CreditLimit
RecordSource
9f2c…a1
2026-03-01
a4e8…
30 days
50,000
Billing
4b7e…d3
2026-03-04
0bd3…
45 days
20,000
Billing
4b7e…d3
2026-04-10
f61c…
30 days
25,000
Billing
Types of satellite
Type
Attached to
What it tracks
Example
Standard
Hub or link
Descriptive attributes over time
Customer name and country
Multi-active
Hub
Several rows that are valid at the same time for one key
A customer's work, mobile and home phone numbers
Effectivity
Link
When a relationship starts and ends
Which account manager looks after a customer, and from when
Status / record tracking
Hub or link
When a key appears in or disappears from a source, including deletions
A customer deleted from CRM
Computed (business vault)
Hub or link
Values derived by business rules rather than loaded from a source
Customer lifetime value, risk segment
PIT and bridge tables
A vault stores data in many narrow tables, so queries need many joins and date comparisons. PIT and bridge tables are helper tables, usually in the business vault, that do that work in advance. They hold keys and dates rather than new business data, and can be rebuilt at any time.
Point-in-time (PIT) table. For each hub key and snapshot date, it records which row of each satellite was current. A query then joins on exact keys instead of working out "the latest row before this date" for every satellite.
PIT_Customer
CustomerHK
SnapshotDate
Sat_CRM_LoadDate
Sat_Billing_LoadDate
9f2c…a1
2026-03-31
2026-03-01
2026-03-01
4b7e…d3
2026-03-31
2026-03-04
2026-03-04
9f2c…a1
2026-06-30
2026-06-15
2026-03-01
4b7e…d3
2026-06-30
2026-03-04
2026-04-10
At the end of June, Northwind's PIT row points to the renamed CRM record, and Contoso's points to the updated billing terms. Each satellite moves independently.
Data Vault bridge table. It pre-joins a chain of hubs and links for a snapshot date, storing the hash keys along the path. Queries that cross several links then need one join instead of many. It can also hold pre-calculated measures.
Bridge_Customer_Order
SnapshotDate
CustomerHK
LinkHK
OrderHK
2026-03-31
9f2c…a1
2d8a…19
1a9d…e0
2026-03-31
4b7e…d3
6e04…c7
7c3f…b2
2026-03-31
9f2c…a1
b3f9…5a
e5b1…44
Bridge tables in dimensional models. Kimball uses the same name for something different: a table that resolves a many-to-many relationship between a fact and a dimension, or a ragged hierarchy. A weighting factor splits a measure across the related rows so totals still add up. Here, one project's revenue is shared between two consultants:
FactProjectRevenue
ProjectKey
ConsultantGroupKey
Revenue
77
900
40,000
BridgeConsultantGroup
ConsultantGroupKey
ConsultantKey
Weighting
900
1 (Asha)
0.6
900
2 (Ben)
0.4
Weighted, Asha is credited with 24,000 and Ben with 16,000, so the total stays at 40,000. Without the weighting, each would show the full 40,000 and the total would double-count.
Where it sits in the architecture
Source systems→Staging→Raw vault→Business vault→Information marts (star schemas)
Strengths
Full, auditable history
New sources add satellites without redesign
Parallel, insert-only loading that scales
Highly repeatable patterns that suit automation
Watch-outs
Many tables and joins
Not built for direct reporting; needs marts on top
Needs automation tooling and skills
Only as good as the business keys loaded into the hubs
Link to RDM and DQ: a hub is only as clean as its business keys. If two systems identify the same customer differently, the vault faithfully stores two customers. Master data and reference data work decides what a hub key means, and Data Vault reference tables are a natural home for governed code lists.
Time-based: Anchor Modeling
Anchor Modeling was created by Lars Rönnbäck and Olle Regardt. It is highly normalised, close to Sixth Normal Form: every attribute gets its own table. New attributes become new tables, so existing tables never change, and history is kept with a ChangedAt column.
Building block
What it holds
Example name
Anchor
The identity of an entity: a surrogate ID and nothing else
CU_Customer
Attribute
One property of an anchor, optionally historised
CU_NAM_Customer_Name
Tie
A relationship between anchors
CU_placed_OR_Order
Knot
A small, shared set of values
COU_Country
CU_Customer (anchor)
CU_ID
1
2
COU_Country (knot)
COU_ID
COU_Country
1
GB
2
FR
CU_NAM_Customer_Name (historised attribute)
CU_ID
CU_NAM_Customer_Name
CU_NAM_ChangedAt
1
Northwind Ltd
2026-03-01
1
Northwind Group Ltd
2026-06-15
2
Contoso SA
2026-03-04
CU_COU_Customer_Country (knotted attribute)
CU_ID
COU_ID
1
1
2
2
CU_placed_OR_Order (tie)
CU_ID
OR_ID
1
101
2
102
1
103
Aspect
Data Vault
Anchor Modeling
Unit of storage
A satellite groups many attributes
One attribute per table
Identity
Hash of the business key
Surrogate ID in the anchor
Shared code lists
Reference tables
Knots
Schema change
Additive: new satellites
Additive: new attribute tables
Link to RDM: knots are reference data by design: small, stable, shared code lists that many attributes point to.
NoSQL: document model
A document database (for example MongoDB or Couchbase) stores self-contained documents, usually JSON. You model around how the application reads data: things that are read together are stored together.
The data is read together, owned by the parent, and limited in size
Order lines inside the order
Reference
The data is shared, large, or changes independently
A separate customer document, linked by customerId
Here the customer's name is copied into every order on purpose, so each order keeps the name at the time of sale. The cost is redundancy: renaming Northwind doesn't update old orders, and that has to be a deliberate decision.
NoSQL: key-value model
The simplest model. A unique key maps to a value that the database doesn't interpret (for example Redis, or DynamoDB used simply). Lookups by key are extremely fast, but you can't query by what's inside the value. Designing the keys is the modelling.
Key
Value
customer:C1
{"name": "Northwind Ltd", "country": "GB"}
order:1001:status
"Shipped"
cart:session:8f3a2c
["P10", "P20", "P20"]
ref:country:GB
"United Kingdom"
country:GB:customers
["C1"]
The keys follow an entity:id:attribute convention. To answer "which customers are in GB?", you must maintain an extra key like country:GB:customers yourself.
NoSQL: wide-column model
Wide-column stores such as Apache Cassandra and HBase spread data across many servers by a partition key. Design starts from the queries: typically one table per query, with data deliberately duplicated. The query here is "show a customer's orders, newest first".
CREATE TABLE orders_by_customer (
customer_id text,
order_date date,
order_id int,
customer_name text,
total decimal,
PRIMARY KEY ((customer_id), order_date, order_id)
) WITH CLUSTERING ORDER BY (order_date DESC, order_id DESC);
orders_by_customer
customer_id (partition)
order_date (cluster)
order_id
customer_name
total
C1
2026-03-09
1003
Northwind Ltd
99.95
C1
2026-03-02
1001
Northwind Ltd
938.98
C2
2026-03-05
1002
Contoso SA
249.00
Partition key (customer_id): decides which server holds the rows and keeps one customer's orders together.
Clustering columns (order_date, order_id): sort the rows inside each partition, newest first.
A different question, such as "who bought product P20?", would need a second table, orders_by_product, loaded with the same data.
NoSQL: graph model
A graph database (for example Neo4j or Amazon Neptune) stores nodes (things) and relationships (connections), and both can have properties. It's strongest when the connections are the point: recommendations, fraud rings, networks and data lineage.
Orders as a graph (Order to Product edges are CONTAINS)
A query in Cypher (Neo4j's query language) that follows the connections: how many of each product has Northwind bought?
MATCH (c:Customer {id: 'C1'})-[:PLACED]->(:Order)-[r:CONTAINS]->(p:Product)
RETURN p.name AS product, sum(r.qty) AS unitsBought;
product
unitsBought
Laptop
1
Mouse
7
The schemes compared
Scheme
Organises data by
Best for
Main trade-off
Relational
Normalised tables linked by keys
Operational systems, consistency
Many joins for analysis
Dimensional
Facts surrounded by dimensions
BI and reporting
Redundancy in dimensions
Object-oriented
Classes with data and behaviour
Application design
Needs mapping to storage
Fact-based
Verbalised facts and constraints
Getting requirements right with the business
Less familiar; needs translation to tables
Data Vault
Hubs, links and satellites
Auditable, historised integration
Many tables; needs marts on top
Anchor
One table per attribute
Highly changeable, fully historised models
Very many tables
Document
Self-contained JSON documents
Apps that read whole objects
Duplication; weak cross-document queries
Key-value
A key and an opaque value
Caches, sessions, very fast lookups
No querying by value
Wide-column
Partitions designed per query
Huge write volumes at scale
One table per query; duplication
Graph
Nodes and relationships
Highly connected data
Less suited to bulk aggregation
Levels of detail
Business
Conceptual
High-level business concepts and how they relate. Focused on scope and vocabulary.
Requirements
Logical
Detailed requirements with attributes and keys, independent of any technology.
Technology
Physical
Tables, columns, data types, indexes, partitioning, views and denormalisation for a specific platform.
The physical level also covers canonical models, views, partitioning and denormalisation.
Normal forms
Normalisation applies rules to remove redundancy and the update problems it causes. Each form builds on the one before. Third Normal Form is the usual target for operational models.
Form
The rule in one line
Introduced by
1NF
Every cell holds one atomic value, with no repeating groups
E.F. Codd, 1970
2NF
No attribute depends on only part of a composite key
E.F. Codd, 1971
3NF
No attribute depends on another non-key attribute
E.F. Codd, 1971
BCNF
Every determinant is a candidate key
Raymond Boyce and E.F. Codd, 1974
4NF
No independent multi-valued facts stored together
Ronald Fagin, 1977
5NF
No table can be split into smaller tables without losing information, unless the split follows its keys
Ronald Fagin, 1979
A common way to remember 1NF to 3NF: every non-key attribute depends on the key, the whole key, and nothing but the key. Beyond 5NF sit Domain-Key Normal Form (Fagin, 1981) and Sixth Normal Form (Date, Darwen and Lorentzos, early 2000s), which is used in temporal designs and Anchor Modeling.
Worked examples with sample data
1NF to 3NF follow one running example of order records. BCNF, 4NF and 5NF need their own classic cases, because the problems they fix only appear in particular patterns.
Underlined = primary key (or part of it)Italic = foreign keyRepeated or problem value
Unnormalised the starting point
OrderID
OrderDate
Customer
Country
Products
1001
2026-03-02
Northwind Ltd
GB, United Kingdom
Laptop ×1, Mouse ×2
1002
2026-03-05
Contoso SA
FR, France
Monitor ×1
1003
2026-03-09
Northwind Ltd
GB, United Kingdom
Mouse ×5
Problem: Products holds a repeating group of several items in one cell, and Country packs two values into one. You can't query or total by product.
First Normal Form (1NF) Codd, 1970
RuleEvery cell holds one atomic value, there are no repeating groups, and a key identifies each row.
Split the product list into one row per order line, and split Country into code and name. The key becomes OrderID + ProductID.
OrderLine
OrderID
ProductID
OrderDate
CustomerID
CustomerName
CountryCode
CountryName
ProductName
Qty
1001
P10
2026-03-02
C1
Northwind Ltd
GB
United Kingdom
Laptop
1
1001
P20
2026-03-02
C1
Northwind Ltd
GB
United Kingdom
Mouse
2
1002
P30
2026-03-05
C2
Contoso SA
FR
France
Monitor
1
1003
P20
2026-03-09
C1
Northwind Ltd
GB
United Kingdom
Mouse
5
Problem: partial dependencies. ProductName depends only on ProductID, and OrderDate and the customer depend only on OrderID. Each is just part of the key. Renaming "Mouse" means updating every row that sold one.
Second Normal Form (2NF) Codd, 1971
RuleIn 1NF, and every non-key attribute depends on the whole key. This only matters when the key is composite.
Move order-level facts to Orders and product facts to Products. OrderLines keeps only what depends on both parts of the key.
Orders
OrderID
OrderDate
CustomerID
CustomerName
CountryCode
CountryName
1001
2026-03-02
C1
Northwind Ltd
GB
United Kingdom
1002
2026-03-05
C2
Contoso SA
FR
France
1003
2026-03-09
C1
Northwind Ltd
GB
United Kingdom
OrderLines
OrderID
ProductID
Qty
1001
P10
1
1001
P20
2
1002
P30
1
1003
P20
5
Products
ProductID
ProductName
P10
Laptop
P20
Mouse
P30
Monitor
Problem: transitive dependencies. In Orders, CustomerName depends on CustomerID rather than OrderID, and CountryName depends on CountryCode. Northwind's name and "United Kingdom" repeat on every order.
Third Normal Form (3NF) Codd, 1971
RuleIn 2NF, and no non-key attribute depends on another non-key attribute.
Give customers and countries their own tables. OrderLines and Products stay as they were in 2NF.
Orders
OrderID
OrderDate
CustomerID
1001
2026-03-02
C1
1002
2026-03-05
C2
1003
2026-03-09
C1
Customers
CustomerID
CustomerName
CountryCode
C1
Northwind Ltd
GB
C2
Contoso SA
FR
Countries
CountryCode
CountryName
GB
United Kingdom
FR
France
Link to RDM: the Countries table is reference data, and it appears naturally from normalisation. Customers is master data and Orders is transactional data, matching the DMBOK's types of data.
Boyce-Codd Normal Form (BCNF) Boyce and Codd, 1974
RuleEvery determinant (anything that decides another value) is a candidate key.
Consultant assignments, with two business rules: each consultant specialises in exactly one service, and each client has one consultant per service. The key is Client + Service.
Assignment
Client
Service
Consultant
Northwind
Data Quality
Asha
Northwind
Reference Data
Ben
Contoso
Data Quality
Asha
Fabrikam
Data Quality
Chloe
Problem: Consultant determines Service, but Consultant isn't a key. The table is in 3NF, because Service is part of the key, yet "Asha does Data Quality" is stored twice. You also can't record a new MDM specialist until they have a client.
ConsultantService
Consultant
Service
Asha
Data Quality
Ben
Reference Data
Chloe
Data Quality
ClientConsultant
Client
Consultant
Northwind
Asha
Northwind
Ben
Contoso
Asha
Fabrikam
Chloe
Trade-off: after the split, the rule "one consultant per client per service" can no longer be enforced by a key alone. This is why some designs stop at 3NF.
Fourth Normal Form (4NF) Fagin, 1977
RuleIn BCNF, and a table doesn't hold two or more independent multi-valued facts about the same thing.
A consultant's skills and languages are unrelated to each other, but they're stored together.
ConsultantProfile
Consultant
Skill
Language
Asha
Data Quality
English
Asha
Data Quality
French
Asha
MDM
English
Asha
MDM
French
Problem: every skill must be paired with every language. Adding Spanish means adding two rows, and missing one makes the data say something false.
ConsultantSkill
Consultant
Skill
Asha
Data Quality
Asha
MDM
ConsultantLanguage
Consultant
Language
Asha
English
Asha
French
Fifth Normal Form (5NF) Fagin, 1979
RuleIn 4NF, and the table can't be split into smaller tables and rejoined without loss, unless the split follows its keys. Also called Project-Join Normal Form.
Supply rights, with one business rule: if a supplier supplies a product, the supplier serves a region, and the product is sold in that region, then the supplier supplies that product in that region.
SupplyRights
Supplier
Product
Region
Acme
Widget
North
Acme
Widget
South
Acme
Gadget
South
Bolt
Widget
North
Problem: the three-way table repeats facts that really come from three separate pairings. Adding one pairing can require adding several rows to keep the rule true.
SupplierProduct
Supplier
Product
Acme
Widget
Acme
Gadget
Bolt
Widget
SupplierRegion
Supplier
Region
Acme
North
Acme
South
Bolt
North
ProductRegion
Product
Region
Widget
North
Widget
South
Gadget
South
Joining all three tables gives back exactly the four original rows. Joining only two is not enough. For example, SupplierProduct joined to SupplierRegion invents a false row: Acme, Gadget, North.
Abstraction: supertypes and subtypes
Abstraction removes detail to make a model more widely applicable. Its main tool is generalisation and specialisation: a supertype holds what entities have in common, and each subtype holds what is specific to it. Every subtype inherits the supertype's attributes and relationships. It makes models more flexible and reusable, but harder to understand.
Example: customers can be companies or individuals. Both have an ID and a name, but only companies have a registration number and only people have a date of birth.
Party supertype with two subtypes
Rule
Meaning
Example
Exclusive (disjoint)
An instance belongs to at most one subtype
A party is an Organisation or a Person, never both
Inclusive (overlapping)
An instance can belong to several subtypes
A party can be both a Customer and a Supplier
Complete
Every instance belongs to a subtype
No party is "just a party"
Incomplete
Some instances belong to no subtype
A party not yet classified
Discriminator
An attribute on the supertype that says which subtype applies
PartyType = ORG or PER
Implementing supertypes and subtypes
The logical model shows the hierarchy. The physical model has to choose one of three common designs.
Option 1: one table for the whole hierarchy, with a discriminator column. Simple and fast, but subtype columns are empty for rows they don't apply to.
Party
PartyID
PartyType
Name
CompanyRegNo
DateOfBirth
P1
ORG
Northwind Ltd
01234567
(null)
P2
ORG
Contoso SA
FR998877
(null)
P3
PER
Sam Taylor
(null)
1985-04-12
Option 2: a supertype table plus one table per subtype. Each subtype's primary key is also a foreign key to Party. No empty columns and a faithful match to the logical model, but reading a full record needs a join.
Party
PartyID
PartyType
Name
P1
ORG
Northwind Ltd
P2
ORG
Contoso SA
P3
PER
Sam Taylor
Organisation
PartyID
CompanyRegNo
P1
01234567
P2
FR998877
Person
PartyID
DateOfBirth
P3
1985-04-12
Option 3: one table per subtype only. Each table repeats the shared columns. Fast for single-subtype queries, but "all parties" needs a UNION, and keeping PartyID unique across tables is harder.
Organisation
PartyID
Name
CompanyRegNo
P1
Northwind Ltd
01234567
P2
Contoso SA
FR998877
Person
PartyID
Name
DateOfBirth
P3
Sam Taylor
1985-04-12
Link to RDM and MDM: the Party pattern is a classic master data design, and the discriminator values (ORG, PER) are a reference data code list that needs an owner.
03Activities
Plan for data modelling. Standards, scope, deliverables, tools and how models will be stored and managed.
Build the data model. Forward engineering runs conceptual, then logical, then physical. Reverse engineering documents existing databases, which is common in legacy estates.
Review the data models. Peer review, business validation and checks against standards.
Maintain the data models. Keep them current as requirements and systems change, with version control and lineage.
04Tools
Data modelling tools
Forward and reverse engineering.
Lineage tools
Trace where data comes from and goes.
Data profiling
Check the model matches the real data.
Metadata repositories
Catalogues that store and share models.
Patterns and industry models
Starting points so you don't model from scratch.
05Best practices
Naming conventions
Consistent, business-meaningful names, using class words such as AmountCodeDateIdentifier.
PRISM design principles for databases
PPerformance and ease of use
RReusability
IIntegrity
SSecurity
MMaintainability
06Data model governance
Quality management
Develop data modelling and design standards
Review the quality of data models and database designs
Manage model versioning and integration
Metrics: the Data Model Scorecard
The chapter refers to Steve Hoberman's scorecard, which grades a model against categories such as:
Category
Question it asks
Correctness
How well does the model meet requirements?
Completeness
Is anything missing, or is there scope creep?
Scheme
Does it fit its scheme (conceptual, relational, dimensional)?
Structure
Is it structurally sound?
Abstraction
Does it use generic structures well?
Standards
Does it follow naming standards?
Readability
Is it easy to read?
Definitions
Are definitions clear and complete?
Consistency
Is it consistent with the enterprise model?
Data
Does the metadata match the actual data?
07Context diagram summary
Inputs
Existing data models and databases, data standards, data sets, initial requirements, business processes, enterprise architecture.
Deliverables
Conceptual, logical and physical models; data definitions and lineage; modelling standards.
Participants
Business professionals, data architects, data modellers, DBAs and developers, data stewards.
Consumers
Application and database developers, BI and analytics teams, data governance, business users.
08Links to reference data and data quality
Domains are where RDM and DQ meet the model. A well-defined domain, such as an approved list of country codes, is governed reference data. Many DQ validity rules are domain rules.
"Category information" gives you DMBOK language for arguing that reference data deserves the same design attention as master and transactional data.
Reverse engineering, profiling and the scorecard are practical tools for legacy-heavy clients. They show the gap between what the model says and what the data actually contains.
Relational discipline: domains, keys and integrity constraints come straight from the relational model associated with Codd and Date.
Chapter 6 · Knowledge area
Data Storage and Operations
Designing, implementing and supporting stored data to maximise its value: database technology, transaction processing, backup and recovery, performance, retention and the day-to-day work of database administration.
01Introduction
Definition
Data storage and operations is the design, implementation and support of stored data to maximise its value throughout its lifecycle, from creation or acquisition to disposal.
It has two sub-activities: database support (the data lifecycle activities, from implementing environments to archiving and purging) and database technology support (defining technical requirements, architecture, installing and administering the technology, and resolving issues).
Business drivers
Companies rely on their systems to run. Data storage and operations protects business continuity: data must be available, intact and recoverable.
Goals
Manage the availability of data throughout the data lifecycle
Ensure the integrity of data assets
Manage the performance of data transactions
Database administrator roles
Role
Focus
Production DBA
Runs production databases: performance, backup and recovery, security, availability, change control
Application DBA
Responsible for the databases of one or more applications, across all environments
Procedural and development DBA
Reviews and administers procedural database objects such as stored procedures, triggers and functions
Network storage administrator
Manages the storage arrays and networks that hold the data
02Essential concepts
Database architecture types
Type
Description
Example
Centralised
All data in one system in one place
A single ERP database
Distributed (federated)
Data spread over many nodes. Federated databases give one view over autonomous databases.
Regional order databases queried together
Blockchain
A distributed ledger in which records are linked by cryptographic hashes and can't be altered
Supply chain provenance
Virtualisation and cloud
Databases run on virtual or cloud infrastructure, as a managed service or in containers
A managed cloud database service
Database processing: ACID, BASE and CAP
ACID (relational)
Meaning
Atomicity
All operations in a transaction happen, or none do
Consistency
A transaction moves the database from one valid state to another, meeting all rules
Isolation
Concurrent transactions don't interfere with each other
Durability
Once committed, a transaction survives failures
BASE (many NoSQL stores)
Meaning
Basically Available
The system always responds, even if with stale data
Soft state
Data may be changing as replicas converge
Eventual consistency
Given time with no new updates, all replicas will match
Worked example: an atomic order transaction
Placing order 1003 must record the order, add its line and reduce stock together. If any step fails, none of them should stick.
BEGIN TRANSACTION;
INSERT INTO "Order" (OrderID, OrderDate, CustomerID) VALUES (1003, '2026-03-09', 'C1');
INSERT INTO OrderLine (OrderID, ProductID, Qty) VALUES (1003, 'P20', 5);
UPDATE Stock SET OnHand = OnHand - 5 WHERE ProductID = 'P20';
-- if OnHand would go below zero, a CHECK constraint fails here
COMMIT; -- all three changes become permanent together-- on any error: ROLLBACK; and nothing has changed
The CAP theorem
Eric Brewer's CAP theorem says a distributed system can guarantee at most two of three properties at the same time when the network splits.
CAP: pick two during a network partition
In practice partitions happen, so a distributed system chooses between consistency and availability when they do. A bank ledger chooses consistency; a product catalogue can usually choose availability.
Isolation levels and anomalies
Isolation level
Dirty read
Non-repeatable read
Phantom read
Read uncommitted
Possible
Possible
Possible
Read committed
Prevented
Possible
Possible
Repeatable read
Prevented
Prevented
Possible
Serializable
Prevented
Prevented
Prevented
A dirty read sees another transaction's uncommitted change. A non-repeatable read gets a different value when reading the same row twice. A phantom read finds new rows appearing in a repeated query. Higher isolation means fewer anomalies but more locking.
Scale, flexible schemas, connected data (see Chapter 5)
Common database processes
Archiving
Moving data off primary storage to cheaper media, while keeping it retrievable.
Capacity and growth projections
Planning storage and compute ahead of demand.
Change data capture
Detecting and delivering changes since the last load.
Purging
Permanently removing data that has passed its retention period.
Replication
Copying data to other locations for availability or performance.
Resiliency and recovery
Surviving failures and restoring service and data.
Retention
Keeping data for as long as required, and no longer.
Sharding
Splitting data across servers by a key, so each holds a subset.
Worked example: RPO and RTO
Recovery point objective (RPO) is how much data you can afford to lose, measured in time. Recovery time objective (RTO) is how long you can afford to be down.
Order database failure at 14:00
If the business says it can lose at most 5 minutes of orders, 15-minute log backups aren't enough. Tighter RPO and RTO cost more, so they should be set by the data owner based on business impact.
Worked example: a backup schedule
Backup type
Frequency
Contains
Retention
Full
Sunday 01:00
Entire database
8 weeks
Differential
Monday to Saturday 01:00
Changes since the last full backup
2 weeks
Transaction log
Every 5 minutes
Every committed transaction
7 days
Off-site copy
Daily
Latest full and differential backups
1 year
To recover to 14:00 on Wednesday: restore Sunday's full backup, then Wednesday's differential, then every log backup up to 14:00. This achieves an RPO of 5 minutes. Backups are only proven by regular test restores.
Worked example: sharding by region
Shard A: UK and Ireland
CustomerID
Name
Country
C1
Northwind Ltd
GB
Shard B: continental Europe
CustomerID
Name
Country
C2
Contoso SA
FR
Queries for one region hit one server. A query across all customers must visit every shard. Note that the shard key here is reference data: if country codes are inconsistent, rows land on the wrong shard.
Worked example: a retention schedule
Data
Retention
Trigger
Then
Reason
Orders and invoices
6 years
End of financial year
Archive, then purge
Tax law (UK, illustrative)
Customer contact details
3 years
Last order
Delete or anonymise
Storage limitation (GDPR)
Web server logs
90 days
Creation
Purge
Security monitoring only
Reference code lists
Permanent, versioned
Superseded
Keep history
Needed to interpret old records
03Activities
Manage database technology
Understand database technology characteristics: how products work, their strengths and limits.
Evaluate database technology: against requirements such as volume, velocity, ACID needs, skills and cost.
Manage and monitor database technology: installation, patching, upgrades and vendor relationships.
Manage databases
Understand requirements: storage, usage patterns, availability and recovery needs.
Plan for business continuity: backup, recovery and disaster recovery tested regularly.
Develop database instances: manage physical storage, data access controls, and environments for development, test, QA and production.
Manage database performance: set performance service levels, manage availability, and tune queries and indexes.
Manage test data sets: often masked copies of production data.
Manage data migration: moving data between storage, systems or formats.
Worked example: tuning with an index
-- Slow: scans every order to find one customer's orders
SELECT * FROM "Order" WHERE CustomerID = 'C1';
-- Add an index on the column used in the WHERE clause
CREATE INDEX ix_order_customer ON "Order" (CustomerID);
-- The same query now looks up C1 directly instead of scanning the table
Indexes speed up reads but slow down writes and use storage, so they're chosen based on real query patterns.
Worked example: a performance service level
Measure
Target
Measured over
Availability, 07:00–22:00
99.9%
Monthly
Order entry response time
95% under 1 second
Weekly
Nightly load complete by
06:00
Daily
RPO / RTO
5 minutes / 1 hour
Tested quarterly
04Tools and techniques
Data modelling tools
Generate and compare physical schemas.
Database monitoring tools
Track performance, locks, space and errors.
Database management tools
Administration, configuration and scripting.
Developer support tools
Query editors, profilers and plan analysers.
Test in lower environments
Never make untested changes in production.
Physical naming standards
Consistent names for tables, columns, indexes and constraints.
Script all changes
So they can be reviewed, repeated and rolled back.
05Governance and metrics
Category
Example metrics
Data storage
Count of databases by type; aggregated transaction statistics; capacity used vs available
Performance
Transaction frequency and volume; query performance; service levels met
Operations
Backup success rate; time to restore; data retrieval time; failed changes
Service
Issues raised, resolved and escalated, and resolution time
Governance concerns include data audits and validation, change control, and making sure retention and purging follow policy.
06Context summary
Inputs
Data architecture, data requirements, data models, service level agreements.
Deliverables
Database technology evaluation criteria, database environments, migrated, replicated and versioned data, business continuity plans, database performance OLAs.
Participants
Database administrators, data architects, network and storage administrators, application developers.
Consumers
Data creators and users, application developers, data architects, data modellers, software developers.
07Links to reference data and data quality
Constraints are quality controls. Foreign keys to reference tables, NOT NULL and CHECK constraints stop bad data at the database layer.
Reference data needs versioned retention. Old transactions can't be interpreted if the code lists that described them have been purged.
Shard keys, partition keys and replication rules often depend on reference data such as region or country, so code errors can become operational errors.
Test data must be realistic to catch quality issues, which means masked copies that keep valid code values.
Chapter 7 · Knowledge area
Data Security
Planning, developing and running security policies and procedures so that data and information assets get the right authentication, authorisation, access and auditing.
01Introduction
Definition
Data security is the definition, planning, development and execution of security policies and procedures to provide proper authentication, authorisation, access and auditing of data and information assets.
Security requirements come from four sources:
Stakeholders
Stakeholder concerns
Privacy and confidentiality expected by customers, patients, students, employees and partners.
Government
Regulations
Laws such as GDPR, sector rules and breach notification duties.
Business
Proprietary business concerns
Trade secrets, pricing, plans and research.
Access
Legitimate access needs
People need data to do their jobs; security must enable that, not just block.
Business drivers
Risk reduction: avoid breaches, fines, lawsuits and lost trust.
Business growth: strong security lets organisations share data safely with partners and customers.
Security as an asset: knowing where sensitive data is supports better decisions about it.
Goals and principles
Enable appropriate access, and prevent inappropriate access, to enterprise data assets
Understand and comply with all relevant regulations and policies for privacy, protection and confidentiality
Ensure the privacy and confidentiality needs of all stakeholders are enforced and audited
Principles include collaboration across IT security, data stewards and audit; an enterprise approach; proactive management; clear accountability; metadata-driven classification; and reducing risk by reducing exposure.
02Essential concepts
Vulnerability, threat and risk
Term
Meaning
Example
Vulnerability
A weakness or defect that can be exploited
Unpatched database server; shared admin password
Threat
A potential offensive action against the organisation
Phishing campaign; malicious insider
Risk
The possibility of loss, combining the likelihood of a threat with the impact if it succeeds
Customer data stolen through the unpatched server
Worked example: a risk register
Risk
Likelihood (1–5)
Impact (1–5)
Score
Rating
Control
Phishing exposes CRM credentials
4
4
16
High
MFA; awareness training
Unmasked production data in test
3
4
12
Medium
Persistent masking for all test copies
Lost unencrypted laptop
2
5
10
Medium
Full-disk encryption
Defacement of public price list
2
2
4
Low
Web application firewall
The CIA triad
C
Confidentiality
Only authorised people can see the data.
I
Integrity
Data can't be changed improperly, and changes are traceable.
A
Availability
Authorised people can use the data when they need it.
The four As, plus entitlement
Process
Question
Example
Access
Can this person connect to the system or data?
Asha can log in to CRM
Audit
What did people do, and was it allowed?
Log of who exported the customer list
Authentication
Are they who they say they are?
Password plus an authenticator app
Authorisation
What are they allowed to do?
Asha may edit addresses but not credit limits
Entitlement
The total of all the permissions a person holds
Asha's role grants CRM edit and warehouse read
Data classification
The DMBOK distinguishes two kinds of classification. Confidentiality levels come from business need; regulatory categories come from external rules. A data element can have one of each.
Confidentiality level
Meaning
For general audiences
Available to anyone, including the public
Internal use only
Limited to employees or members; minimal risk if shared
Confidential
Can't be shared outside the organisation without a valid need and agreement
Restricted confidential
Limited to people performing certain roles with a need to know
Registered confidential
So sensitive that anyone accessing it must sign a legal agreement
Regulatory categories include personally identifiable information (PII), financially sensitive data, medically sensitive data (PHI), educational records, and payment card data (PCI DSS).
Worked example: classifying Northwind's data
Data element
Confidentiality
Regulatory category
Handling
Product name and list price
General audiences
None
Public website
Country code list
General audiences
None
Shared freely
Customer company name
Internal use only
None
Employees
Contact person's email and phone
Confidential
PII
Sales and support roles; masked in test
Customer credit limit
Restricted confidential
Financially sensitive
Finance roles only
Stored card numbers
Registered confidential
PCI
Tokenised; vault access only
Role-based access
Permissions are granted to roles, and people are assigned to roles. That's easier to manage and audit than granting rights person by person.
Role and data access grid
Role
Company name
Contact details
Credit limit
Card token
Sales rep
Read / write
Read / write
Read
None
Support agent
Read
Read
None
None
Credit controller
Read
Read
Read / write
None
Data analyst
Read
Masked
Banded
None
Payments service
None
None
None
Use
Roles can be arranged in a hierarchy, so a senior role inherits the permissions of junior ones. Combine this with the principle of least privilege (give only what's needed) and separation of duties (no one person can both create and approve a payment).
Worked example: row-level security
-- Sales reps see only customers in their own region
CREATE POLICY region_filter ON Customer
FOR SELECT
USING (Region = current_setting('app.user_region'));
-- A rep in region 'UKI' querying Customer sees C1 (GB) but not C2 (FR)
Encryption and hashing
Method
How it works
Reversible?
Use
Hash
A one-way function turns input into a fixed-length digest (for example SHA-256)
No
Storing passwords (with a salt); integrity checks
Symmetric (private key)
The same secret key encrypts and decrypts (for example AES)
Yes, with the key
Encrypting databases and files
Asymmetric (public key)
A public key encrypts; only the matching private key decrypts (for example RSA)
Yes, with the private key
Exchanging keys; secure web connections; signatures
Encrypt data at rest (stored), in transit (on the network) and, where possible, in use. Key management matters as much as the algorithm: a key stored next to the data protects nothing.
Data masking
Type
How
Example
Persistent masking
Data is permanently masked when copied, for example from production to test
Test CRM holds fictional contact names
Dynamic masking
Data is masked on the fly when shown to users without permission; the stored data is unchanged
Support agents see j***@example.com
Masking methods include substitution, shuffling, temporal variance (shifting dates), value variance (adding small random amounts), nulling or deleting, randomisation, encryption, expression masking and key masking. Chapter 2 has worked examples.
System security risks
Risk
Description
Countermeasure
Abuse of excessive privilege
Users hold more rights than their job needs
Least privilege; regular access reviews
Abuse of legitimate privilege
Authorised users misuse their access
Monitoring; separation of duties
Unauthorised privilege elevation
Attackers exploit flaws to gain higher rights
Patching; intrusion prevention
Service account or shared account abuse
Shared credentials make actions untraceable
Named accounts; vaulted secrets
Platform intrusion attacks
Exploiting operating system or database flaws
Patching; hardening
SQL injection
Malicious input is executed as part of a query
Parameterised queries; input validation
Default passwords
Factory or default credentials left unchanged
Change on installation; scan regularly
Backup data abuse
Unprotected backups are stolen
Encrypt backups; control access
Social engineering and phishing
People are tricked into revealing credentials or data
Training; MFA; reporting routes
Malware
Viruses, worms, ransomware, spyware
Endpoint protection; tested backups
Worked example: preventing SQL injection
-- Unsafe: the user's input is pasted into the query text
query = "SELECT * FROM Customer WHERE CustomerID = '" + userInput + "'"
-- Safe: a parameterised query treats the input only as a value
query = "SELECT * FROM Customer WHERE CustomerID = ?"
execute(query, [userInput])
03Activities
Identify data security requirements: business requirements, external regulatory restrictions, and the rules implied by them.
Define data security policy: enterprise security policy, IT security policy and data security policy, with clear ownership.
Define data security standards: confidentiality levels, regulatory categories, security roles, and how risk is assessed.
Assess current security risks: identify sensitive data, where it is stored, and the threats and vulnerabilities around it.
Implement controls and procedures: assign confidentiality levels and regulatory categories, manage and maintain security, control access, manage users, passwords and groups, and monitor user authentication and access behaviour.
Worked example: a quarterly access review
User
Role held
Last used
Manager decision
Asha
Sales rep
Yesterday
Keep
Ben
Credit controller
142 days ago
Remove (moved team)
Chloe
Data analyst
Last week
Keep
svc_export
Service account, full read
Daily
Reduce scope
04Tools and techniques
Anti-virus and endpoint protection
Protect devices from malware.
HTTPS and network security
Encrypt data in transit.
Identity management
Single sign-on, MFA, provisioning and de-provisioning.
Intrusion detection and prevention
Spot and block attacks.
Firewalls
Control network traffic.
Metadata tracking
Tag sensitive data so controls follow it.
Data masking and encryption
Protect data even if accessed.
CRUD matrix and security patches
Know who touches what; keep systems patched.
05Governance and metrics
Category
Example metrics
Implementation
Percentage of systems with current patches; percentage of sensitive data classified; percentage of staff trained
Protection
Count of intrusion attempts blocked; percentage of sensitive data encrypted or masked
Policy compliance
Access reviews completed on time; exceptions approved and expired
Incidents
Number and severity of security incidents; time to detect and to contain
06Context summary
Inputs
Business goals and strategy, business rules and processes, regulatory requirements, enterprise architecture standards, enterprise data model.
Deliverables
Data security architecture, data security policies, privacy and confidentiality standards, access controls, user access views, audit trails and reports, security assessment reports.
Participants
Data stewards, information security team, internal auditors, process analysts, IT governance, data architects.
Consumers
Business users, regulatory auditors.
07Links to reference data and data quality
Classification levels and regulatory categories are reference data. They need a governed code list, owner and definitions, or controls are applied inconsistently.
Integrity is part of security and overlaps directly with data quality: unauthorised changes are a quality failure.
Masked test data should keep valid reference codes (for example real country codes with fictional names), or testing won't catch validation problems.
Over-restrictive security harms quality by pushing people to keep private spreadsheet copies that drift from the source.
Chapter 8 · Knowledge area
Data Integration and Interoperability
Moving and consolidating data within and between data stores, applications and organisations: extraction, transformation, loading, latency, interaction models, mapping and lineage.
01Introduction
Definition
Data integration and interoperability (DII) describes processes related to the movement and consolidation of data within and between data stores, applications and organisations.
Integration consolidates data into consistent forms, physical or virtual. Interoperability is the ability of multiple systems to communicate. DII underpins data warehousing, master data management, migration, and sharing with partners.
Business drivers
Managing complexity: without a plan, every new system adds point-to-point interfaces
Reducing the cost of maintaining many custom interfaces
Supporting compliance through consistent, traceable data movement
Enabling analytics, master data and data sharing
Goals
Make data available in the format and timeframe needed by consumers
Consolidate data physically and virtually into data hubs
Lower cost and complexity by developing shared models and interfaces
Identify meaningful events and automatically trigger alerts and actions
Support business intelligence, analytics, master data management and operational efficiency
02Essential concepts
Extract, transform, load
Extract
Select and copy
Pull the required data from sources, often into a staging area.
Transform
Make it compatible
Change format, structure and meaning to fit the target.
Load
Store or present
Write to the target, or deliver to the consumer.
ETL versus ELT
Sources→Transform in integration engine→Load targetETL
Sources→Load raw into target→Transform inside targetELT
ELT has become common with cloud warehouses and lakes, which have the power to transform data where it lands. ELT keeps raw data available, but needs discipline so raw data isn't used by mistake.
Types of transformation
Type
What changes
Example
Format
Representation of values
"02/03/2026" (UK text) to DATE 2026-03-02
Structure
Shape of the data
A flat file split into Customer and Order tables
Semantic
Meaning made consistent
"UK", "GBR" and "826" all become "GB"
Enrichment
Values added
Region derived from country
Aggregation
Detail summarised
Daily sales totals from order lines
De-duplication
Repeated records removed or linked
Two Northwind records matched to one
Latency
Latency is the time between data being created in a source and being available in a target.
Approach
How it works
Typical latency
Use
Batch
Data moves in scheduled groups
Hours to a day
Nightly warehouse loads
Change data capture (CDC)
Only changed data moves, detected from logs, timestamps or flags
Minutes
Keeping a replica close to current
Near real time and event-driven
Small sets move as events occur, often via message queues
Seconds
Stock levels; fraud alerts
Asynchronous
The sender doesn't wait for the receiver to confirm
Seconds
Order confirmation emails
Real time, synchronous
The sender waits until the receiver confirms
Milliseconds
Payment authorisation
Low latency or streaming
Continuous flows processed as they arrive
Sub-second
Sensor data; trading
Worked example: change data capture
Change log read from the CRM database
LSN
Time
Operation
Table
Key
Changed values
10471
2026-06-15 09:02
UPDATE
Customer
C1
Name: Northwind Ltd → Northwind Group Ltd
10472
2026-06-15 09:05
INSERT
Customer
C3
Name: Fabrikam GmbH, Country: DE
10473
2026-06-15 09:11
DELETE
Contact
K77
(row removed)
The integration process remembers the last log sequence number (LSN) it processed and reads only newer entries. Only three rows move, not the whole customer table.
Replication and archiving
Replication keeps copies of data in multiple places for availability, performance or distribution. Archiving moves rarely used data to cheaper storage, and integration processes must still be able to read it, including old formats.
Enterprise message format and canonical model
A canonical model is a common format that every system translates to and from, so each system needs one translation instead of one per partner.
{
"messageType": "CustomerUpdated",
"version": "1.2",
"occurredAt": "2026-06-15T09:02:00Z",
"source": "CRM",
"customer": {
"customerId": "C1",
"legalName": "Northwind Group Ltd",
"countryCode": "GB", // ISO 3166-1 alpha-2, from the governed code list
"status": "ACTIVE" // from the CustomerStatus code list
}
}
Interaction models
Point-to-point: every system connects to every otherHub-and-spoke: every system connects once, to the hub
Connections needed if every system exchanges data with every other
Systems (n)
Point-to-point: n(n−1)/2
Hub-and-spoke: n
5
10
5
10
45
10
20
190
20
50
1,225
50
Model
Strength
Weakness
Point-to-point
Simple for a few systems; fast
Connections grow with the square of the number of systems; logic duplicated
Hub-and-spoke
One translation per system; central monitoring
The hub can become a bottleneck or single point of failure
Publish and subscribe
Publishers don't need to know subscribers; new consumers join easily
Needs good event design, ordering and error handling
Worked example: publish and subscribe
CRM publishes a CustomerUpdated event when Northwind renames itself. Each subscriber reacts in its own way:
Subscriber
Action on CustomerUpdated
Billing
Updates the name on future invoices
Data warehouse
Adds a new SCD Type 2 row for C1
Master data hub
Re-runs matching for C1
Marketing
Ignores it (not subscribed to name changes)
DII architecture concepts
Concept
Description
Application coupling
How dependent systems are on each other. Loose coupling, through services and messages, is preferred.
Orchestration and process controls
Coordinating the order of data flows and checking each step: job scheduling, audit and balancing, alerts
Enterprise application integration (EAI)
Software modules interact only through well-defined interfaces
Enterprise service bus (ESB)
Middleware that routes and transforms messages between systems
Service-oriented architecture (SOA)
Functionality exposed as reusable services with defined contracts
Complex event processing (CEP)
Combining many events to detect meaningful patterns, such as possible fraud
Data federation and virtualisation
Querying data in place across sources, without moving it
Data as a service (DaaS)
Data provided on demand by a service, often from a third party
Cloud-based integration
Integration platforms delivered as cloud services (iPaaS)
Data exchange standards
Formal agreements on the structure of exchanged data, often sector-wide: for example ISO 20022 in financial messaging, HL7 and FHIR in healthcare, and EDI standards in supply chains. Standards cut the cost of each new partner connection.
03Activities
1. Plan and analyse
Define data integration and lifecycle requirements: what data, how often, which latency, what volumes, which rules.
Perform data discovery: find candidate sources and understand them.
Document data lineage: how data flows today.
Profile data: check content, formats and quality before designing transformations.
Collect business rules: for validation, transformation and matching.
2. Design DII solutions
Design the integration architecture: interaction model, latency, tools
Model data hubs, interfaces, messages and data services
Map data sources to targets
Design data orchestration
Worked example: a source-to-target mapping
CRM.Account to Warehouse.DimCustomer
Source field
Sample value
Target field
Rule
Result
acct_no
C1
CustomerID
Direct copy; trim spaces
C1
acct_name
NORTHWIND LTD.
CustomerName
Proper case; remove trailing full stop
Northwind Ltd
ctry
UK
CountryCode
Look up in the country crosswalk to ISO 3166-1 alpha-2; reject if no match
GB
created
02/03/2026
FirstSeenDate
Parse as DD/MM/YYYY
2026-03-02
(none)
CustomerKey
Generate surrogate key
1001
(none)
Region
Derive from CountryCode via the country-to-region list
UKI
Country crosswalk (reference data)
Source system
Source code
ISO code
CRM
UK
GB
CRM
France
FR
ERP
GBR
GB
Billing
826
GB
Most of the hard work in a mapping is semantic, and most semantic transformations are reference data lookups. See Chapter 10 for crosswalks.
3. Develop DII solutions
Develop data services, data flows and integration jobs; develop the data migration approach; develop a publication approach; develop complex event processing flows; and maintain DII metadata.
4. Implement and monitor
Activate the services, monitor them against service levels, and handle errors: rejected records go to an exception queue with a clear reason so stewards can fix them at source.
Worked example: audit and balance controls
Check
Source
Loaded
Rejected
Result
Row count, customers
2,504
2,498
6
Balances
Sum of order value (£)
1,284,310.55
1,284,310.55
0.00
Balances
Unmapped country codes
–
–
6
Steward review: "U.K.", "Eire"
04Tools and techniques
Data transformation engine (ETL tool)
Builds and runs extract, transform and load jobs.
Data virtualisation server
Presents a virtual view over multiple sources.
Enterprise service bus
Routes messages between systems.
Business rules engine
Holds transformation and validation rules outside code.
Data and process modelling tools
Design flows and structures.
Data profiling tool
Understand source data before mapping.
Metadata repository
Stores mappings and lineage.
05Governance and metrics
Data availability: availability of data requested.
Data volumes and latency: volumes transported and transformed; time from source to target.
Data service response times.
Mean time to resolve integration issues; cost of developing and maintaining interfaces.
Governance covers data sharing agreements, lineage documentation, and approval of new interfaces against the integration architecture.
06Context summary
Inputs
Business goals and strategy, data needs and standards, regulatory and compliance requirements, enterprise architecture, data models, data quality requirements.
Deliverables
DII architecture, data exchange specifications, data access agreements, data services, complex event processing thresholds and alerts.
Participants
Data architects, business and data analysts, data modellers, data stewards, ETL, service and interface developers, project and programme managers.
Consumers
Information consumers, knowledge workers, managers and executives.
07Links to reference data and data quality
Semantic transformation is mostly reference data work. Crosswalks such as the country table above are the most reused assets in any integration layer.
Integration is where quality problems surface. Profiling sources first and routing rejects to stewards turns integration into a quality feedback loop.
Canonical models depend on governed code lists; without them, every interface invents its own values.
For the workshop's "where to start" question: integration delivers value fastest when reference data is agreed first, because every mapping reuses it.
Chapter 9 · Knowledge area
Document and Content Management
Controlling the capture, storage, access and use of data held outside relational databases: documents, records, email, images and web content, plus the vocabularies and taxonomies that make them findable.
01Introduction
Definition
Document and content management covers planning, implementation and control activities for the lifecycle management of data and information found in any form or medium.
Much of an organisation's information is unstructured: contracts, emails, presentations, scanned forms, images, video and web pages. The same principles apply as for structured data: it needs to be trustworthy, protected, findable and disposed of on time.
Business drivers
Regulatory compliance: laws require records to be kept for set periods and produced on request.
Litigation and e-discovery: organisations must find and preserve relevant documents quickly.
Efficiency: people waste time searching for, recreating or using wrong versions of documents.
Goals
Ensure effective and efficient retrieval and use of data and information in unstructured formats
Ensure integration capabilities between structured and unstructured data
Comply with legal obligations and customer expectations
Principles
Everyone has a role in protecting the organisation's future: everyone creates, uses, retrieves and disposes of records. Experts in handling records and content should be fully engaged in policy and planning.
02Essential concepts
Content, documents and records
Term
Meaning
Example
Content
Data and information inside a file, document or website
The text and images of a product page
Document
A unit of content, electronic or paper, that can be managed
A draft proposal for Northwind
Record
A document kept as evidence of an activity or transaction, which must not change
The signed contract with Northwind
All records are documents, but not all documents are records. A draft becomes a record when it is declared final as evidence.
The content lifecycle
Create or capture→Manage (version, review, approve)→Store→Publish and deliver→Retain→Dispose or preserve
Content management
Content metadata: describes content so it can be found, managed and protected.
Content modelling: breaks content into components and defines their structure and relationships, so pieces can be reused.
The Dublin Core Metadata Initiative defines fifteen widely used elements. A record for Northwind's contract:
Element
Value
Title
Master Services Agreement: Northwind Ltd
Creator
Legal Department
Subject
Customer contracts; Data services
Description
Three-year services agreement, signed
Date
2026-02-20
Type
Text
Format
application/pdf
Identifier
CON-2026-0042
Language
en-GB
Relation
Customer C1
Rights
Confidential
The Relation element links the document to customer C1 in the master data, joining unstructured and structured data. Subject and Rights values should come from controlled vocabularies.
Controlled vocabularies
A controlled vocabulary is a defined list of explicitly allowed terms used to index, categorise, tag, sort and retrieve content. They range from simple lists to rich ontologies:
Controlled list with one preferred form of each name
"Northwind Ltd" preferred over "Northwind Limited"
Taxonomy
Hierarchy of broader and narrower categories
Electronics → Computers → Laptops
Classification scheme
Taxonomy with codes, often for filing
Records file plan: FIN-03-02
Thesaurus
Terms with broader, narrower, related and equivalence relationships
See below
Ontology
Formal model of concepts, properties and relationships, which machines can reason over
A Customer places an Order; an Order contains Products
Types of taxonomy
Flat
No hierarchy: a simple set of categories.
Hierarchical
A tree; each item has one parent.
Polyhierarchy
An item can sit under more than one parent: "Docking station" under Accessories and under Laptops.
Facet
Independent dimensions combined freely: type × brand × price band.
Network
Items linked by many kinds of relationship.
Worked example: a thesaurus entry
Term: Laptop
Relationship
Code
Term
Broader term
BT
Computers
Narrower term
NT
Ultrabook; 2-in-1 laptop
Related term
RT
Docking station; Laptop bag
Used for (non-preferred)
UF
Notebook computer; Portable computer
Scope note
SN
Portable personal computers with integrated screen and keyboard
Someone searching for "notebook computer" is redirected (USE) to the preferred term Laptop, so they find everything tagged Laptop.
Worked example: faceted classification
Product
Category facet
Brand facet
Price band facet
Connectivity facet
P10 Laptop
Computers
Brand A
£500–£999
Wi-Fi, USB-C
P20 Mouse
Accessories
Brand B
Under £50
Bluetooth
P30 Monitor
Displays
Brand A
£100–£499
HDMI, USB-C
A shopper can filter "Brand A" and "USB-C" and get P10 and P30, without anyone building a "Brand A USB-C" category.
Semantic models and linked data
Semantic technologies represent knowledge as statements a machine can process. In RDF (Resource Description Framework), every fact is a triple: subject, predicate, object. SPARQL queries RDF data, and schema.org provides shared vocabularies for web content.
@prefix ex: <https://example.com/id/> .
@prefix schema: <https://schema.org/> .
ex:C1 a schema:Organization ;
schema:legalName "Northwind Group Ltd" ;
schema:address [ schema:addressCountry "GB" ] .
ex:CON-2026-0042 schema:about ex:C1 . # the contract is about customer C1
Records management
ARMA International's Generally Accepted Recordkeeping Principles (GARP) set out what good recordkeeping looks like:
Accountability
A senior executive oversees recordkeeping.
Integrity
Records are authentic and reliable.
Protection
Records are protected appropriately.
Compliance
Recordkeeping meets laws and policies.
Availability
Records can be retrieved efficiently.
Retention
Records are kept for the right time.
Disposition
Records are disposed of securely when due.
Transparency
Processes are documented and understandable.
Worked example: a records retention schedule
Record series
Retention
Trigger
Disposition
Owner
Customer contracts
Contract end + 6 years
Contract end
Secure destruction
Legal
Invoices
6 years
End of financial year
Secure destruction
Finance
Board minutes
Permanent
–
Archive
Company Secretary
Marketing drafts
1 year
Creation
Delete
Marketing
Job applications (unsuccessful)
6 months
Decision
Delete
HR
Periods are illustrative. A legal hold suspends disposal for anything relevant to litigation or investigation until the hold is lifted.
E-discovery
E-discovery is the process of finding and producing electronically stored information for legal proceedings. The Electronic Discovery Reference Model (EDRM) describes the stages:
Information governance→Identification→Preservation→Collection→Processing→Review→Analysis→Production→Presentation
Good information governance at the start, especially disposing of records on schedule, makes every later stage cheaper. Data that was kept past its retention period can still be discoverable.
Information architecture and search
Information architecture organises content so people can find it: navigation, labels, taxonomies and search. Search engines index content and metadata; quality depends on good metadata and vocabularies, not just the engine.
03Activities
1. Plan for lifecycle management
Plan for records management: including a records file plan and retention schedule.
Develop a content strategy: what content exists, who it's for and how it's governed.
Create content handling policies: social media, device access, handling sensitive data, responding to litigation.
Define content information architecture.
2. Manage the lifecycle
Capture records and content, including scanning paper
Manage versioning and control: check-in and check-out, approval workflows
Back up and recover
Manage retention and disposal
Audit documents and records
3. Publish and deliver content
Provide access, search and retrieval, and deliver content through the right channels, while enforcing security and rights.
04Tools and techniques
Enterprise content management systems
Document management, records management, digital asset management, workflow, collaboration.
Collaboration tools
Shared workspaces and team sites.
Controlled vocabulary and metadata tools
Manage taxonomies and thesauri.
Standard markup and exchange formats
XML, JSON, RDF, schema.org.
E-discovery technology
Legal hold, collection and review.
Social media policies
Rules for official and personal posts.
05Governance and metrics
Area
Example metrics
Records management
Compliance with retention schedules; percentage of records disposed of on time; legal holds managed
E-discovery
Time and cost to respond to requests
Content management
Search success rate; content reuse; user satisfaction; percentage of content with required metadata
06Context summary
Inputs
Business strategy, IT strategy, legal retention requirements, text files, email, documents, images, audio, video, social media content.
Deliverables
Content and records management strategy, policies and procedures, content repository, managed records, audit trails and logs.
Participants
Records managers, content managers, taxonomists, information architects, legal and compliance, IT.
Consumers
Business users, IT users, government regulatory agencies, auditors, external litigants and regulators, external customers.
07Links to reference data and data quality
Controlled vocabularies are reference data. Pick lists, taxonomies and thesauri need owners, versioning and change control, exactly like code lists.
Authority lists overlap with master data: one preferred form of a customer or supplier name.
Metadata quality decides findability. Missing or inconsistent tags are a data quality problem in unstructured content.
Retention schedules depend on classification, which depends on a governed list of record series.
Chapter 10 · Knowledge area
Reference and Master Data
Managing the shared data that the rest of the organisation depends on: code lists and hierarchies (reference data), and the core business entities such as customers, products and suppliers (master data), from matching and survivorship to hub architectures.
01Introduction
Definition
Reference and master data management covers managing shared data to meet organisational goals, reduce risks associated with data redundancy, ensure higher quality, and reduce the costs of data integration.
Different systems often hold the same data in different forms: three customer records for Northwind, four codes for the United Kingdom. Reference and master data management (RDM and MDM) create agreed, shared versions that every system can rely on.
Reference, master and transactional data
Characteristic
Reference data
Master data
Transactional data
What it is
Values used to classify or categorise other data
Data about the core business entities
Records of business events
Examples
Country codes, currencies, order status, product categories
Meeting organisational data requirements: many processes need the same customer, product and code values.
Managing data quality: inconsistent shared data causes errors that spread.
Managing the costs of data integration: every interface that maps codes or matches customers repeats the same work.
Reducing risk: regulators expect a consistent view of customers, counterparties and products.
Goals and principles
Enable sharing of information assets across business domains and applications
Provide an authoritative source of reconciled and quality-assessed master and reference data
Lower cost and complexity through standards, common data models and integration patterns
Shared data
Reference and master data must be managed so they can be shared across the organisation.
Ownership
It belongs to the organisation, not to an application or department.
Quality
It needs ongoing quality monitoring and governance.
Stewardship
Business stewards are accountable for controlling and ensuring its quality.
Controlled change
Master data values represent the organisation's best understanding at a point in time; reference data values change only through a defined process.
Authority
Values should come only from the system of record; a reference system should enable reciprocal sharing.
02Reference data
Definition
Reference data is any data used to characterise or classify other data, or to relate data to information external to an organisation.
Reference data looks simple, but it is everywhere. Every coded field in every system points to a reference data set, and an error in one value affects every record that uses it.
Reference data structures
The DMBOK describes four main structures, from simplest to richest.
1. Lists
A code paired with a description, sometimes with a definition.
OrderStatus
Code
Description
Definition
NEW
New
Order captured but not yet checked
CONF
Confirmed
Stock and credit checks passed
SHIP
Shipped
Handed to carrier
CANC
Cancelled
Cancelled before shipment
Definitions matter as much as codes. Without one, "Confirmed" might mean "customer confirmed" to Sales and "credit approved" to Finance.
2. Cross-reference lists (crosswalks)
Different systems and standards use different codes for the same thing. A crosswalk maps them.
Country crosswalk
ISO alpha-2
ISO alpha-3
ISO numeric
CRM
ERP
Billing
Name
GB
GBR
826
UK
GBR
826
United Kingdom
FR
FRA
250
France
FRA
250
France
DE
DEU
276
Germany
DEU
276
Germany
IE
IRL
372
Eire
IRL
372
Ireland
The CRM uses non-standard values ("UK", "Eire", full names). The crosswalk lets integration translate them consistently until the CRM is fixed at source.
3. Taxonomies
Hierarchies that support classification at different levels of detail, such as product categories or industry codes.
Product taxonomy with parent codes
Code
Name
Parent
Level
EL
Electronics
–
Department
EL-CMP
Computers
EL
Category
EL-CMP-LAP
Laptops
EL-CMP
Subcategory
EL-ACC
Accessories
EL
Category
EL-ACC-MSE
Mice
EL-ACC
Subcategory
EL-DSP
Displays
EL
Category
Industry examples: the North American Industry Classification System (NAICS), UK Standard Industrial Classification (SIC), and the UN Standard Products and Services Code (UNSPSC).
4. Ontologies
Ontologies add formal relationships and rules on top of taxonomies, so machines can reason about the meaning. For example: "a Laptop is a Computer; a Computer requires a Power Supply". See Chapter 9 for RDF examples.
Sources of reference data
Type
Description
Examples
Proprietary or internal
Created and owned by the organisation
Order status, customer segment, internal product categories
Every reference data set should be described, so people know what it is, where it came from and whether they can trust it.
Metadata for the Country reference data set
Attribute
Value
Name
Country
Description
Countries and territories for addresses, tax and reporting
Authoritative source
ISO 3166-1, extended with internal crosswalk columns
Owner
Head of Finance Operations
Steward
Asha
Current version
2026.2, effective 2026-07-01
Update trigger
ISO newsletter changes; internal requests
Number of values
249
Distribution
Reference data service API; nightly file; event on change
Consumers
CRM, ERP, Billing, warehouse, web shop
Classification
For general audiences
Reference data change and versioning
Reference data changes rarely, but when it does, the change must be controlled, dated and communicated. Old values must stay available to interpret historical records.
Request→Impact analysis→Owner approves→Publish new version with effective date→Notify consumers→Systems adopt
Worked example: adding and retiring product categories
New smartwatch range launching in September; no PDAs sold since 2019
Impact
Web shop navigation, warehouse product dimension, two BI reports; 14 historical products remain in EL-PDA
Decision
Approved by the product data owner, 2026-08-04
Effective
2026-09-01
ProductCategory after the change
Code
Name
ValidFrom
ValidTo
Status
EL-CMP
Computers
2015-01-01
9999-12-31
Active
EL-PDA
PDAs
2015-01-01
2026-08-31
Retired
EL-WEA
Wearables
2026-09-01
9999-12-31
Active
EL-PDA is end-dated, not deleted. Historical products and sales still resolve to a valid description, and new records can't use the retired code.
03Master data
Definition
Master data is data about the business entities (such as employees, customers, products, financial structures, assets and locations) that provide context for business transactions and analysis.
System of record, system of reference and golden record
Term
Meaning
Example
System of record
The authoritative system where data is created and maintained under defined rules
Billing for credit limits; CRM for contact details
System of reference
An authoritative system that consumers can rely on, even though it may not create the data itself
The MDM hub publishing the golden customer record
Trusted source, golden record
The best version of the truth for an entity, built from the best available data
Master customer M-000001
Master data domains
Party
Party master data
Individuals and organisations and their roles: customers, suppliers, employees, partners. Customer data integration and customer relationship management sit here.
Financial
Financial master data
Business units, cost centres, profit centres, general ledger accounts, budgets and projects.
Legal
Legal master data
Contracts, regulations and other legal matters.
Product
Product master data
Internal products and services or industry-wide products. Often managed with product lifecycle management (PLM) tools.
Location
Location master data
Business addresses and positions, geopolitical boundaries and their relationships.
Industry
Industry master data and directories
Externally maintained lists of companies, people and products, such as company registers.
Key MDM processing steps
Data model management: agree the master model and definitions.
Data acquisition: bring in records from source systems and external providers.
Data validation, standardisation and enrichment: check values, put them in standard formats, and add data from trusted sources.
Entity resolution and identifier management: work out which records refer to the same real-world entity, and give it one identifier.
Data sharing and stewardship: publish the master data and manage exceptions.
04Worked example: from four records to one golden record
Four systems hold customer records that might be Northwind. This example follows them through standardisation, matching, survivorship and cross-referencing.
Step 1: source records
Source
Source ID
Name
Address
Country
Phone
Other
CRM
C1
Northwind Ltd
12 Dock St, Manchester M1 4BT
UK
0161 496 0000
Billing
7781
NORTHWIND LIMITED
12 Dock Street, Manchester, M1 4BT
826
+44 161 496 0000
Credit limit 50,000
ERP
N-0042
Northwind Ltd.
12 Dock St
GBR
(blank)
Company no. 01234567
Web shop
W-559
Northwind Logistics
3 Canal Rd, Leeds LS1 1AA
GB
0113 496 0123
Step 2: validate and standardise
Field
Rule
Before
After
Name
Upper case; remove punctuation; "LIMITED" → "LTD"
NORTHWIND LIMITED; Northwind Ltd.
NORTHWIND LTD
Address
Expand abbreviations; parse postcode
12 Dock St
12 DOCK STREET
Country
Translate through the country crosswalk
UK; 826; GBR
GB
Phone
Format to international E.164
0161 496 0000
+441614960000
Standardisation runs before matching. Without the country crosswalk, "UK", "826" and "GBR" would look like three different countries and the records would score lower.
Step 3: match
Two matching approaches:
Approach
How it works
Example rule
Deterministic
Records match if specified fields are equal
Same company registration number; or same standardised name and postcode
Probabilistic
Fields are compared for similarity and weighted into an overall score
Name 50%, address 35%, phone 15%
Probabilistic scores for each pair, compared with CRM record C1 (a missing value scores a neutral 0.5):
The steward checks the ERP record, finds the company number matches Northwind's, and confirms the match. W-559 Northwind Logistics is a different company in Leeds, so it gets its own master record.
Raising the threshold reduces false positives but increases false negatives. The thresholds are a business decision, set by data owners based on which error costs more.
The DMBOK names three types of match rule: duplicate identification match rules (find likely duplicates), match-merge rules (combine matched records into one, with survivorship) and match-link rules (link matched records through a cross-reference, without merging them).
Step 4: survivorship
Survivorship rules decide which source supplies each attribute of the golden record.
Attribute
Survivorship rule
Winning source
Golden value
Legal name
Most trusted source for names
CRM
Northwind Ltd
Address
Most complete standardised address
CRM / Billing (equal)
12 Dock Street, Manchester M1 4BT
Country
Crosswalked value; all agree
All
GB
Phone
Most recently verified
Billing
+44 161 496 0000
Company number
Only source
ERP
01234567
Credit limit
System of record for credit
Billing
50,000
Step 5: golden record and cross-reference
Master customer (golden record)
MasterID
LegalName
Address
Country
Phone
CompanyNo
CreditLimit
M-000001
Northwind Ltd
12 Dock Street, Manchester M1 4BT
GB
+44 161 496 0000
01234567
50,000
M-000002
Northwind Logistics
3 Canal Road, Leeds LS1 1AA
GB
+44 113 496 0123
–
–
Cross-reference (xref)
MasterID
Source
SourceID
MatchMethod
LinkedOn
M-000001
CRM
C1
Anchor record
2026-03-01
M-000001
Billing
7781
Auto, score 1.00
2026-03-01
M-000001
ERP
N-0042
Steward confirmed, score 0.82
2026-03-03
M-000002
Web shop
W-559
Anchor record
2026-03-01
The cross-reference is what lets any system ask "what is the master ID for Billing customer 7781?" It also keeps the audit trail of how each link was made, so a wrong match can be undone.
05Data sharing architectures
The DMBOK describes three basic approaches to sharing master data: registry, transaction hub and a consolidated (hybrid) approach. Industry analysts often add a fourth, coexistence. Coexistence: beyond DMBOK
Sources keep their data; the hub holds an index of identifiers and cross-references, and assembles a view on request
Least disruptive; quick to start
No single cleansed copy; quality depends on sources; slower queries
Consolidated
Sources feed a hub that matches and cleanses into a golden copy, mainly for reporting and analytics
Good for BI; sources unchanged
Corrections don't flow back to sources
Coexistence
Like consolidated, but the golden record is synchronised back to sources
Sources improve over time
Two-way sync is complex; conflict rules needed
Transaction hub
Master data is created and maintained in the hub; all systems use it
Highest consistency and control
Most disruptive; source applications must change
Many organisations start with registry or consolidated, prove value, then move toward coexistence or a transaction hub for selected domains.
06Activities
MDM activities
Define MDM drivers and requirements: which problems, which domains, which systems.
Evaluate and assess data sources: profile candidate sources for quality, overlap and coverage.
Define the architectural approach: registry, consolidated, coexistence or hub.
Model master data: agree the master model and definitions.
Define stewardship and maintenance processes: who reviews matches, fixes exceptions and approves changes.
Establish governance policies to enforce use of master data: new systems must use the hub; local copies aren't allowed.
RDM activities
Define drivers and requirements: which code lists cause the most pain or risk.
Assess data sources: find every copy of each code list and how they differ.
Define the architectural approach: a central reference data service, distribution methods and versioning.
Model reference data sets: structure, crosswalks and hierarchies.
Define stewardship and maintenance processes: owners, change requests and effective dating.
Establish policies: systems take reference data from the authoritative source.
Worked example: a reference data inventory
Code list
Copies found
Value differences
Owner
Business impact
Priority
Country
6
31
Finance Ops
Invoice and tax errors, shipping failures
1
Product category
4
18
Product
Reports disagree on sales by category
2
Currency
3
2
Treasury
Minor; mostly consistent
3
Customer segment
5
12
(none)
Marketing targeting inconsistent
4: needs owner first
07Tools and techniques
MDM platforms
Matching engines, survivorship, stewardship workflows and hubs.
RDM tools
Code list management, crosswalks, versioning, approval workflows and distribution APIs.
Data quality tools
Profiling, parsing, standardisation and address validation.
Data integration tools
Moving master data to and from the hub.
External reference sources
Standards bodies, company registers and data providers for enrichment.
Metadata repositories
Definitions, lineage and ownership of master and reference data.
08Implementation guidelines
Adhere to master data architecture and reuse it across projects.
Monitor data movement into and out of the hub.
Manage reference data change through a controlled, communicated process.
Agree data sharing agreements between providers and consumers.
Start small: one domain, a few critical attributes, a few systems. Prove value, then expand.
Reference data first: MDM matching and standardisation depend on clean code lists, as the worked example shows.
Making the business case
Illustrative benefits for Northwind
Benefit
Measure
Before
After
Fewer duplicate customers
Duplicate rate
11%
1.5%
Fewer invoice disputes
Disputes per month
140
85
Faster integration
Days to map a new system's codes
15
3
Consistent reporting
Category reports that reconcile
60%
100%
Lower risk
Credit exposure visible per legal entity
No
Yes
09Governance and metrics
Area
Example metrics
Data quality and compliance
Data quality dashboards for master data; conformance to standards; percentage of records with a master ID
Data change activity
Volume of changes; reference data change requests and cycle time
Data ingestion and consumption
Number of systems contributing to and consuming from the hub
Service level agreements
Time to publish a new reference value; time to resolve a match exception
Data steward coverage
Domains and code lists with a named owner and steward
Total cost of ownership
Cost of the hub against savings in integration and error handling
Data sharing volume and usage
API calls and file distributions
10Context summary
Inputs
Business drivers, data requirements, policy and regulations, standards, integration needs, data quality metrics, metadata.
Deliverables
Master and reference data requirements, data models and documentation, reliable reference and master data, golden record data lineage, data quality metrics and reports, data cleansing services.
Participants
Data stewards, subject matter experts, data architects, data analysts, application architects, data governance council, data providers.
Consumers
Application users, BI and reporting users, application developers and architects, data integration developers and architects, BI developers and architects, vendors, customers and partners.
11Why this chapter matters for an RDM and DQ consultancy
Reference data is the cheapest, highest-leverage fix. It's small, but every coded field and every integration depends on it.
MDM depends on RDM. Standardisation and matching quality improve directly when crosswalks and code lists are governed first.
Data quality measures symptoms; RDM fixes a root cause. Validity and consistency checks need an agreed reference to test against.
The inventory is a quick win: counting copies and differences of each code list turns an abstract problem into numbers a board understands.
This is the question 5 theme from the Mike Ferguson workshop: why reference data and data quality rarely reach the roadmap, and how to get them prioritised.
Chapter 11 · Knowledge area
Data Warehousing and Business Intelligence
Integrating data from many sources into a trusted analytical store, and delivering it through reports, dashboards and analysis that support decisions.
01Introduction
Definition
Data warehousing and business intelligence covers planning, implementation and control processes to provide decision support data and to support knowledge workers engaged in reporting, query and analysis.
A data warehouse is an integrated, historical store of data built for analysis. Business intelligence (BI) is both the analysis that turns data into insight, and the set of technologies that support it.
Business drivers
Operational support: regulatory and operational reporting from integrated data.
Business value and decision-making: a single, trusted view of performance over time.
Analytics: a foundation for data science and advanced analysis.
Goals and principles
Support business intelligence activity
Enable effective business analysis and decision-making
Find ways to innovate based on insights from data
Principles include: focus on business goals; start with the end in mind (business priority and scope drive the build); think and design globally, act and build locally; summarise and optimise last, not first; promote transparency and self-service; build metadata with the warehouse; collaborate; and recognise that one size does not fit all.
02Essential concepts
Inmon and Kimball
Two influential approaches shaped data warehousing.
Inmon: Corporate Information FactoryKimball: dimensional bus architecture
Aspect
Bill Inmon (Corporate Information Factory)
Ralph Kimball (DW Chassis / Bus)
Definition of the warehouse
A subject-oriented, integrated, time-variant and non-volatile collection of data supporting management decisions
A copy of transaction data structured for query and analysis
Core store
Normalised (3NF) enterprise data warehouse
Dimensional data marts, integrated through conformed dimensions
Build order
Top-down: enterprise model first
Bottom-up: one business process at a time
Data marts
Derived from the warehouse
Together, they are the warehouse
Strength
Strong integration and flexibility for new uses
Faster delivery; easy for users to query
Risk
Long time before business value appears
Silos if dimensions aren't truly conformed
Many modern warehouses blend the two: a normalised or Data Vault integration layer feeding dimensional marts. See Chapter 5 for star schemas, SCDs and Data Vault.
Warehouse architecture components
Source systems→Data integration→Staging area→Central warehouse + reference and master data→Data marts and cubes→BI tools
Component
Purpose
Source systems
Operational applications and external data
Data integration
Extract, transform and load processes (see Chapter 8)
Staging area
Intermediate store between sources and warehouse; usually not open to users
Reference and master data conformed dimensions
Shared dimensions such as Customer, Product and Date, used consistently by all marts
Central warehouse
Integrated, historical store
Operational data store (ODS)
Integrated, current-valued data for operational reporting, with little history
Data marts
Subsets focused on a subject or department
Cubes
Pre-aggregated multidimensional structures for fast analysis
Operational data store versus data warehouse
Aspect
ODS
Data warehouse
Time
Current values
History over years
Updates
Updated in place
Mostly insert-only; non-volatile
Latency
Near real time
Often daily
Use
"What is Northwind's current order status?"
"How have Northwind's orders trended over three years?"
The enterprise bus matrix
Kimball's bus matrix plans the warehouse: business processes as rows, conformed dimensions as columns. A tick means the process's fact table uses that dimension.
Business process
Date
Customer
Product
Warehouse
Promotion
Carrier
Orders
✓
✓
✓
✓
Shipments
✓
✓
✓
✓
✓
Invoicing
✓
✓
✓
✓
Returns
✓
✓
✓
✓
✓
Inventory snapshot
✓
✓
✓
Columns used by many rows are where conforming pays off most. Date, Customer and Product appear in almost every process, so they must be built once and shared. They are master and reference data.
Types of load processing
Type
Description
Historical data
The initial load of past data, often the hardest because old data is the messiest
Batch change data capture
Periodic loads of changes, using timestamps, logs or comparisons
Near real time
Small, frequent loads, such as micro-batches or trickle feeds
Real time
Continuous streaming into the warehouse
OLAP and multidimensional analysis
Online analytical processing (OLAP) lets users explore measures across dimensions. Implementations include ROLAP (relational tables), MOLAP (specialised cube storage) and HOLAP (a hybrid).
Worked example: the five OLAP operations
Base data: sales (£k) by region, quarter and category
Region
Quarter
Computers
Accessories
UKI
Q1
120
30
UKI
Q2
140
35
EU
Q1
90
20
EU
Q2
100
25
Roll-up: drop category
Region
Q1
Q2
UKI
150
175
EU
110
125
Total
260
300
Slice: Quarter = Q2
Region
Computers
Accessories
UKI
140
35
EU
100
25
Dice: UKI and Computers
Quarter
Sales
Q1
120
Q2
140
Drill-down: UKI Computers Q2 by month
Month
Sales
April
45
May
48
June
47
Q2
140
Pivot: quarters as rows, regions as columns
Quarter
UKI
EU
Q1
150
110
Q2
175
125
Every operation relies on the Region dimension being right. If CRM's "UK" and "Eire" weren't mapped through the country-to-region list, some UKI sales would fall into an "Unknown" region and the roll-up wouldn't reconcile.
Types of BI tool
Type
Use
Example
Operational reporting
Detailed, routine reports for running the business
Daily list of unshipped orders
Business performance management
Scorecards and KPIs against targets
Monthly sales against plan
Ad hoc query and self-service
Users build their own queries and visuals
An analyst exploring returns by product
OLAP and analytic applications
Multidimensional exploration
Drill from region to store to product
Descriptive, predictive and prescriptive analytics
What happened; what will happen; what should we do
Sales history; churn forecast; recommended offer
03Activities
Understand requirements: business goals, the questions people need answered, data sources, latency and history needs.
Define and maintain the DW/BI architecture: technical architecture, management processes and the bus matrix.
Develop the data warehouse and data marts: map sources to targets, remediate and transform data.
Populate the data warehouse: initial historical load, then ongoing loads.
Implement the BI portfolio: group users by need, match tools to needs.
Maintain data products: release management, development lifecycle, monitoring and tuning loads and BI activity.
Worked example: capturing a BI requirement
Element
Content
Business question
Which product categories are growing fastest in each region?
Measures
Sales amount, quantity, growth % quarter on quarter
Dimensions
Date (quarter, month), Product (category), Customer (region)
Grain
One row per order line
History needed
Three years
Latency
Daily is fine
Critical reference data
Product category taxonomy; country-to-region list
04Tools and techniques
Metadata repository
Business definitions, lineage and data models for the warehouse.
Data integration tools
For loading and transforming.
Business intelligence tools
Reporting, dashboards, OLAP and self-service analytics.
Prototypes
Show users early versions to confirm requirements.
Self-service BI
Empowers users, but needs governed, well-described data.
Audit data that can be queried
Load statistics and data lineage available to users.
05Governance and metrics
Category
Example metrics
Usage
Registered and active users; queries and reports run; content popularity
Subject area coverage
Percentage of business processes in the bus matrix that are delivered
Reconciliation to source totals; rows rejected; "Unknown" dimension members
Governance includes business acceptance, release management, certification of trusted reports, and data lineage so users can see where numbers came from.
06Context summary
Inputs
Business requirements, IT strategy, data from source systems, data models, metadata, data quality standards.
Deliverables
DW and BI architecture, data products, populated warehouse and marts, BI tools and reports, data lineage, release plans.
Participants
Business executives and managers, DM execs, BI programme manager, SMEs, data stewards, data architects, DBAs, data integration and BI specialists.
Consumers
Knowledge workers, managers and executives, external customers and systems, internal customers and systems, data professionals, other data scientists and analysts.
07Links to reference data and data quality
Conformed dimensions are master and reference data. The bus matrix shows which ones are shared most and should be governed first.
"Unknown" dimension members are a quality metric: every fact that can't be mapped to a valid code or customer is visible evidence of RDM and MDM gaps.
Hierarchies drive roll-ups. If the product taxonomy or region list is wrong, every aggregated report is wrong, even when the detail is right.
The warehouse is often where quality problems are first noticed, but the fix belongs upstream at source.
Chapter 12 · Knowledge area
Metadata Management
Data about data: the definitions, structures, lineage and operational facts that let people find, understand, trust and manage everything else.
01Introduction
Definition
Metadata management covers planning, implementation and control activities to enable access to high quality, integrated metadata, including definitions, models, data flows and other information critical to understanding data and the systems through which it is created, maintained and accessed.
A common analogy is a library catalogue: without it, the books are still there, but nobody can find the one they need or know whether it's the right edition. Metadata does the same job for data.
Business drivers
Increase confidence in data by providing context and enabling quality checks
Increase the value of strategic information by enabling multiple uses
Improve operational efficiency by finding redundant data and processes
Prevent the use of out-of-date or incorrect data
Reduce data-oriented research time
Improve communication between data consumers and IT professionals
Create accurate impact analysis, reducing the risk of project failure
Improve time-to-market by reducing system development lifecycle time
Reduce training costs and the impact of staff turnover through thorough documentation
Support regulatory compliance
Goals and principles
Document and manage organisational knowledge of data-related business terminology
Collect and integrate metadata from diverse sources
Provide a standard way to access metadata
Ensure metadata quality, consistency, currency and security
Principles include organisational commitment, a strategy, an enterprise perspective, socialisation, access, quality, auditing and continuous improvement.
02Essential concepts
Types of metadata
Type
Describes
Examples
Business metadata
The meaning of data, in business terms
Definitions, business rules, data owners, valid values, quality requirements, glossary terms
Technical metadata
Technical details and structures
Table and column names, data types, keys, indexes, ETL mappings, lineage, schemas
Operational metadata
Details of processing and access
Job run times, row counts, errors, schedules, access logs, data quality scores, backup and archive records
Worked example: one data element, three kinds of metadata
CustomerName
Type
Metadata item
Value
Business
Definition
The registered legal name of the customer organisation
Owner / steward
Sales Director / Asha
Business rule
Must match the company register for UK companies
Classification
Internal use only
Technical
Physical name
dw.DimCustomer.CustomerName
Data type
VARCHAR(200), NOT NULL
Source
crm.Account.acct_name, proper-cased
SCD type
Type 2
Operational
Last loaded
2026-06-15 02:14, 2,498 rows
Completeness
100%
Last changed
C1 renamed to Northwind Group Ltd, 2026-06-15
ISO/IEC 11179 metadata registries
ISO/IEC 11179 is the international standard for metadata registries. It describes how to define data elements precisely and consistently. A data element name is built from parts:
Part
Meaning
Example
Object class
The thing being described
Customer
Property
The characteristic of the object
Legal name
Representation term
The form of the value
Text
Resulting data element
Customer Legal Name Text
The same approach gives Order Date Date, Product Price Amount and Customer Country Code. Representation terms such as Code, Amount, Date and Text are the "class words" from Chapter 5's naming conventions.
Sources of metadata
Application metadata repositories
Built into applications.
Business glossary
Business terms and definitions.
BI tools
Report definitions and calculations.
Configuration management tools
Versions of system components.
Data dictionaries
Names and definitions of data elements.
Data integration tools
Mappings and lineage.
Database catalogues
Tables, columns and constraints.
Data quality tools
Rules and scores.
Modelling tools
Conceptual, logical and physical models.
Reference data repositories
Code lists, crosswalks and their versions.
Service registries
APIs and services.
Event messaging tools
Message formats and topics.
Metadata architecture types
Type
How it works
Strengths
Weaknesses
Centralised
One repository holds copies of metadata from all sources
High availability; consistent quality; easy to query
Copies can go stale; complex to maintain
Distributed
A query engine fetches metadata from sources in real time; nothing is stored centrally
Always current; no replication
Depends on sources being available; limited ability to add or enrich
Hybrid
A central repository for some metadata, with real-time links to sources for others
Balance of currency and enrichment
More design effort
Bi-directional
Metadata changes can be made in the repository and flow back to sources
One place to maintain
Complex synchronisation
Data lineage and impact analysis
Lineage shows where data came from and how it was transformed. Impact analysis uses the same information in the other direction: what will be affected if something changes.
Lineage of CustomerName, from CRM to dashboard
Worked example: impact analysis
The CRM team plans to rename acct_name to account_legal_name and lengthen it to 300 characters. Following the lineage forward shows what's affected:
Affected item
Type
Impact
Action
stg.Customer load job
ETL
Breaks: source column missing
Update the mapping
dw.DimCustomer.CustomerName
Column
Values over 200 characters would be truncated
Lengthen to 300
Sales dashboard
Report
No change if the warehouse column keeps its name
Regression test
Churn model
Model
No change
Regression test
Without lineage metadata, the first sign of this change would be a failed overnight load.
Metamodels
A metamodel is a data model of the metadata itself: which kinds of metadata exist and how they relate. For example: a Business Term is defined by a Definition, is owned by a Steward, and is implemented by one or more Columns; each Column belongs to a Table and may be governed by a Data Quality Rule and constrained by a Reference Data Set.
Data catalogues and active metadata Beyond DMBOK
Modern data catalogues combine glossary, technical metadata, lineage and usage in one searchable place. Active metadata goes further: metadata is used continuously to automate work, such as recommending datasets, flagging quality problems, applying access policies or triggering pipelines. This is the foundation of the "data fabric" idea that Mike Ferguson often discusses.
A catalogue that only lists assets, with no quality scores, owners or certification, can document chaos without fixing it. That was question 1's follow-up for the workshop.
03Activities
Define a metadata strategy: why metadata matters to the organisation, which metadata to manage first, and how it will be used.
Understand metadata requirements: business user requirements (terms, definitions, ownership) and technical requirements (volatility, synchronisation, history, access rights, structure, integration, maintenance, quality, security).
Create and maintain metadata: integrate metadata from sources, and distribute and deliver it.
Query, report and analyse metadata: impact analysis, lineage, usage and quality reporting.
04Tools and techniques
Metadata repository management tools
Collect, integrate and publish metadata.
Data catalogues
Search, glossary, lineage and collaboration.
Data lineage and impact analysis
Automated scanning of code and pipelines.
Metadata for big data ingest
Capture metadata as data lands in lakes.
05Governance and metrics
Metric
Example
Metadata repository completeness
Percentage of critical data elements with definition, owner and lineage
Metadata management maturity
Assessed with a maturity model (see Chapter 15)
Steward representation
Business units with active stewards
Metadata usage
Repository logins, searches and page views
Business glossary activity
Terms added, approved and used
Master data service data compliance
Reuse of data through services
Metadata documentation quality
Accuracy checked against source systems
Metadata repository availability
Uptime and processing time
06Context summary
Inputs
Business requirements, metadata issues, data architecture, business metadata, technical metadata, process metadata, operational metadata, data stewardship metadata.
Deliverables
Metadata strategy, metadata standards, metadata architecture, metamodel, unified metadata, metadata stores, data lineage, impact analysis, dependency analysis, metadata control process.
Participants
Metadata specialists, data stewards, enterprise architects, data architects, data modellers, BI developers.
Consumers
Data stewards, data professionals, other IT professionals, knowledge workers, managers and executives, customers and collaborators, business users.
07Links to reference data and data quality
Reference data sets are metadata-rich assets. Their definitions, owners, versions and crosswalks belong in the catalogue alongside tables and terms.
Valid values are business metadata. Linking a column to its governed reference data set tells everyone what values are allowed.
Quality scores are operational metadata. Publishing them next to datasets is what turns a catalogue from an inventory into a trust signal.
Lineage shows where reference data is applied, such as the crosswalk in the diagram, so its importance is visible.
Chapter 13 · Knowledge area
Data Quality
Making data fit for purpose: defining what "good" means, measuring it with rules and dimensions, finding root causes, and improving continuously.
01Introduction
Definition
Data quality management is the planning, implementation and control of activities that apply quality management techniques to data, in order to assure it is fit for consumption and meets the needs of data consumers.
The term "data quality" means both the characteristics of high-quality data and the processes used to measure and improve it. Data is high quality when it meets the expectations and needs of its consumers, so quality is always relative to a purpose.
Business drivers
Increasing the value of organisational data and the opportunities to use it
Reducing risks and costs associated with poor quality data
Improving organisational efficiency and productivity
Protecting and enhancing the organisation's reputation
Costs of poor quality
Poor quality data shows up as scrap and rework, workarounds and hidden correction processes, organisational inefficiencies, lost revenue and opportunities, damaged reputation, regulatory fines and poor decisions.
Goals and principles
Develop a governed approach to make data fit for purpose based on consumers' requirements
Define standards, requirements and specifications for quality controls as part of the data lifecycle
Define and implement processes to measure, monitor and report on data quality levels
Identify and advocate for opportunities to improve the quality of data, through changes to processes and systems
Criticality
Focus on the data most critical to the business and its customers.
Lifecycle management
Manage quality across the whole data lifecycle.
Prevention
Prevent errors and the conditions that cause them, rather than just correcting records.
Root cause remediation
Fix the cause, not only the symptoms.
Governance
Governance supports quality with ownership and decision rights.
Standards-driven
Quality requirements are expressed as measurable standards.
Objective measurement and transparency
Measure consistently and share results openly.
Embedded in business processes
Business owners are accountable for the quality of data their processes produce.
Systematically enforced
System owners enforce quality requirements.
Connected to service levels
Quality reporting and issue management are part of SLAs.
02Essential concepts
Critical data
Not all data needs the same attention. Critical data elements (CDEs) are those used in regulatory reporting, financial reporting, business policy, ongoing operations or business strategy, especially when errors would cause significant harm. Focus measurement and improvement on them first.
Data quality dimensions
A dimension is a measurable feature of data. The DMBOK reviews several frameworks:
Framework
Structure
Strong and Wang (1996)
Four categories, with fifteen dimensions in total: intrinsic (accuracy, objectivity, believability, reputation), contextual (value-added, relevancy, timeliness, completeness, appropriate amount), representational (interpretability, ease of understanding, consistency, concise representation) and accessibility (accessibility, access security)
Thomas Redman
Dimensions grouped by the data model, data values and data representation
Larry English
Inherent and pragmatic quality characteristics
DAMA UK (2013)
Six core dimensions, below
Core dimension
Question it answers
Typical measure
Completeness
Is all the required data present?
% of required values that are populated
Uniqueness
Is each entity recorded only once?
% of records that are not duplicates
Timeliness
Is the data available when it's needed, and current enough?
% delivered within the agreed time
Validity
Does the data conform to the required syntax, format, type and range?
% of values that pass validity rules
Accuracy
Does the data correctly describe the real-world object or event?
% of values matching an authoritative source
Consistency
Is the data free of contradiction, within and across datasets?
% of records passing consistency rules
The DMBOK also notes other useful dimensions, such as usability, timing issues, flexibility, confidence and value.
Worked example: measuring the six dimensions
Ten customer records from CRM, on 12 October 2026. Highlighted cells break a rule.
ID
Name
Country
Postcode
Email
Created
C1
Northwind Ltd
GB
M1 4BT
accounts@northwind.example
2026-03-01
C2
Contoso SA
FR
75002
(null)
2026-03-04
C3
Fabrikam GmbH
DE
10115
info@fabrikam.example
2026-03-05
C4
Northwind Limited
UK
M1 4BT
accounts@northwind.example
2026-03-06
C5
Tailspin Toys
GB
LS1 1AA
sales@tailspin.example
2026-03-07
C6
Litware Inc
US
98052
hello@litware
2026-03-08
C7
Adatum Corp
XX
00000
ops@adatum.example
2026-03-09
C8
Proseware Ltd
GB
(null)
team@proseware.example
2026-03-10
C9
Wingtip Ltd
IE
D02 X285
(null)
2026-03-11
C10
Coho Winery
FR
33000
contact@coho.example
2027-01-15
Scores
Dimension
Rule
Calculation
Score
Target
Status
Completeness
Email is populated
8 of 10 populated
80%
95%
Red
Completeness
Postcode is populated
9 of 10 populated
90%
98%
Red
Validity
Country is in the ISO 3166-1 reference list
8 of 10 valid (UK, XX fail)
80%
100%
Red
Validity
Email matches the email format
7 of 8 populated values valid
87.5%
99%
Red
Validity
Created date is not in the future
9 of 10 valid
90%
100%
Red
Uniqueness
No two records for the same company
9 distinct companies in 10 records (C1 = C4)
90%
99%
Red
Consistency
Postcode format matches the country's pattern
8 of 9 populated values consistent (C7 fails)
88.9%
98%
Red
Accuracy
Name matches the company register (sample of 5)
4 of 5 match (C4 doesn't)
80%
95%
Red
Timeliness
New records reach the warehouse within 24 hours
9 of 10 on time
90%
95%
Amber
Notice how many problems trace back to reference data: the country validity failures, the postcode consistency failure (no valid country to check against), and the duplicate, where "UK" and "GB" made C1 and C4 look like different customers.
Types of data quality business rules
Rule type
Checks that
Example from the dataset
Definitional conformance
Data is understood and used as defined
"Created" means the date the account was opened, not the date of the first order
Value presence and record completeness
Required values are present
Email must be populated for active customers
Format rule conformance
Values follow the required pattern
Email has text, @, domain and a dot
Value domain membership
Values come from an allowed list
Country is in the ISO reference list
Range conformance
Values fall within limits
Created date between 2000-01-01 and today
Mapping conformance
Values map correctly to an equivalent domain
CRM "UK" maps to ISO "GB" through the crosswalk
Consistency rules
Related values agree
Postcode format matches country
Accuracy verification
Values match an authoritative source
Name matches the company register
Uniqueness verification
Entities appear only once
One record per company registration number
Timeliness validation
Data is available on time
New customers in the warehouse within 24 hours
Worked example: rules as SQL checks
-- Value domain membership: countries not in the reference list
SELECT c.CustomerID, c.Country
FROM crm.Customer c
LEFT JOIN ref.Country r ON r.Code = c.Country
WHERE r.Code IS NULL; -- returns C4 (UK), C7 (XX)-- Completeness: share of customers with an email
SELECT 100.0 * COUNT(Email) / COUNT(*) AS email_completeness
FROM crm.Customer; -- 80.0-- Range conformance: created dates in the future
SELECT CustomerID FROM crm.Customer
WHERE CreatedDate > CURRENT_DATE; -- returns C10-- Uniqueness: possible duplicates on standardised name and postcode
SELECT std_name(Name) AS n, Postcode, COUNT(*)
FROM crm.Customer
GROUP BY std_name(Name), Postcode
HAVING COUNT(*) > 1; -- NORTHWIND LTD, M1 4BT, 2
Data profiling
Profiling analyses the actual content of data to discover its structure, patterns and problems. It's usually the first step of any quality assessment.
Profiling results for the ten records
Column
Nulls
Distinct
Patterns found
Findings
Country
0
7
AA (100%)
Values GB ×3, FR ×2, DE, UK, US, XX, IE. UK and XX aren't valid ISO codes.
Postcode
1
8
99999 ×5; A9 9AA ×2; AA9 9AA ×1; A99 A999 ×1
00000 is a placeholder value
Email
2
7
x@x.x ×7; x@x ×1
One value lacks a domain suffix; one value appears twice
Created
0
10
YYYY-MM-DD
Range 2026-03-01 to 2027-01-15; the maximum is in the future
Common causes of data quality issues
Cause
Examples
Lack of leadership
No ownership, no governance, no awareness of the impact of poor data
Data entry processes
Free-text fields, confusing screens, no validation, incentives for speed over accuracy, training gaps
Data processing functions
Wrong assumptions about sources, stale business rules, changes to data structures
System design
Missing constraints, poor keys, defaults that become "data", inconsistent reference data
Fixing issues
Manual patches that fix records but not causes, or create new errors
Worked example: root cause with the five whys
Why?
Answer
1. Why are some countries invalid?
Sales reps typed "UK" and "XX" into the CRM.
2. Why could they type anything?
The country field in the CRM is free text, not a pick list.
3. Why is it free text?
The CRM was configured before there was an approved country list.
4. Why was there no approved list?
Nobody owned country reference data.
5. Why did nobody own it?
There was no governance process for shared reference data.
Fix the cause, not the records: appoint an owner, publish the ISO list from a reference data service, and make the CRM field a pick list fed by it. Then correct the existing records once.
The data quality improvement lifecycle
The DMBOK bases data quality improvement on the Shewhart/Deming cycle: Plan, Do, Check, Act.
Plan, Do, Check, Act
Data quality and data processing
Process
What it does
Example
Data cleansing (scrubbing)
Transforms data to conform to standards and rules
Change "UK" to "GB"
Data enhancement (enrichment)
Adds attributes to improve quality and usability
Add company number from the register
Data parsing and formatting
Analyses data against pattern rules and splits it into parts
Split an address into street, town and postcode
Data transformation and standardisation
Maps values into standard formats
Phone numbers to international format
ISO 8000
ISO 8000 is the international standard for data quality. It covers the exchange of master data, data quality management, and the requirements for data to be portable and verifiable.
03Activities
Define high quality data: with stakeholders, agree what "fit for purpose" means for each use.
Define a data quality strategy: priorities, scope, roles and approach, aligned with business strategy.
Identify critical data and business rules.
Perform an initial data quality assessment: profile the data and measure it against the rules.
Identify and prioritise potential improvements: based on impact and effort.
Define goals for data quality improvement.
Develop and deploy data quality operations: manage rules, measure and monitor, manage issues, set SLAs, and report.
Worked example: a data quality issue and SLA
Issue DQ-208
Field
Value
Detected
2026-10-12 by automated rule "Country in ISO list"
Description
2 of 10 new CRM customers have invalid country codes
Severity
High: invoices for these customers will fail tax validation
Owner
Asha (business steward, customer data)
SLA
Acknowledge in 1 working day; resolve in 5
Root cause
Free-text country field (see five whys)
Fix
Records corrected; CRM change request CR-77 raised for a pick list
Status
Records fixed 2026-10-13; root cause fix scheduled for release 4.2
Data quality service level agreements
A data quality SLA sets out the data elements covered, expected dimensions and thresholds, methods of measurement, and the escalation and remediation process when thresholds are breached, including who is notified and how quickly.
04Tools and techniques
Data profiling tools
Discover patterns, nulls, ranges and distributions.
Data querying tools
Ad hoc checks and investigations.
Modelling and ETL tools
Build quality rules into design and integration.
Data quality rule templates
Reusable rule definitions.
Metadata repositories
Store rules, scores and definitions.
Preventive actions
Validation at entry, pick lists, constraints, training.
Corrective actions
Automated, manually directed and manual correction.
Statistical process control
Control charts that spot when quality drifts outside normal limits.
Root cause analysis
Five whys, fishbone diagrams and Pareto analysis.
05Governance and metrics
Return on investment: cost of improvements against benefits.
Levels of quality: number and percentage of errors or requirement violations within a dataset or across datasets.
Data quality trends: quality improvement over time, against thresholds and targets.
Data issue management: counts of issues by dimension, severity and owner; time to resolve; issues fixed at root cause.
Conformance to service levels.
Data quality plan illustrations: as-is and roadmap.
06Context summary
Inputs
Data policy and standards, data quality expectations, business requirements, business rules, data requirements, business metadata, technical metadata, data sources and data stores, data lineage.
Deliverables
Data quality strategy and framework, data quality programme organisation, analyses from data profiling, recommendations based on root cause analysis, data quality management procedures, data quality reports, data quality governance reports, data quality SLAs, data quality policies and guidelines.
Participants
CDO, data quality analysts, data stewards, data owners, data analysts, database administrators, data professionals, data quality managers, IT operations, data integration architects, compliance team.
Consumers
Business data consumers, data stewards, data professionals, IT professionals, knowledge workers, data governance bodies, partner organisations, centres of excellence, compliance team, data consumers.
07Links to reference data management
Validity needs a reference. Value domain membership, mapping conformance and many consistency rules can't be checked without a governed code list.
Reference data errors multiply. In the worked example, one ungoverned country list caused validity, consistency and uniqueness failures at once.
The five whys usually end at ownership. Many data quality root causes are missing reference data governance.
DQ measures the symptoms; RDM fixes a root cause. That's the positioning line for a consultancy that offers both.
Chapter 14 · Beyond the wheel
Big Data and Data Science
Collecting and analysing large, varied and fast-moving data to find patterns, predict outcomes and drive decisions, and the data management needed to make that trustworthy.
01Introduction
Definitions
Big data refers to data that is too large, too fast or too varied for traditional tools to handle well. Data science merges data mining, statistical analysis and machine learning with data integration and modelling, to build predictive models that explore patterns in data.
Business drivers
The desire to find and act on business opportunities that may be discovered through datasets generated by a diversified range of processes. Big data can drive innovation, and data science can find answers to questions nobody thought to ask.
Principles
Organisations should carefully manage metadata related to big data sources to have an accurate inventory of files, their origins and their value. Big data doesn't remove the need for data management; it makes it harder and more important.
02Essential concepts
The Vs of big data
V
Meaning
Northwind example
Volume
Very large amounts of data
Billions of website click events a year
Velocity
The speed at which data is captured, generated or shared
Warehouse sensor readings every second
Variety (variability)
Many forms and sources of data
Orders, emails, call recordings, social media posts
Viscosity
How difficult the data is to use or integrate
Supplier files in inconsistent formats
Volatility
How often data changes, and so how long it's useful
Live stock levels are stale within minutes
Veracity
How trustworthy the data is
Social media sentiment from possible bot accounts
The data science process
Define the big data strategy and business needs: what questions should be answered, and what decisions improved?
Choose data sources: understand their origin, format, quality and permissions.
Acquire and ingest data sources.
Develop data hypotheses and methods: what patterns might exist, and how to test for them.
Integrate and align data for analysis: join sources on common keys and reference data.
Explore data using models: populate predictive models, train them, evaluate and validate, and create visualisations.
Deploy and monitor: put models into use and track their accuracy over time.
Big data architecture
Data lakes
A data lake is an environment where large amounts of data of various types can be ingested, stored, assessed and analysed, often in its raw form. Without management, it becomes a data swamp: full of data nobody can find, understand or trust.
Typical data lake zones Beyond DMBOK
Zone
Contents
Rules
Raw (landing)
Data exactly as received
Immutable; metadata captured on arrival
Standardised (cleansed)
Formats fixed; codes mapped to reference data; duplicates flagged
Quality rules applied; rejects logged
Curated
Integrated, business-ready datasets and data products
Owned, documented, certified
Sandbox
Experiments by data scientists
Temporary; not used for decisions without promotion
Services-based architecture (lambda)
The DMBOK describes a services-based architecture that combines a batch layer for complete, accurate data with a speed layer for recent data, merged in a serving layer. This is often called the lambda architecture.
Lambda architecture
Sources and ingestion
Big data sources include social media, machine and sensor data (the Internet of Things), web logs, mobile data, images, video, audio and documents. Ingestion may be batch or streaming. Capturing metadata on ingestion, such as source, date, format and owner, is essential.
Machine learning
Type
How it learns
Example
Supervised learning
From labelled examples with known outcomes
Predict which customers will churn, trained on past churners
Unsupervised learning
Finds structure in unlabelled data
Group customers into segments by behaviour
Reinforcement learning
Learns by trial and reward
Optimise warehouse picking routes
Worked example: evaluating a churn model
A model predicts which of 200 customers will leave in the next quarter. Afterwards, 40 actually left.
Confusion matrix
Predicted: churn
Predicted: stay
Total
Actually churned
30 (true positives)
10 (false negatives)
40
Actually stayed
20 (false positives)
140 (true negatives)
160
Total
50
150
200
Measure
Formula
Result
Meaning
Accuracy
(30 + 140) / 200
85%
Share of all predictions that were right
Precision
30 / (30 + 20)
60%
Of customers flagged, how many really churned
Recall
30 / (30 + 10)
75%
Of real churners, how many were caught
F1 score
2 × 0.60 × 0.75 / (0.60 + 0.75)
0.67
Balance of precision and recall
Accuracy looks good at 85%, but a model that predicted "stay" for everyone would score 80%. Precision and recall tell you more. Which matters more is a business choice: the cost of a wasted retention offer against the cost of a lost customer.
Data and text mining
Mining techniques include profiling, data reduction, association, clustering and self-organising maps. Text mining applies similar ideas to unstructured text, for example sentiment analysis of reviews.
Worked example: association rules
In 10 recent orders, 4 contained a laptop, 6 contained a mouse, and 3 contained both.
Measure
Formula
Result
Meaning
Support (Laptop and Mouse)
3 / 10
30%
How often the pair appears together
Confidence (Laptop → Mouse)
3 / 4
75%
Of orders with a laptop, the share that also had a mouse
Lift (Laptop → Mouse)
0.75 / (6 / 10)
1.25
Laptop buyers are 25% more likely than average to buy a mouse
Lift above 1 suggests a real association, so suggesting a mouse at laptop checkout is worth testing. With only 10 orders, the result is a hypothesis, not proof.
Worked example: clustering
Customers grouped by annual spend and order frequency
Customer
Annual spend (£k)
Orders per year
Cluster
Northwind
48
26
Key accounts
Fabrikam
52
22
Key accounts
Contoso
9
4
Occasional
Coho Winery
7
3
Occasional
Tailspin Toys
3
18
Frequent small orders
Proseware
4
21
Frequent small orders
Nobody defined the clusters in advance; the algorithm found them. A person still has to name them and decide what to do with each.
Analytics types
Predictive analytics
Uses models to forecast outcomes, such as demand or churn.
Prescriptive analytics
Recommends actions and shows their likely effect.
Unstructured data analytics
Text, image and audio analysis.
Operational analytics
Analytics embedded in operations, such as real-time fraud scoring.
Data visualisation
Making patterns visible and understandable.
Data mashups
Combining data and services to create new insights or apps.
03Activities
Define big data strategy and business needs
Choose data sources
Acquire and ingest data sources
Develop data hypotheses and methods
Integrate and align data for analysis
Explore data using models
Deploy and monitor
Worked example: monitoring a deployed model
Month
Recall
Precision
Input data check
Action
July
75%
60%
Stable
–
August
73%
59%
Stable
–
September
61%
52%
New segment codes in CRM not in training data
Map new codes; retrain
Model drift is often a data management problem: here, a reference data change that nobody told the data science team about.
04Tools and techniques
MPP shared-nothing technologies
Massively parallel processing across many nodes.
Distributed file-based databases
Such as Hadoop-style storage and processing.
In-database algorithms
Run analysis where the data lives.
Big data cloud solutions
Elastic storage and compute.
Statistical computing and graphical languages
Such as R and Python.
Data visualisation tools
Dashboards and exploratory tools.
Analytic modelling
Building, testing and versioning models.
05Governance and metrics
Technical usage metrics: storage, processing and query volumes.
Loading and scanning metrics: ingestion rates and errors.
Learnings and stories: insights delivered and actions taken as a result.
Model performance: accuracy, precision, recall and drift over time.
Governance covers visualisation channels management, data science and visualisation standards, data security, metadata, and data quality, including the ethics of how models affect people (see Chapter 2).
06Context summary
Inputs
Business strategy and goals, build or buy decision tree, IT standards, data sources.
Deliverables
Big data strategy and standards, data sourcing plan, acquired data sources, initial data analysis and hypotheses, data insights and findings, enhancement plan.
Participants
Big data platform architects, ingestion architects, data SMEs, data scientists, analytic design leads, DM managers, metadata specialists.
Consumers
Business partners, business executives, IT executives.
07Links to reference data and data quality
Veracity is data quality under another name. Big data makes quality harder to check, so profiling and rules matter more, not less.
Joining varied sources needs shared keys and codes. The standardised zone of a lake is where reference data mapping happens.
Models encode reference data. When a code list changes, models trained on the old values drift, as the monitoring example shows.
This links to the workshop's AI agents question: agents and models need data products with explicit definitions, valid values and quality signals they can check.
Chapter 15 · Beyond the wheel
Data Management Maturity Assessment
Measuring how capable an organisation is at managing data, comparing that with where it needs to be, and turning the gap into a prioritised improvement plan.
01Introduction
Definition
A data management maturity assessment (DMMA) is a way to evaluate an organisation's data management capabilities against a defined scale, to understand its current state and plan improvements.
Maturity models come from the Capability Maturity Model (CMM), developed at the Software Engineering Institute at Carnegie Mellon University for software development. The idea is that organisations improve in stages, and each stage builds on the one before.
Business drivers
Regulation
Regulators expect evidence of data management capability.
Data governance
Assessments provide a baseline for governance programmes.
Organisational readiness for process improvement
They show where the organisation can realistically improve next.
Organisational change
Mergers, new systems or strategies are moments to reassess.
New technology
Show whether the organisation can adopt it.
Data management issues
Find the causes behind recurring problems.
Goals
Educate stakeholders about data management concepts, principles and practices
Clarify stakeholder roles and responsibilities
Highlight requirements for managing data as a critical asset
Broaden recognition of data management activities
Contribute to effective collaboration
Comprehensively evaluate the current state and identify gaps
02Essential concepts
Assessment levels
Level
Name
Characteristics
Reference data example
0
No capability
No organised practices or formal processes
Every system has its own code lists; nobody knows how many
1
Initial or ad hoc
Success depends on individuals; little governance; processes vary
One analyst maintains a country spreadsheet some teams use
2
Repeatable
Some consistent practices and tools emerge; roles start to be defined
A shared country list exists; a few projects use it
3
Defined
Standards and processes are agreed and used across the organisation
Named owners for key code lists; documented change process
4
Managed
Performance is measured and controlled with metrics
Reference data service with SLAs; conformance measured in every system
Changes automated end to end; issues prevented, not fixed
Assessment criteria
Each knowledge area is assessed against criteria such as:
Activity
To what degree is the activity or process in place?
Tools
To what degree is the activity automated and supported by tools?
Standards
To what degree is the activity supported by standards?
People and resources
To what degree is the organisation staffed to carry out the activity?
Existing frameworks
Framework
From
Focus
CMMI Data Management Maturity (DMM) Model
CMMI Institute
Data management strategy, governance, quality, operations, platform and architecture, supporting processes
Data Management Capability Assessment Model (DCAM)
EDM Council
Widely used in financial services; covers strategy, business case, programme, governance, architecture, technology, quality and operations
IBM Data Governance Council Maturity Model
IBM-led council
Governance outcomes, enablers, core disciplines and supporting disciplines
Stanford Data Governance Maturity Model
Stanford University
Foundational and project components of governance
Gartner Enterprise Information Management Maturity Model
Gartner
Vision, strategy, metrics, governance, roles, lifecycle and infrastructure
The DAMA-DMBOK knowledge areas can themselves be used as an assessment framework, which is what the worked example below does.
03Activities
1. Plan assessment activities
Establish scope and approach: whole enterprise, or selected areas or units.
Choose the framework.
Define the interaction approach: questionnaires, workshops, interviews, reviews of artefacts.
Plan communications: explain the purpose so people answer honestly. It's about the organisation, not judging individuals.
2. Perform the maturity assessment
Gather information
Perform the assessment: score each area against the criteria
Interpret the results: look for patterns and dependencies, not just low scores
Worked example: assessment questions for reference data
Criterion
Question
Evidence
Score
Activity
Is there a defined process for requesting and approving changes to shared code lists?
No; changes made directly in each system
1
Tools
Is reference data held in a central tool or service?
A spreadsheet on a shared drive
1
Standards
Are external standards (such as ISO 3166) adopted?
Used in ERP only
2
People
Do key code lists have named owners and stewards?
None formally named
1
Reference data area score (average, rounded down)
1
Worked example: results across knowledge areas
Northwind maturity: current and target
Knowledge area
Current
Target
Gap
Key finding
Data governance
2
3
1
Council exists but issues take months to resolve
Data architecture
2
3
1
No enterprise subject area model
Data modeling
3
3
0
Good standards in the warehouse team
Data security
3
4
1
Classification not applied to analytics data
Data integration
2
3
1
Many point-to-point interfaces
Reference and master data
1
3
2
No owners; six copies of the country list
Metadata
1
3
2
Catalogue bought but barely populated
Data quality
2
3
1
Quality checked only in the warehouse, not at source
The biggest gaps are reference and master data, and metadata. They are also dependencies for integration and data quality, so fixing them first lifts several other areas.
3. Develop recommendations
Recommendations should be specific, prioritised by value and dependency, and realistic: moving one level in 12 to 18 months is typical.
4. Generate a targeted improvement programme
Quarter
Initiative
Raises
Q1
Name owners for the top 10 code lists; agree the reference data policy
Ref and master data, governance
Q2
Publish country, currency and category lists from a reference data service
Ref and master data, integration
Q3
Populate the catalogue for critical data elements, with quality scores
Metadata, data quality
Q4
Move quality rules to source systems; reassess
Data quality
5. Re-assess maturity
Reassess periodically, often yearly, using the same framework and approach so results can be compared.
04Tools and techniques
Data management maturity framework
The chosen model and its scoring guidance.
Communication plan
Explains purpose, process and use of results.
Collaboration tools
Collect responses and evidence.
Knowledge management and metadata repositories
Store results and evidence over time.
Selecting a framework
Consider accessibility, comprehensiveness, extensibility, future progress and industry relevance.
Benchmarking
Compare with peers, using the same model.
05Governance and metrics
DMMA ratings: current and target levels by area, and progress between assessments.
Resource utilisation: effort and cost of the assessment and of improvements.
Risk exposure: how gaps translate into business risk.
Spend management: data management investment against plan.
Inputs to the assessment: participation and completeness of evidence.
Rate of change: speed of improvement in each area.
06Context summary
Inputs
Business strategy and goals, culture and risk tolerance, maturity frameworks and DAMA-DMBOK, policies, processes, standards, operating models, benchmarks.
Deliverables
Maturity ratings, gap analysis and improvement opportunities, current state and roadmap, organisation-specific communication plan.
Participants
CDO, CIO, data governance bodies, data stewards, business and technical stakeholders.
Consumers
Executives, data governance, data management staff, regulators.
07Links to reference data and data quality
Reference data is often the lowest-scoring area because it isn't seen as a discipline at all. An assessment makes that visible.
Dependencies matter more than scores. Low reference data and metadata maturity hold back integration and quality, so they belong early in the roadmap.
A focused RDM and DQ assessment is a practical consulting entry point: quick, evidence-based, and it produces a roadmap clients can act on.
Chapter 16 · Beyond the wheel
Data Management Organisation and Role Expectations
How to structure data management in an organisation: operating models, the data management organisation, the Chief Data Officer, and the roles that make data management work.
01Introduction
The data landscape changes quickly, and organisations need to change how they manage data with it. There is no single right structure; the best model depends on the organisation's culture, size, maturity and goals. This chapter covers how to understand the existing organisation, choose a model, and define roles.
Understand existing organisation and cultural norms
Before designing anything, understand how decisions are made today, how the organisation responds to change, who has influence, and what data management already happens informally. A design that fights the culture is likely to fail.
Questions to ask first
Area
Question
Value of data
Do leaders see data as an asset or a by-product?
Decision-making
Are decisions made centrally, or by business units?
Accountability
Who is blamed today when reports disagree?
Change history
How did previous enterprise initiatives go?
Existing roles
Who already acts as a steward, even without the title?
02Data management organisation constructs
Model
How it works
Advantages
Disadvantages
Decentralised
Data management responsibilities spread across business and IT functions, coordinated by a council
Flat structure; close to the business; quick to start
Inconsistent; hard to coordinate; many decision makers
Network
A decentralised model formalised with a documented set of connections and accountabilities (often a RACI)
Uses existing people; little hierarchy
Relies on goodwill; hard to sustain
Hybrid
A central data management centre of excellence works with business-unit groups
Central direction with local execution; supports enterprise standards
Needs careful definition of who does what; risk of tension
Federated
A variant of hybrid with additional layers of centralisation, for large and complex enterprises with multiple divisions
Enterprise direction plus division-level autonomy
Complex; more layers to coordinate
Centralised
A single data management organisation reports to a senior executive, such as a CDO
Clear authority and accountability; consistent standards
Can be seen as remote or bureaucratic; may slow local delivery
Choosing a model
Situation
Likely fit
SME, no CDO, a few data-minded people
Network: named owners plus a part-time executive sponsor
Centralised, or hybrid with strong central authority
Models usually evolve: many organisations start decentralised or networked, then move toward hybrid as maturity grows. This links to the workshop's SME question.
03Critical success factors
Executive sponsorship
A sponsor who understands and supports the programme.
Clear vision
Everyone understands what success looks like.
Proactive change management
Change is planned, not left to chance (see Chapter 17).
Leadership alignment
Leaders agree on the need and the approach.
Communication
Consistent, frequent and two-way.
Stakeholder engagement
Those affected are involved.
Orientation and training
People know what's expected and how.
Adoption measurement
Track whether new ways of working stick.
Adherence to guiding principles
Decisions follow agreed principles.
Evolution, not revolution
Incremental change that builds on what works.
04Building the data management organisation
Identify current data management participants: who already does this work, formally or informally.
Identify committee participants: who should sit on governance bodies.
Identify and analyse stakeholders: their interests, influence and concerns.
Involve the stakeholders: engage them in design and decisions.
Worked example: stakeholder analysis
Power and interest grid for a reference data programme
05Interactions with other data-oriented bodies
The Chief Data Officer
The CDO leads data management and governance across the enterprise and is accountable for the value of data as an asset. CDO responsibilities typically include:
Establishing the organisation's data strategy
Aligning data-centric requirements with available IT and business resources
Establishing data governance standards, policies and procedures
Providing advice, and possibly services, to the business for data-dependent initiatives such as analytics and BI
Communicating the importance of good data management to internal and external stakeholders
Overseeing the use of data in analytics and BI
Where the CDO reports matters: to the CEO or COO usually signals a business focus; to the CIO can make data seem an IT concern; to the CFO or chief risk officer often reflects regulatory drivers.
Other bodies
Body
Interaction with data management
Data governance
Sets policy and decision rights that data management follows
Data quality team
Often part of the data management organisation; works closely with stewards
Enterprise architecture
Aligns data architecture with business, application and technology architecture
Global organisations
Balance global standards with regional laws, languages and practices
Worked example: a RACI across the organisation
Activity
CDO
Governance office
Business data owner
Data steward
IT / platform
Approve data strategy
A
R
C
I
C
Define a reference data set
I
C
A
R
C
Run the reference data service
I
I
C
C
A / R
Set quality thresholds for customer data
I
C
A
R
I
Fix a source system defect causing bad data
I
I
C
C
A / R
06Data management roles
Organisational roles
IT and business organisational units with data responsibilities, such as a data management services group, a BI or analytics centre of excellence, an architecture group and the governance office.
Individual roles
Group
Roles
Focus
Executive
CDO, CIO, chief analytics officer, data governance executive sponsor
Strategy, sponsorship, accountability
Business
Data owners, executive and business data stewards, subject matter experts, business analysts
Definitions, rules, quality expectations, decisions about data
IT
Data architects, data modellers, DBAs, data integration architects and developers, BI and analytics developers, data security administrators, metadata specialists
Design, build and run data platforms and services
Hybrid
Data quality analysts, metadata specialists, technical data stewards, data scientists, data product owners
Bridge business needs and technical delivery
Worked example: data roles in an SME without a CDO
Role
Held by
Time
Responsibilities
Executive sponsor
Finance Director
Half a day a month
Chairs a quarterly data review; resolves disputes
Customer data owner
Head of Sales
Ad hoc
Approves definitions and quality targets
Reference data owner
Finance Operations Manager
Ad hoc
Owns country, currency and category lists
Data lead
Senior analyst
One day a week
Keeps the glossary, runs quality checks, coordinates fixes
Technical steward
IT systems administrator
As needed
Implements pick lists, validation and access
07Summary
Key idea
Start from the organisation's culture and existing practices, not an ideal chart.
Key idea
Choose the model that fits size, maturity and decision-making style, and expect it to evolve.
Key idea
The CDO's reporting line signals how the organisation sees data.
Key idea
Business ownership is essential; data management can't be delegated entirely to IT.
08Links to reference data and data quality
Reference data needs an owner even when there's no CDO. In smaller organisations, Finance Operations is often a natural owner of shared code lists.
Data quality accountability sits with the business that creates the data, supported by technical stewards who build the controls.
The workshop's "why doesn't the CDO step in" question often comes down to this chapter: a CDO without authority over business processes, or an organisation model that doesn't match the strategy.
Chapter 17 · Beyond the wheel
Data Management and Organisational Change Management
Data management changes how people work. This chapter covers why change is hard, the models that help (Bridges, Kotter, the change formula, diffusion of innovations), and how to plan the communication that makes change stick.
01Introduction
For most organisations, improving data management means changing behaviour: entering data differently, accepting new owners and rules, using shared sources instead of private spreadsheets. Technology alone doesn't do that. Organisational change management (OCM) is what turns a data strategy into new habits.
Laws of change
Organisations don't change, people change
Change happens one person at a time.
People don't resist change; they resist being changed
Involve people, and resistance drops.
Things are the way they are because they got that way
Understand the history before trying to change it.
Unless someone pushes, change likely won't happen
Change needs active leadership.
Change would be easy if it weren't for all the people
The technical part is usually the easier part.
02Managing a transition, not a change
William Bridges distinguishes change (the external event, such as a new system) from transition (the internal, psychological process people go through to adapt). Projects manage change; leaders must also manage transition.
Phase 1
Ending, losing, letting go
People let go of the old way and their old identity. Expect loss, anger and anxiety.
Phase 2
The neutral zone
The old way has gone but the new isn't working yet. Confusing, but also creative.
Phase 3
The new beginning
People take on the new identity and ways of working.
Worked example: a sales team losing its spreadsheet
Phase
What the team experiences
What leaders do
Ending
"Our spreadsheet worked fine; now we have to wait for the data team to add a country code."
Acknowledge the loss; explain why; keep what was good (for example, a fast request route)
Neutral zone
Pick lists feel slow; some codes are missing; frustration
Quick fixes; a named contact; collect feedback visibly
New beginning
Invoices stop bouncing; reports agree
Celebrate the results; share the numbers
03Kotter's eight errors and eight stages
John Kotter identified common errors that cause change to fail:
Error
In data management
Allowing too much complacency
"Our data is fine"
Failing to create a sufficiently powerful guiding coalition
A governance programme run by IT alone
Underestimating the power of vision
A strategy of tool purchases, not outcomes
Under-communicating the vision by a factor of 10, 100 or 1,000
One launch email, then silence
Permitting obstacles to block the vision
Managers rewarding speed of data entry over accuracy
Failing to create short-term wins
A two-year catalogue project with nothing to show
Declaring victory too soon
Disbanding the team after the first domain goes live
Neglecting to anchor changes firmly in the corporate culture
Stewardship not in job descriptions, so it fades
Worked example: Kotter's eight stages for a reference data programme
Establish a sense of urgency: show that 40% of invoice disputes trace back to inconsistent country and customer data.
Create the guiding coalition: CFO as sponsor, Sales Director, Head of IT and a respected steward.
Develop a vision and strategy: "One trusted list for every shared code, owned by the business."
Communicate the change vision: repeat it in team meetings, dashboards and newsletters, not once but constantly.
Empower broad-based action: remove obstacles, such as giving stewards time and changing CRM fields to pick lists.
Generate short-term wins: fix the country list in 90 days and publish the drop in invoice rejections.
Consolidate gains and produce more change: use the credibility to tackle product categories and customer duplicates.
Anchor new approaches in the culture: stewardship in job roles, governance checkpoints in every project.
This maps directly onto the workshop question about strategies that stall: most stall at stages 4 to 6, through weak communication, obstacles left in place, and no visible early win.
04The formula for change
Gleicher's formula, popularised by Beckhard and later Dannemiller, says change happens when:
D × V × F > R
Term
Meaning
D: Dissatisfaction
With the present situation
V: Vision
Of what's possible
F: First steps
Concrete, achievable steps toward the vision
R: Resistance
The cost of change, in effort, risk and loss
Worked example: scoring a proposal
Factor (scored 0–5)
Before
After work
What changed
Dissatisfaction
2
4
Shared the cost of invoice disputes with sales leaders
Vision
3
4
Showed a mock-up of one reliable customer view
First steps
1
4
Defined a 90-day country list pilot
D × V × F
6
64
Resistance (scored 0–125)
40
30
Kept a fast request route for new codes
Change likely?
No
Yes
Because the terms multiply, a zero in any one of them means no change, however strong the others. Vague first steps are a common reason good strategies don't move.
05Diffusion of innovations
Everett Rogers described how new ideas spread through a population. People adopt at different speeds, in five groups:
Adopter categories
Early adopters are often opinion leaders. Winning them over first, and making their success visible, helps the early majority follow.
Codes appear as pick lists in the tools people already use
Complexity
Is it easy to understand and use?
A one-page guide and a simple request form
Trialability
Can I try it first?
Pilot with one sales team
Observability
Can I see the results?
A dashboard of disputes falling
06Sustaining change
A sense of urgency that stays alive, through regular reporting of the business problem being solved.
The guiding vision, kept visible and linked to every decision.
A guiding coalition that stays engaged beyond launch.
Relative advantage and short-term wins, measured and celebrated.
A communication plan that keeps going after go-live.
Worked example: a communication plan
Audience
Message
Channel
Frequency
Owner
Executive team
Progress against business outcomes; decisions needed
Steering meeting
Monthly
CFO (sponsor)
Sales managers
What changes for their teams, and when; early results
Team briefings
Fortnightly during rollout
Sales Director
Sales reps
How to use the new pick lists; how to request a new code
Short video, intranet guide
At launch, then on change
Asha (steward)
IT and integration team
Technical changes, API details, timelines
Technical working group
Weekly
Ben
Whole company
Why shared data matters; stories of success
Newsletter
Quarterly
Governance office
Other models Beyond DMBOK
Prosci's ADKAR model describes the outcomes each individual needs: Awareness of the need for change, Desire to take part, Knowledge of how to change, Ability to do it, and Reinforcement to sustain it. It's a useful checklist for each stakeholder group in the communication plan.
07Links to reference data and data quality
Reference data programmes are change programmes. Taking away local code lists and spreadsheets takes something from people; Bridges' model explains the reaction.
Data quality improves when behaviour changes at the point of entry. That needs dissatisfaction, a vision and clear first steps, not just new validation rules.
Short-term wins are easier with reference data than with large MDM or catalogue programmes, which makes RDM a good first step for building momentum.
This ties back to the workshop question on strategy versus reality: when practice drifts back to old habits, it's usually a change management gap, not a strategy gap.