Explainer

What is a data contract?

A data contract is an agreement between the producer and the consumers of data. What it contains, and how it differs from a catalog and documentation.

By Steffen Sande9 min read

Anyone who has worked with data for a while has seen it happen: a report that worked yesterday shows empty fields today. Nobody did anything wrong on purpose. A system further up the chain changed, and the change reached the report before it reached the people who depended on it.

A data contract is an attempt to make dependencies like these visible and binding. This article explains what a data contract is, what it should contain, and how it differs from tools and documents you may already have.

An agreement between producer and consumer

On one side is the producer: the team or system that creates and maintains the data, such as the team behind an order system. On the other side are the consumers, who use the data downstream: analysts building reports, a data platform team loading the data into a warehouse, or another system fetching it through an interface.

The agreement describes what the producer promises to deliver, and so what the consumers can build on. That covers not just which columns exist, but also what they mean, who owns them, what quality they should have, and how often they are updated.

The key word is agreement. A contract is not a description of how the data happens to look today, but a commitment to how it will look. If the producer changes something that breaks the contract, that change has to be announced and agreed, not left for the consumers to discover on their own.

A data contract is usually written in a machine-readable format, typically YAML, and stored alongside the code in a version control system such as Git. Changes to the contract can then be reviewed the same way as changes to code, and machines can check that the data actually meets it.

Why data contracts are needed

Picture an online shop with three core datasets: customer, order and product. The order system is owned by a development team. The finance department has a monthly revenue report that reads from order and joins it with product to show sales by product category.

One day the development team tidies up the order table. The column amount gets the more precise name amount_excl_vat, and product_id changes from an integer to a string because a new warehouse system uses alphanumeric item numbers. For the development team these are sensible improvements. Their application works just as before.

For the finance department it looks different. The report can no longer find the column amount, and the join with product returns no matches, because product_id now has a different type on each side. At best the report fails with a clear error. At worst it shows numbers that look right but are wrong, and someone makes decisions based on them.

The example shows two problems that keep coming back:

  • Changes upstream break things downstream. The producer does not know who uses the data, or how. The consumers do not hear about changes until something has broken.
  • Ownership is unclear. When the report shows the wrong numbers, it is not obvious who should fix it. Is it the development team that changed the table, the data platform team that loads it, or the analyst who wrote the query?

A data contract does not solve these problems by itself, but it gives them a fixed place. The contract states that amount is part of what the producer has promised to deliver. Renaming the column therefore becomes a contract change that has to go through an agreed process, with notice to the consumers and ideally a transition period in which both columns exist. And the contract states who owns the dataset, so it is clear who to contact when something is wrong.

What a data contract contains

The contents vary between organisations and standards, but most contracts cover the same five areas.

Schema

The schema describes the structure: which tables or objects the dataset consists of, which columns they have, and the data type of each column. It also states whether a column can be empty, whether it is a primary key, and how the tables relate to each other. For the order table in the example, the schema would establish that product_id is an integer and a foreign key to product.

The schema is the part of the contract that is easiest to check automatically, and the part most often broken by accident.

Meaning

A column name rarely says enough. Is amount with or without VAT? Is it the amount the customer paid, or the list price before discount? Is an order with the status delivered delivered to the customer, or just shipped from the warehouse?

The contract should therefore describe every table and every column, written so that someone outside the producer team understands it. Many organisations also link the columns to defined business terms, so that “revenue” means the same in the order data as in the finance system. This is where many misunderstandings between the business and technology arise, and where a contract does the most good beyond the purely technical.

Owner

The contract names who is responsible for the dataset: a team, a role or a named person, and how to reach them. Ownership means someone is responsible for the data meeting the contract, for changes being announced, and for questions being answered. Without an owner, a contract is just a description.

Quality rules

Quality rules describe what must be true about the data, beyond it having the right shape. Examples are that order_id must be unique, that amount_excl_vat must never be negative, that every order must point to a customer that exists, or that the share of empty email addresses in customer must stay below an agreed limit.

The rules should be concrete enough to run automatically against the data. Breaches can then be caught when the data is produced, rather than when someone reads a report.

Service levels

Service levels describe expectations for the delivery. The most common are:

  • Freshness: how old the data may be, for example that yesterday’s orders must be available by 6 am the next morning.
  • Availability: how much of the time the dataset must be readable.
  • Update frequency: whether the data is loaded continuously, every hour or once a day.
  • Retention: how long historical data is kept.

Many contracts also say where the data lives, who has access, and which version of the contract applies.

Here is what a simple contract for orders can look like in ODCS, the standard described further down:

apiVersion: v3.2.0
kind: DataContract
id: order
name: Order
version: 1.0.0
status: active
team:
  name: Order system
schema:
  - name: order
    properties:
      - name: order_id
        logicalType: string
        required: true
        description: Unique id of the order
        quality:
          - metric: duplicateValues
            mustBe: 0
      - name: customer_id
        logicalType: string
        required: true
        description: The customer who placed the order
      - name: amount_excl_vat
        logicalType: number
        required: true
        description: Amount excluding VAT

How a data contract differs from a data catalog

A data catalog is an overview of the datasets that exist in an organisation. It helps people find data, and often shows schemas, descriptions, owners and lineage, meaning where the data comes from and where it goes.

At first glance this overlaps a lot with a data contract. The difference lies in direction and commitment:

  • A catalog describes what is. It usually collects information from the systems and shows how the data looks right now. If someone changes a column, the catalog is updated.
  • A contract describes what is promised. It is written before or alongside the data and is a commitment from the producer. If someone changes a column in breach of the contract, that is a violation to be caught and dealt with.

The two complement each other. A catalog can show the contracts, so consumers see both what exists and what they can rely on. But a catalog on its own does not stop the order table from changing without notice.

How a data contract differs from documentation

Many teams already document their datasets, in a wiki, a spreadsheet or a README file. That is useful, but it is not a contract.

The first difference is that documentation is written for people, while a contract is also written for machines. Because the contract has a fixed structure, tools can read it, compare it with the actual data and stop a change that breaks it. A wiki page cannot stop anything.

The second difference is that documentation easily goes out of date. It is written once and updated when someone remembers. A contract that lives in version control alongside the code, and is checked automatically, goes out of date visibly: the mismatch produces an error, not just a silent gap between text and reality.

The third difference is commitment. Documentation explains, while a contract promises. The producer takes on responsibility for keeping to what the contract says, and the consumers get something concrete to build on.

Open standards for data contracts

You can write a data contract in a format of your own, but there are open standards that make contracts easier to share between teams and tools. One of the most widely used is the Open Data Contract Standard (ODCS), maintained by the Bitol project under Linux Foundation AI & Data. An ODCS contract is a single YAML file with fixed sections for schema, quality, service levels and team, among others. The current version is v3.2.0.

The advantage of a standard is that you do not have to invent the structure yourself, and that tools for validation, cataloguing and code generation can read the same files.

Where to start

A data contract does not have to cover everything from day one. A realistic starting point is to pick one dataset where a change recently caused problems, and write down the schema, a short description of each column and who owns it. Once that is in place, quality rules and service levels can be added as the producer and consumers agree on them.

What matters most is not the format, but the conversation the contract forces: what the producer actually promises, and what the consumers actually need.

Sources