Databricks Medallion Architecture: A Field Guide to Bronze, Silver and Gold

Quick Answer: Databricks medallion architecture: The medallion pattern splits your lakehouse into three layers with explicit contracts: bronze keeps data close to how it arrived, silver conforms and de-duplicates it once, gold serves modelled business data. The pattern’s value is not tidiness — it is that a defect in silver can be rebuilt from bronze without touching production consumers, and every consumer can be traced back to an exact input snapshot. It fails in three predictable places: uncontrolled small files in bronze, backfills that replay silver against changed bronze, and governance bolted on afterwards instead of designed in.

What this guide covers

• What each layer is actually for, and the contract between them
• The four failure modes we see in medallion implementations
• Delta Lake mechanics that make the pattern possible: the transaction log, ACID writes, time travel, schema evolution
• Unity Catalog governance: namespaces, grants, row filters, column masks, lineage
• Keeping it fast: OPTIMIZE, Z-ordering, liquid clustering, and the retention trade-off with VACUUM
• Where the pattern breaks and what to do instead
• The hiring signal, in India specifically

The scenario: eight terabytes, four copies, and no authoritative answer

Here is a real shape of problem, and it is on the “which copy is authoritative?” question.

A table starts at 200 GB. Finance’s dashboard points at it. Then the nightly job starts producing a second copy because a schema change broke the original transform, and a third copy appears when someone re-runs the pipeline manually after a bad upstream day. A year later the table is 8 TB, the daily job takes four hours, and there are four candidate copies of the same data. Nobody can say which one finance is reading, and the auditors want a reconciliation.

This is not a Databricks problem. It is an architecture problem that the medallion pattern exists to prevent. The pattern’s claim is simple: separate the data according to how much you trust it, and make each layer’s contract explicit. When you can rebuild silver from bronze, “which copy is authoritative” becomes a question with an answer.

The Databricks data engineering documentation and the Delta Lake documentation describe the primitives. This guide describes what breaks when teams use them without designing the contracts.

The three layers and the contracts between them

Bronze — the landing layer

Bronze holds data close to how it arrived: append-only, minimally transformed, with the source metadata attached. The job of bronze is to be complete and cheap, not clean.

What belongs in bronze:

• One table per source, one file per source batch, partitioned by ingestion date
• The raw payload plus ingestion metadata: source system, ingestion timestamp, file path, record count
• Schema evolution handled leniently — merge new fields rather than rejecting the batch

What does not belong in bronze: business logic, joins, deduplication, renames. The moment you transform in bronze, you have lost the ability to reprocess correctly when the logic changes.

-- Bronze: append-only landing table with ingestion metadata
CREATE TABLE bronze.orders_raw (
  order_id    STRING,
  customer_id STRING,
  amount      DECIMAL(12,2),
  status      STRING,
  _ingested_at TIMESTAMP,
  _source_file STRING
)
USING DELTA
PARTITIONED BY (_ingested_at);

The discipline that matters: bronze is append-only. No updates, no deletes, no merges. If you need to remove a bad batch, you record the exclusion in the silver layer, not by editing history.

Silver — the conformed layer

Silver is where the pipeline does its thinking. Data is typed, deduplicated, conformed to shared definitions, and checked. Silver is the layer you query when you need the truth.

Typical silver transformations:

• Deduplication by business key, keeping the latest version
• Conformed dimensions: one customer table, one product table, one date table, reused across sources
• Type casting and standardisation: currency codes, date formats, status vocabularies
• Slowly changing dimensions where history matters (SCD2 with effective dating)
• Data quality expectations that reject malformed rows rather than passing them downstream

from delta.tables import DeltaTable

# Silver: idempotent upsert keyed on business key, not ingestion time
silver_path = "silver.orders"

delta = DeltaTable.forPath(spark, silver_path)
(
    delta.alias("t")
    .merge(source=bronze_orders.alias("s"), condition="t.order_id = s.order_id")
    .whenMatchedUpdateAll(condition="s._ingested_at > t._ingested_at")
    .whenNotMatchedInsertAll()
    .execute()
)

The merge condition matters more than it looks. whenMatchedUpdateAll without a freshness predicate will let an out-of-order backfill overwrite newer data with older values — a bug that only appears when history is replayed, and one we have seen in review.

Gold — the serving layer

Gold is what consumers query: business marts, aggregates, modelled star schemas, feature tables. The rule for gold is that it should be cheap to query and expensive to build, because it is the layer with the most readers.

• Star schemas modelled for a specific subject — orders, revenue, inventory, customer
• Pre-aggregated tables for dashboard queries that would otherwise aggregate at query time
• Materialised views for logic that has to stay consistent with the source
• Narrow, purpose-built tables — a gold table per use case, not one table for all use cases

-- Gold: a modelled revenue mart for BI consumption
CREATE TABLE gold.revenue_daily
USING DELTA
AS
SELECT
  s.order_date,
  c.region,
  c.segment,
  SUM(o.amount)            AS revenue,
  COUNT(DISTINCT o.order_id) AS orders
FROM silver.orders o
JOIN silver.customers  c ON o.customer_id = c.customer_id
JOIN silver.dim_date    d ON o.order_date    = d.order_date
WHERE o.status = 'COMPLETED'
GROUP BY s.order_date, c.region, c.segment;

The contract, stated plainly

The value of three layers is the contract between them:

What makes this possible: the Delta Lake transaction log

The medallion pattern is not new — data warehouses have had staging, integration and presentation layers for decades. What changed is that the layers can now sit on object storage with transactional guarantees.

Delta Lake provides that through its transaction log. Every write produces an entry describing what changed, which is what gives you:

• ACID writes. No half-written tables for readers to trip over.
• Time travel. The Delta Lake documentation describes querying a historical snapshot of a table, which is how you answer “what did finance see on the 14th” without a separate audit copy.
• Schema enforcement and evolution. New columns are added without a migration; incompatible changes are blocked rather than silently corrupting data.
• Change data capture. You can read the changes as a stream, which is what makes incremental downstream jobs possible.

The distinction between a Delta table and a directory of Parquet files is the log. Any team that has had a job write a partial file and corrupt a table understands why that matters.

Failure mode 1: the small-file problem

This is the most common reason a medallion implementation underperforms, and it arrives quietly.

Bronze is written in small batches — a minute of CDC, one file per source partition, one file per micro-batch. Six months later the table has 400,000 small files. Every read now pays planning overhead across all of them, and the jobs that used to take four minutes take forty.

Delta Lake’s answer is OPTIMIZE, which compacts small files into larger ones, and Z-ordering, which rearranges data so that queries filtering on the ordering columns read fewer files. The mechanics are documented in the Delta Lake documentation.

-- Compact small files and lay out data for date-range filtering
OPTIMIZE silver.orders;

-- Multi-column Z-ordering on the columns most queries filter by
OPTIMIZE silver.orders
  ZORDER BY (order_date, region);

Two cautions. First, Z-ordering has a cost. It rewrites the affected files, so running it on every table on every schedule is itself expensive. Run it on a schedule, on the tables that need it. Second, Z-ordering only pays off on the columns you filter by — ordering by a column nobody filters on is a rewrite with no benefit.

For tables where the filter pattern changes over time, liquid clustering is the newer alternative: you specify clustering keys and a file-size target, and the table maintains its own layout as data arrives, so there is no periodic rewrite job to babysit. Which approach fits depends on how stable your access patterns are.

The retention trade-off

Time travel is only useful while the history is still there. VACUUM removes files no longer referenced by the transaction log, which is what stops storage growing forever — but it also deletes the history that time travel depends on. Set a retention window deliberately, and remember that recovery time after a bad run is a function of that window.

Failure mode 2: backfills that replay against changed bronze

The medallion pattern assumes silver can be rebuilt from bronze. That is true, and it is also the thing that breaks quietly.

If silver is built by merging bronze with no freshness guard, replaying an old bronze batch can overwrite newer silver rows with older values. The data looks fine; it is just wrong. And nobody notices until a quarterly report disagrees with the source system.

The fix is the freshness predicate in the merge condition, shown in the silver example above: only update when the incoming record is newer than the stored one. The second fix is to make bronze immutable — never edit a bronze file after it lands, because the moment you do, “rebuild silver from bronze” stops being a well-defined operation.

If your bronze layer does need corrections, record them as new bronze records rather than as edits to old ones.

Failure mode 3: governance designed after the fact

The Unity Catalog documentation describes the governance layer, and the temptation is to add it once the platform is running. That is backwards, because the namespace decisions are hard to change later.

The four decisions to make before the first table exists:

1. Namespace structure. Unity Catalog uses a three-level namespace — catalog, schema, table — and the catalog boundary is an organisational one. If every team gets its own catalog, cross-team joins need explicit grants and the lakehouse fragments. If everything sits in one catalog, permissions become unmanageable. The pragmatic middle is one catalog per environment, schemas per domain, tables per use case.
2. Who can read what. Grants should follow a policy, not a habit. The access model is a securable-object hierarchy with inheritance, so granting at the right level matters more than the number of grants.
3. Row filters and column masks. If finance may see revenue but not margin, and sales may see their own region only, that is row-level security and column masking — configured as policy on the table, not as a filter in every query. A filter in a query is a convention; a policy is a control.
4. Lineage. Delta Lake records table lineage through the log, and Unity Catalog surfaces it. This is what makes “which reports read this table” answerable, and it is the feature that turns a data platform from a file dump into something an audit can interrogate.

Unity Catalog has also moved outside Databricks as an open specification, with its own documentation site and an Iceberg-compatible REST interface. If you are choosing a governance layer today, that portability is worth evaluating rather than assuming.

Where the pattern breaks

Medallion is not universal, and knowing when to stop is part of the architecture skill.

• One source, one consumer, trivial transform. Three layers for a nightly copy of a single report table is overhead without benefit. A single well-named table with a documented contract is better.
• Streaming-first workloads. Medallion works with streaming, but the batch-flavored mental model (reprocess a date range) does not map neatly. Streams want change-driven processing rather than range replay.
• Environments where the transformation logic is not reproducible. If a step was manual — someone fixed a value in a spreadsheet and wrote it back — you cannot rebuild silver from bronze. That step has to become code before the pattern is safe.

The pattern is also not the same as a warehouse. A warehouse gives you a curated, centrally-governed model with tight query performance and a high price per query. A lakehouse gives you open file formats, lower cost, and more engineering responsibility. Our published comparisons — Snowflake vs Databricks and Azure Synapse vs Databricks — set out the workload-level trade-offs with reproducible benchmark scripts, and they are worth reading before you commit to a platform on brand recognition.

Keeping the cost down

Architecture and cost are the same conversation in a lakehouse.

• Job clusters, not always-on clusters. All-purpose clusters idle and cost. Job clusters start, run and terminate.
• Serverless compute removes the sizing decision entirely and bills for what the job used. The compute configuration documentation covers the options.
• Photon for SQL workloads. A faster engine for SQL and DataFrames is a straightforward win where the workload is SQL-heavy. The Photon documentation covers when it is worth the premium.
• Isolation for the important jobs. A nightly finance load that competes with an ad-hoc exploration run is a scheduling failure waiting to happen. Separate compute pools for production and exploration.
• Budget alerts and query tags. Cost attribution is a prerequisite for optimisation. You cannot tune what you cannot attribute.

We benchmarked the cost and performance envelope of these platforms across enterprise workloads — workload patterns, price-per-query and adoption rates — in the Enterprise Data Lake Benchmark and the Snowflake vs Databricks benchmark.

What hiring managers actually ask

Interview questions about lakehouse design rarely ask whether you know the layer names. They ask:

• “A table is 8 TB and the daily job takes four hours. Where do you look first?” — file sizes, then skew, then partition pruning, then the plan.
• “How do you rebuild a corrupted silver table without breaking gold?” — the answer is bronze immutability plus a freshness-guarded merge.
• “Who can see this column?” — the answer is Unity Catalog policy, not a WHERE clause.
• “Show me a Delta transaction log.” — this one separates people who have run the platform from people who have read about it.

If you are preparing for that conversation, the Databricks Training course runs over 12 weeks and 72 hours across eight modules: the lakehouse and workspace, Delta Lake in depth including time travel and Z-ordering, Unity Catalog with row filters and lineage, notebooks and SQL modelling, jobs and pipelines with retries and alerting, MLflow with vector search, applied skills including reading the Spark UI and fixing a job that slowed after data growth, and a capstone that assembles a governed pipeline with a runbook, a cost estimate and an access review.

The vendor-neutral version, covering Delta Lake, Spark, Airflow, Kafka and dbt across cloud providers, is the Data Engineering Training programme.

Pay bands for lakehouse and data engineering roles in India and internationally are in the AWS Data Engineering Salary Report 2026.

How to read these numbers

Where we cite our own benchmark work, it is published with the methodology and the reproducible scripts, and linked above. Adoption and demand figures quoted elsewhere on this site come from public sources with their provenance stated. Nothing in this article is a claim about Databricks’ internal performance beyond what the vendor documents publish.

Key terms

Frequently asked questions

What is the difference between the medallion pattern and a traditional ETL staging area?

Structurally they are similar — landing, integration, presentation. The difference is the contract and the substrate. The landing area is versioned and transactional, so the integration layer can be rebuilt from a known input state, and the history of any table is queryable. In a traditional ETL stack, reprocessing usually means re-reading source systems.

Is medallion architecture still relevant with Iceberg?

Yes, and the two are complementary. Medallion is a layering pattern; Iceberg is a table format with its own transaction model. A common production shape is medallion layers stored as Iceberg tables, with governance through an Iceberg-compatible catalog. What changes is the mechanics of compaction and layout — Iceberg handles file layout as a table property rather than through a separate optimisation command.

How do you prevent the small-file problem from the start?

Write fewer, larger files. Batch bronze writes rather than appending per micro-batch, set a target file size when writing, and run compaction on a schedule for the tables that need it. Partitioning by ingestion date gives you a natural compaction boundary: compact yesterday’s files once the day is complete.

What is the correct way to handle a backfill in the silver layer?

Make the merge idempotent and freshness-guarded, so replaying an old bronze batch cannot overwrite newer silver rows. Keep bronze immutable. Then the backfill becomes a rerun rather than a repair, which is the difference between a five-minute job and a week of reconciliation.

Do you need Unity Catalog if you already have a lakehouse?

If your lakehouse has more than a handful of tables and more than one team reading them, you need somewhere to record who may read what. Without a governance layer, access control ends up duplicated into every query — which is a convention, and conventions leak. The decision is which governance layer, not whether.

How many layers should a small team have?

Fewer than the diagram suggests. Many teams start with a landing table and a serving table and add a conformed layer when the same transformation is written twice. Add layers when you feel the pain that the layer solves, not because a diagram has three boxes.

Does medallion work with streaming pipelines?

Yes. Streaming ingestion into bronze, change-driven processing into silver, aggregates into gold. What does not translate is the batch mental model of “reprocess this date range” — in streaming, replay is defined by offsets and by Delta’s change data feed, not by a date window.

References

• Databricks — Data engineering documentation
• Databricks — Delta Lake documentation
• Databricks — Delta Lake batch operations
• Databricks — Unity Catalog governance
• Databricks — Compute configuration
• Databricks — Photon
• Databricks — SQL warehouse documentation
• Delta Lake — official site
• Apache Iceberg — documentation
• MLflow — documentation
• SkilBrill — Snowflake vs Databricks benchmark

Written by SkilBrill Training Institute

November 3, 2026 · 16 min read