PySpark Memory Tuning: How to Fix Out-of-Memory Errors and Data Skew

Quick Answer: PySpark memory tuning: A PySpark out-of-memory error is almost never solved by adding memory. It is solved by fixing one of four things: partition sizing, data skew, an unnecessary shuffle, or a Python function that holds more than it should. Work through them in that order — partition count, then skew, then join strategy, then code — and check the Spark UI between each change. Adding executor memory to a job with 2 million tiny partitions does nothing except make the failure more expensive.

What this guide covers

• How to read the four different out-of-memory failures — they have different causes and different fixes
• The unified memory model, and why “more memory” is often the wrong lever
• Partition sizing: shuffle.partitions, file sizes and the adaptive query execution settings that matter
• Data skew: what it looks like in the Spark UI, and three ways to fix it
• Broadcast joins: when they win, when they blow up
• Python UDFs and the memory traps that come with them
• A debugging order that avoids most wasted days
• What each fix does to your compute bill

---

The scenario: the job that worked at 40 GB and died at 400 GB

This is the most common PySpark ticket we get asked about, and the instinct is always the same — “we need a bigger executor”. Then it happens again at 2 TB.

Here is what usually happened. A batch pipeline processed 40 GB of daily events. Everything was fine. Over six months the dataset grew to 400 GB and the same job started failing with a container-killed error on one executor while the other 63 were healthy. Sixty-four executors, one of them dead, and a job that used to take 22 minutes now failing after 40.

The reason is almost never “the data got too big”. It is that the job’s shape did not change with the data. A partition size that was sensible at 40 GB became pathological at 400 GB. A join that was implicitly broadcast at 40 GB became a shuffle join at 400 GB. A Python function that was given 5,000 rows per partition was given 500,000.

That pattern — data grows, job shape stays fixed, failure follows — is the thing to recognise. Every fix below is a way of making the job’s shape track the data.

---

Step zero: read the error before you change anything

PySpark has several distinct failure modes that all get described as “out of memory”. The fix differs in each case, so the message matters.

What you see	What it actually means	Where to look next
Executor container killed by the cluster manager, or a task exceeding memory limits	The executor JVM heap was exhausted — usually one task trying to hold too much	Spark UI → the failed task → its input size; check partition size and skew
java.lang.OutOfMemoryError: Java heap space	JVM heap exhausted inside the executor	Same as above, plus check whether the driver is the one failing
Python worker killed, or a Python-side MemoryError	The Python process died, not the JVM	Check Python UDFs, UDF return size, and pyspark.memory settings
GC overhead exceeded, or long GC pauses before the failure	The JVM was thrashing: allocating and discarding continuously	Look for a shuffle that spills heavily, or a loop that grows a collection
Driver killed while the executors are idle	Driver memory exhausted — collecting a result to the driver	collect(), toPandas(), or a large to_json / count() on a huge frame

Two of these deserve a warning sign. First, driver out-of-memory is almost always a code bug, not a sizing bug: something is pulling data to the driver that should stay distributed. Second, GC overhead means you are not short of memory, you are thrashing with it — the fix is usually to stop shuffling so much data, not to buy a larger heap.

The Spark UI is the instrument panel: the Stages tab shows task duration distribution (one slow task among many equals skew), the SQL tab shows the physical plan (broadcast hash join versus sort-merge join), and the Storage tab shows what is cached and whether it spilled to disk. The Spark monitoring documentation covers the metrics exposed by the UI and the metrics system.

---

The memory model, briefly and correctly

Spark uses unified memory. The executor’s available memory is divided into two pools: storage memory (used for caching blocks and for shuffle output that stays in memory) and execution memory (used for sorting, hashing, joining and aggregation). Neither pool is hard-capped — they can borrow from each other — which is why a job can behave differently run to run.

The relevant knobs, with their shipped defaults:

spark.executor.memory            # JVM heap per executor
spark.executor.memoryOverhead    # extra for PyTorch, pandas, broadcast buffers, shuffle
spark.memory.fraction            # fraction of the heap that is unified memory (default 0.6)
spark.memory.storageFraction     # fraction of unified memory reserved for storage (default 0.5)
spark.memory.offHeap.enabled     # off-heap memory for execution (off by default)

The consequence worth remembering: if you raise spark.executor.memory without raising spark.executor.memoryOverhead, you can make Python-side failures more likely, because the overhead pool is what holds the Python workers and the broadcast blocks. And if your job is mostly Python UDFs, memoryOverhead is often the number to raise, not the heap.

The full list and the semantics of each are in the Spark configuration documentation.

---

Fix 1: partition sizing (do this first)

This is where most of the time is saved.

spark.sql.shuffle.partitions defaults to 200. That default is a blunt instrument: it is fine for a laptop and wrong for both a 2 GB dataset and a 2 TB dataset. After a shuffle, Spark writes output into this many partitions. Too few means each partition is too large and the executor runs out of memory on a single task. Too many means thousands of tiny tasks, enormous scheduling overhead, and slow jobs that never actually OOM but take all afternoon.

Two related settings matter as much:

• spark.sql.files.maxPartitionBytes — the maximum bytes packed into one input partition when reading files. Too large a value and a single task has to hold the whole split.
• spark.sql.adaptive.coalescePartitions.enabled — with adaptive query execution on, Spark coalesces the small output partitions that a shuffle creates back into fewer, right-sized partitions. This single setting often removes the “200 partitions for 12 MB of data” pathology on its own.

The practical rule: target a partition size in the 128–256 MB range for shuffle-heavy jobs, and let adaptive query execution coalesce rather than setting a number once and never revisiting it.

spark.conf.set("spark.sql.shuffle.partitions", "800")
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")
spark.conf.set("spark.sql.adaptive.advisoryPartitionSizeInBytes", "134217728")  # 128 MB
spark.conf.set("spark.sql.files.maxPartitionBytes", "134217728")

Note that spark.sql.shuffle.partitions is read when the shuffle is planned, so set it before the job runs rather than inside a job that has already started.

AQE also rewrites skewed joins at runtime. The Spark SQL performance tuning guide documents the adaptive query execution behaviour in detail, and the join-specific section is worth reading in full.

---

Fix 2: data skew

Skew is not the same as size. It is an uneven distribution across partitions, and it produces the classic signature: one task in a stage takes 40 times longer than the others.

Where it comes from, in order of frequency in our experience:

• A join key with a null or default value that dominates one side — the “null key” problem
• A customer, merchant or account identifier where one entity holds a disproportionate share of rows
• A timestamp or date key where one day holds far more data than the others

Three fixes, in the order we usually apply them.

1. Let adaptive query execution split the skewed partition. With AQE on, Spark detects a skewed join partition at runtime and splits it into smaller sub-partitions, then handles the matching side accordingly.

spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionFactor", 5)   # 5x the median
spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes", "256MB")

2. Broadcast the small side of the join. If the dimension table fits in memory, a broadcast hash join avoids the shuffle entirely — and the skew problem with it. The join performance section of the tuning guide covers the trade-offs.

3. Salt the hot key. When one key genuinely is hot and the data is too large to broadcast, add a random suffix to the key on the large side and explode the small side into matching copies. This is a manual intervention, and it changes the shape of the data, so it belongs in the curated layer rather than the raw layer.

from pyspark.sql import functions as F
from pyspark.sql.window import Window

# Add a salt to the large side, then fan the small side out to match
salted = large_df.withColumn("salt", (F.rand() * 16).cast("int"))
small_exploded = small_df.withColumn("salt", F.explode(F.array([F.lit(i) for i in range(16)])))
joined = salted.join(small_exploded, ["key", "salt"])

We only reach for salting after AQE and broadcast have been ruled out, because it is the fix with the highest maintenance cost — the next person who joins on key alone silently gets a partial result.

---

Fix 3: are you shuffling when you did not need to?

A shuffle is the most expensive thing Spark does and the main source of both memory pressure and cost. Three common accidental shuffles:

• groupBy followed by join — Spark shuffles both sides. Where possible, broadcast the aggregate result.
• distinct() on a large set — usually a shuffle. A cheaper approximation or an approximate distinct counter is often acceptable.
• cache() without a subsequent action — you have paid to cache and then paid again to recompute it, because the cache was never materialised.

# Expensive: shuffle both sides
result = fact.join(agg_by_key, "key")

# Cheaper: aggregate once, then broadcast the small side
agg = (fact.groupBy("key").agg(F.sum("amount").alias("total")))
result = fact.join(F.broadcast(agg), "key")

Also check the physical plan rather than guessing. In the Spark UI SQL tab, a SortMergeJoin over two Exchange nodes is a shuffle join; a BroadcastHashJoin is not. The join syntax reference is in the Spark SQL documentation.

---

Fix 4: Python UDFs, and where they hide memory

Python UDFs are the most common cause of “the executor died but the JVM was fine” errors. Three specific traps:

1. Return size. A Python UDF that returns a list, a dict or a string column is serialised back across the process boundary for every row. Returning a large structure turns into a driver-side problem.
2. In-process state. A UDF that accumulates into a global list grows for the lifetime of the Python worker, which is reused across tasks. The memory does not come back between tasks.
3. Library overhead. Importing a heavy library inside the UDF adds to the executor’s memory footprint and to every task’s startup cost.

# Fragile: global state persists across tasks in a reused worker
from pyspark.sql.types import *
import pandas as pd

accumulated = []

@udf(returnType=StringType())
def risky(x):
    accumulated.append(x)          # grows for the life of the worker
    return str(len(accumulated))

Prefer native Spark SQL functions, then pandas UDFs with a declared return type for genuinely vectorised work, and avoid Python UDFs entirely for logic that can be expressed in SQL. If you must use them, raise spark.executor.memoryOverhead rather than the heap, and declare the return schema so Spark does not have to infer it per row.

---

Fix 5: read less, cache deliberately

Two more levers that cost nothing but attention:

• Column pruning. Reading a wide Parquet file and selecting three columns reads only those columns’ bytes. The Spark performance tuning guide covers predicate and column pushdown for columnar formats.
• Cache with intent. cache() without a hint defaults to memory-only on some paths and can throw. persist(MEMORY_AND_DISK) spills rather than failing. And always unpersist() a cached table you no longer need — a cached 40 GB table on every executor of a 64-executor cluster is 2.5 TB of cluster memory you were not counting.

---

A worked OOM: 40 GB to 400 GB

Here is the timeline from the scenario at the top, reconstructed the way you would actually diagnose it.

09:12 — the nightly job fails. The executor logs show a container killed for exceeding its memory limit, on executor 7 of 64. Sixty-three executors are healthy. The job had been running 41 minutes.

09:14 — the first instinct is to raise the executor memory. The retry succeeds. The team closes the ticket.

09:15 the next day — the job fails again, this time on executor 22. A different executor, the same failure.

That variation is the clue, and it rules out a single oversized input and a single bad record. A per-executor memory limit is fixed. If one executor consistently failed, you would suspect a specific partition. If the failures move around, the work is distributed so unevenly that some executor eventually draws the hot partition.

09:40 — the Spark UI tells the rest of the story. In the Stage view, one task in the shuffle stage takes 38 minutes while the other 199 take 90 seconds each. In the SQL tab, the plan shows a sort-merge join on a key with a heavy null group. The table holds 60 million rows; 14 million of them have a null customer_id, and all 14 million land in one partition.

The fix was three lines, not more memory. AQE skew join handling was enabled, which splits the hot partition at runtime; the null key was excluded from the join explicitly and handled as a separate population; and the shuffle partition count was raised from the default 200 to 800 so the remaining data spread across more, smaller partitions.

09:52 — the job completes in 34 minutes on the same cluster, with no memory change at all. The following week, with 60% more data, it still completes — because the job’s shape now scales with the data.

The lesson worth keeping: an out-of-memory error that appears on a different executor each time is a distribution problem until proven otherwise. Memory limits are fixed per executor, so a genuinely oversized single task fails consistently on whichever executor draws it. Failures that move around point at skew.

---

A debugging order that works

This is the sequence we teach, because it prevents the expensive wrong turns:

1. Look at the Spark UI first. Stage view for the skew signature, SQL view for the plan, Storage view for spill.
2. Fix partition sizing. Set the shuffle partition count from the data volume, enable adaptive query execution, and set a sane file split size.
3. Kill the skew. AQE skew handling, then broadcast, then salt — in that order.
4. Remove accidental shuffles. Broadcast the small side of aggregations and dimension joins.
5. Audit the Python layer. Return types, global state, library overhead.
6. Only then consider more memory — and raise the right pool for the failure you actually had.

Each step should be a separate run with a look at the UI. Changing six things at once is how a team ends up not knowing which change fixed it — and not being able to explain it in an interview.

---

Cluster sizing: where to start

Sizing is a starting position, not a permanent answer, and it is worth doing deliberately rather than copying a blog post.

Workload shape	Executor count	Memory per executor	Notes
Small batch, light Python	2–4	4–8 GB	Rarely fixed by more memory; check the data first
Standard PySpark ETL	8–20	8–16 GB	The default shape; partition sizing matters more than core count
Heavy Python or pandas UDFs	4–12	16–32 GB plus overhead	Raise spark.executor.memoryOverhead alongside the heap
Very large single partition	As needed	Increase partition size or broadcast the small side	More memory does not fix one partition that is too big
Wide joins	Broadcast the small side	—	The join strategy is the lever, not the cluster size

Two rules that survive every environment: size for the largest partition, not the average, and if a job fails only when the data grows, the fix is in the plan, not in the cluster configuration.

---

What the fixes do to your bill

Memory problems are usually billed twice: once for the job, and once for the retries.

At the published rate of $0.44 per DPU-hour (AWS Glue pricing), a 6-DPU Glue Spark job that fails at 18 minutes and retries twice costs roughly $2.64 and delivers nothing. In a daily pipeline with a retry policy applied broadly, that is a real line on the bill — and it is invisible in the “successful run” metric that teams report on.

The bigger win is the job that used to take six hours and now takes fifty minutes, because the skew was handled and the shuffle was removed. Compute scales with data volume; job shape is yours to control. Our published benchmarking work on AWS data estates, including workload and cost distribution figures for Glue, EMR, Athena and Redshift, is in the State of AWS Data Engineering 2026 report.

---

Is PySpark still worth learning? Adoption, not hype

The question behind “should I learn PySpark” is usually “is this going to be on job descriptions next year”. Here is what is measurable, retrieved on 29 October 2026.

Signal	Measured value	Source
PyPI downloads, last month	47,831,780	pypistats — pyspark
PyPI downloads, last week	11,479,386	pypistats — pyspark
PyPI downloads, last day	1,793,922	pypistats — pyspark
GitHub stars	44,368	github.com/apache/spark
GitHub forks	29,546	github.com/apache/spark
Last commit	Today, 29 October 2026	github.com/apache/spark

Roughly 47.8 million downloads a month is a top-tier open-source project by any measure, and 29,546 forks against 44,368 stars tells you something specific: a large share of the people using Spark are shipping inside it rather than just reading about it.

What this means for a learner is narrow but useful. The market does not need more people who can write a groupBy. It needs people who can read a Spark UI, explain why a stage spilled, and know which knob to turn. That is why our Data Engineering Training spends a full module on performance, cost and security rather than on API coverage — the API is what documentation is for.

For search interest rather than adoption volume, the reproducible check is a Google Trends comparison in India you can run for your own region and window.

---

How to read these numbers

Download and repository counts come from PyPI’s public statistics endpoint and the GitHub API, retrieved on 29 October 2026. They are raw counts, not projections, and they move daily — read them as direction, not as a fixed fact. The billing figures are the published rates from the vendor pricing pages, not our own measurements; your discount and region will differ.

---

Key terms

Term	What it means in practice
Unified memory	The executor heap split into storage and execution pools that can borrow from each other.
Spill	When a pool cannot hold its data, it writes to disk. Heavy spill is a sizing or shuffle problem.
Partition	The unit of parallel work. One task handles one partition.
Data skew	An uneven distribution of rows across partitions. Shows up as one very slow task.
Adaptive query execution	Spark rewriting the plan at runtime: coalescing partitions and splitting skewed ones.
Broadcast join	Replicating the small side of a join to every executor. Avoids a shuffle entirely.
Shuffle	The redistribution of data across partitions by key. The main cost and memory driver.
Memory overhead	Executor memory reserved outside the JVM heap for Python workers, broadcast buffers and native libraries.
Predicate pushdown	Reading only the row groups that can match the filter. A columnar-format win.

---

Frequently asked questions

What is the fastest way to fix a PySpark out-of-memory error?

Read the error message, then look at the Spark UI stage view. If one task is far slower than its siblings, you have skew — fix that first. If all tasks are similarly heavy, your partitions are too large, and you raise the shuffle partition count or enable adaptive coalescing. If the error is Python-side, look at your UDFs. Adding memory without reading the UI is the most common waste of time in this problem.

How many shuffle partitions should I use?

There is no fixed number — it depends on the volume of data flowing through the shuffle and the executor size. Target roughly 128–256 MB per output partition, then let adaptive query execution coalesce small ones. The default of 200 is only sensible for small datasets, and it is the wrong number for most production estates.

What is the difference between an executor OOM and a driver OOM?

An executor OOM is one task failing to fit its data in the executor heap. A driver OOM is the driver process failing, usually because it collected results that should have stayed distributed. The driver fix is almost always code: stop calling collect() or toPandas() on large results.

Should I raise executor memory or memory overhead?

Match the failure. JVM-side “Java heap space” errors point at the heap. Python worker deaths, pandas UDF failures and broadcast issues point at spark.executor.memoryOverhead. Raising the heap when the problem is Python-side makes it worse, because the overhead pool is not scaled with it.

Does adaptive query execution replace manual tuning?

No — it removes the most common manual mistake, which is setting a fixed shuffle partition count that suits one dataset and breaks the next. It does not fix skew in workloads where the skewed partition genuinely exceeds memory, and it cannot rescue a Python UDF that accumulates state.

Is caching tables a good idea for performance?

Sometimes, deliberately. Caching pays off when a table is read repeatedly and the recomputation is expensive. It becomes a memory problem when the cached dataset is large, the cache is never unpersisted, or the storage is memory-only on executors that are already under pressure. Use MEMORY_AND_DISK and clean up.

How do I benchmark a fix properly?

Change one thing, run the same job on the same data, and record three numbers: wall-clock time, peak executor memory (from the executor metrics in the Spark UI), and the scan or shuffle volume in the SQL tab. Two of the three usually move before the third does, and that pattern tells you what you actually fixed.

---

Where to take this next

PySpark performance work is a skill that separates someone who can write a pipeline from someone who can make it survive production. That is the difference between a data engineer and a data analyst, and it is what hiring panels test when they ask what you did when a job started failing at scale.

The Data Engineering Training programme covers Spark architecture and partitioning, shuffle tuning, Delta Lake, medallion modelling and cost control across Snowflake and Databricks, with the capstone running a failure-mode simulation rather than a quiz. If you want the lakehouse platform specifically, the Databricks Training course includes reading the Spark UI, fixing a job that slowed after data growth, and the Delta and Unity Catalog specifics that the open-source distribution does not cover.

Salary bands for these roles across regions and experience levels are in the AWS Data Engineering Salary Report 2026.

---

References

• Apache Spark — SQL performance tuning and adaptive query execution
• Apache Spark — Join performance
• Apache Spark — Configuration reference
• Apache Spark — Monitoring and the Spark UI
• Apache Spark — Python API reference
• Apache Spark — Structured streaming
• Amazon Glue pricing
• SkilBrill — State of AWS Data Engineering 2026

---

---

---

---

---