AIPM
Theme

AIPM package

Spark Optimization

@wshobson/spark-optimization@1.0.0

Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning.

Scanned · no issues · Oct 30, 2026

GitHub stars: ★ 42K

LinkedInPost

Install Skill

Pins this version in aipm.package.json and aipm-lock.json.

aipm add @wshobson/spark-optimization@1.0.0

Without project tracking

Writes the skill files only. Add --no-init to skip aipm.package.json and the lockfile.

aipm add @wshobson/spark-optimization@1.0.0 --no-init

How to use

Install the skill above, then run this in your AI tool:

/spark-optimization <your description>

About

Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.

Source

What gets added to your project

Preview every file this package writes on install before you run the command above.

Summary

Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.

Skill instructions

Apache Spark Optimization

Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.

When to Use This Skill

  • Optimizing slow Spark jobs
  • Tuning memory and executor configuration
  • Implementing efficient partitioning strategies
  • Debugging Spark performance issues
  • Scaling Spark pipelines for large datasets
  • Reducing shuffle and data skew

Core Concepts

1. Spark Execution Model

code
Driver Program
    ↓
Job (triggered by action)
    ↓
Stages (separated by shuffles)
    ↓
Tasks (one per partition)

2. Key Performance Factors

FactorImpactSolution
ShuffleNetwork I/O, disk I/OMinimize wide transformations
Data SkewUneven task durationSalting, broadcast joins
SerializationCPU overheadUse Kryo, columnar formats
MemoryGC pressure, spillsTune executor memory
PartitionsParallelismRight-size partitions

Quick Start

python
from pyspark.sql import SparkSession
from pyspark.sql import functions as F

# Create optimized Spark session
spark = (SparkSession.builder
    .appName("OptimizedJob")
    .config("spark.sql.adaptive.enabled", "true")
    .config("spark.sql.adaptive.coalescePartitions.enabled", "true")
    .config("spark.sql.adaptive.skewJoin.enabled", "true")
    .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
    .config("spark.sql.shuffle.partitions", "200")
    .getOrCreate())

# Read with optimized settings
df = (spark.read
    .format("parquet")
    .option("mergeSchema", "false")
    .load("s3://bucket/data/"))

# Efficient transformations
result = (df
    .filter(F.col("date") >= "2024-01-01")
    .select("id", "amount", "category")
    .groupBy("category")
    .agg(F.sum("amount").alias("total")))

result.write.mode("overwrite").parquet("s3://bucket/output/")

Detailed patterns and worked examples

Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.

Best Practices

Do's

  • Enable AQE - Adaptive query execution handles many issues
  • Use Parquet/Delta - Columnar formats with compression
  • Broadcast small tables - Avoid shuffle for small joins
  • Monitor Spark UI - Check for skew, spills, GC
  • Right-size partitions - 128MB - 256MB per partition

Don'ts

  • Don't collect large data - Keep data distributed
  • Don't use UDFs unnecessarily - Use built-in functions
  • Don't over-cache - Memory is limited
  • Don't ignore data skew - It dominates job time
  • Don't use .count() for existence - Use .take(1) or .isEmpty()

Loading package files…

Share this package

Add an Install with AIPM badge

Add this badge to a GitHub README so readers can open the package page and install the same version.

Install with AIPM
[![Install with AIPM](https://www.aipm-registry.com/install-with-aipm.svg)](https://www.aipm-registry.com/skills/wshobson/spark-optimization/1.0.0?utm_source=github&utm_medium=readme&utm_campaign=package_badge)

Publisher

Seth Hobson

This imported publisher has not claimed its AIPM account. A connected account confirms account control, but it does not guarantee that a package is safe.

AI assistant context

What this skill is for

@wshobson/spark-optimization@1.0.0 is an AIPM skill package for All tools. Use it when a project needs the reusable AI behavior described as: Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.

Install
aipm add @wshobson/spark-optimization@1.0.0
Use after install
/spark-optimization <your description>

Tool targets

Install command by target

Add --no-init to any of these when you want the skill files without aipm.package.json or aipm-lock.json.

cursor

Run this variant when you want to install the package into cursor.

aipm add @wshobson/spark-optimization@1.0.0 --target cursor --ci

claude

Run this variant when you want to install the package into claude.

aipm add @wshobson/spark-optimization@1.0.0 --target claude --ci

Discussion

Comments (0)

Loading comments…

Package FAQ

Install and compatibility

How to install
Run aipm add @wshobson/spark-optimization@1.0.0 in a project that has AIPM initialized.
Supported tools
@wshobson/spark-optimization@1.0.0 supports All tools.

Before installing

AIPM skills can add files to your project. Review the package name, target, description, license, and publisher before installing. Use a clean branch when trying a new skill.