# Stop Reprocessing Everything: Build Incremental Data Pipelines on AWS

*How to use AWS Glue, Amazon S3, and PySpark to process only new and changed data without making your pipeline fragile.*

Imagine you have a data pipeline that processes 100 GB of data every morning.

The first day, everything works fine. A few weeks later, the dataset grows to 500 GB. After a few months, it reaches several terabytes.

But here is the interesting part: only a small portion of that data changes every day.

Your pipeline still reads everything.

The job takes longer, consumes more compute resources, and spends most of its time processing records that have already been processed before.

I have always found this to be one of the most important questions in data engineering:

**If only 2% of the data has changed, why should we process 100% of it again?**

This is where incremental data processing becomes useful.

In this article, we will explore how to design an incremental pipeline using Amazon S3, AWS Glue, and PySpark, including the problems that appear when the pipeline starts running in production.

## 1\. Full Refresh vs. Incremental Processing

Let's start with a simple example.

Suppose an application exports customer records to Amazon S3 every day.

Your raw dataset contains 10 million records, but only 50,000 records are new or updated today.

With a full refresh, your pipeline might read and transform all 10 million records.

With incremental processing, it attempts to process only the records that have changed since the previous successful run.

| Approach | Records to process |
| --- | --- |
| Full refresh | 10,000,000 |
| Incremental processing | 50,000 |
| Potential reduction in records processed | 99.5% |

These figures are illustrative, not measured benchmark results. Actual cost and runtime savings depend on file sizes, partition layout, transformation complexity, and the amount of data that must be scanned.

Incremental processing does not automatically make every pipeline faster. If your source contains millions of tiny files, for example, discovering and listing those files can remain expensive even when the number of new records is small.

Still, avoiding unnecessary work is a good starting point.

## 2\. A Simple AWS Architecture

For this example, imagine that customer data arrives in Amazon S3 and AWS Glue transforms it into a curated dataset.

The flow looks like this:

```text
Source Application
        |
        v
Amazon S3 - Raw Layer
        |
        v
AWS Glue + PySpark
        |
        +---- Read New or Changed Data
        |
        +---- Validate and Transform
        |
        v
Amazon S3 - Curated Layer
        |
        v
Downstream Analytics
```

The important part is deciding how the job knows which records are new.

There are several ways to do this:

*   **AWS Glue job bookmarks:** Track processing progress for supported sources and transformations.
    
*   **Watermarks:** Store a previously processed timestamp or another ordered change marker.
    
*   **CDC (Change Data Capture):** Consume inserts, updates, and deletes from a change stream or database log.
    
*   **Manifest-based ingestion:** Track which source files have been successfully processed.
    

These mechanisms solve related but different problems. A file bookmark does not automatically provide record-level change detection inside every file, and a timestamp watermark does not automatically handle deletes.

For our example, we will use a timestamp watermark to explain the core idea.

## 3\. Build an Incremental Pipeline with PySpark

Assume the source provides customer records with the following columns:

```text
customer_id
customer_name
email
updated_at
```

The `updated_at` field represents when the source record was last changed.

For the first run, the pipeline may need to process the entire available dataset. Subsequent runs can use the saved watermark to identify a smaller set of changes.

### Step 1: Read the previous watermark

A watermark is a marker representing the last point in the source data that the pipeline has successfully processed.

For a real implementation, store this state durably in a system such as DynamoDB, a control table, or another suitable metadata store.

Here is a simplified example using a value supplied to the Glue job:

```python
from datetime import datetime, timezone

previous_watermark = "2026-10-15T00:00:00Z"

watermark = datetime.fromisoformat(
    previous_watermark.replace("Z", "+00:00")
)

print(f"Previous watermark: {watermark}")
```

For an initial load, you need an explicit bootstrap strategy rather than assuming a previous watermark already exists.

Also, don't use the current clock time as the new watermark merely because the job started. The checkpoint must reflect data that has actually been processed successfully.

### Step 2: Read and filter the source data

For this example, assume the source data is stored as Parquet files in S3 and the dataset fits the chosen scan strategy.

```python
from pyspark.sql import functions as F

source_path = "s3://example-bucket/raw/customers/"

source_df = (
    spark.read
    .format("parquet")
    .load(source_path)
)

incremental_df = source_df.filter(
    F.col("updated_at") > F.lit(previous_watermark).cast("timestamp")
)
```

This demonstrates the basic filtering logic, but it is important to understand one limitation.

**Filtering after reading Parquet files does not guarantee that Spark reads only the changed data from S3.**

Spark may still need to discover and scan many files before applying the filter. For efficient incremental ingestion, the source layout must support it—for example, by partitioning or organizing data appropriately, reading only newly arrived files, or using a CDC source.

For production systems, validate timestamp types and timezone behavior, too. A watermark comparison is only reliable when the source's change marker has consistent semantics.

### Step 3: Validate the incoming records

Incremental processing reduces the volume of work, but it does not remove the need for data quality checks.

For example, customer IDs should not be null.

```python
valid_df = incremental_df.filter(
    F.col("customer_id").isNotNull()
)

invalid_df = incremental_df.filter(
    F.col("customer_id").isNull()
)

invalid_count = invalid_df.count()

print(f"Invalid records: {invalid_count}")
```

In a production pipeline, invalid records should normally be written to a quarantine location with enough information to investigate and replay them later.

Be careful with `count()`: it can trigger an additional Spark action and cause more computation. If the same DataFrame is used for several actions, consider whether persistence or a single aggregation is appropriate.

Also decide whether invalid records should block the entire batch or be quarantined while valid records continue. That decision should follow the data contract and business requirements.

### Step 4: Write the processed data

For a simple append-only dataset, you can write the valid records to a curated S3 location:

```python
target_path = "s3://example-bucket/curated/customers/"

(
    valid_df.write
    .format("parquet")
    .mode("append")
    .save(target_path)
)
```

This is fine as an illustration of writing new records, but it is not sufficient for a general customer table.

What if an existing customer's email address changes?

Appending the updated record may create two versions of the same customer. If the target must contain one current record per customer, use a storage format and write strategy that supports the required upsert behavior, such as an appropriately configured transactional table with `MERGE` support.

The target design should match the business requirement:

*   Append-only events: append new events.
    
*   Current-state records: apply inserts and updates using a stable key.
    
*   Sources that support deletions: propagate delete events as well.
    

Never assume that an incremental source filter alone makes the target correct.

## 4\. The Hard Part: When Should You Update the Watermark?

This is where a seemingly simple pipeline can lose data.

Imagine the job reads new records, writes some of them to the target, and then fails before it finishes.

If the pipeline advances its watermark too early, the next run may skip records that were never successfully written.

If the pipeline retries the batch, it may write records that were already committed during the previous attempt.

These are two different failure modes: missing data and duplicate data.

A safer design follows this sequence:

1.  Determine the source range or batch to process.
    
2.  Read and validate that batch.
    
3.  Write the batch using an idempotent strategy.
    
4.  Verify that the write completed according to the pipeline's requirements.
    
5.  Advance the checkpoint only when the batch is safely committed.
    

However, the order alone does not guarantee exactly-once processing. A crash can occur after the target write succeeds but before the checkpoint is updated.

For that reason, the target write must also be safe to retry, or the system needs a coordinated commit mechanism.

For example, a deterministic batch ID, a deduplication key, and an idempotent merge can help make retries safe. The exact approach depends on the target storage format and the semantics of its write operations.

## 5\. What About Records That Arrive Late?

Consider a record that has a source timestamp of October 14 but does not arrive until October 16.

If your pipeline has already advanced its watermark to October 15, a strict timestamp filter could miss that record.

There are several ways to handle this:

**Option 1: Use a lookback window.**

Read a small period before the last watermark and deduplicate or merge the results into the target.

This is simple and useful when the maximum lateness is known. It is not a complete solution if records can arrive arbitrarily late.

**Option 2: Use a reliable change sequence.**

A database log sequence number, CDC offset, or another source-defined change marker can be more reliable than application timestamps, provided the source guarantees the required ordering and retention.

**Option 3: Reconcile periodically.**

Compare source and target counts, keys, or other business-level checks to identify missing or inconsistent records.

Reconciliation is especially useful when source delivery and target ingestion are separate processes.

The right solution depends on how the source behaves. A pipeline that receives complete daily snapshots has different requirements from one that consumes a continuous database change stream.

## 6\. What If the Source Record Is Updated?

Suppose customer `C101` originally has this email:

```text
customer_id: C101
email: old@example.com
```

Later, the source changes the email to:

```text
customer_id: C101
email: new@example.com
```

An incremental pipeline should not blindly append the second record if the target represents the latest state of each customer.

Instead, it should identify the business key and apply the new version.

Conceptually, the operation is:

```sql
MERGE INTO target_customers AS target
USING staged_customer_changes AS source
ON target.customer_id = source.customer_id

WHEN MATCHED THEN
  UPDATE SET
    target.email = source.email

WHEN NOT MATCHED THEN
  INSERT (customer_id, email)
  VALUES (source.customer_id, source.email);
```

This is illustrative SQL. The exact syntax and supported behavior depend on the table format and query engine.

In practice, you should also define how to handle multiple updates to the same customer within a batch, deletes, schema evolution, and conflicting updates from concurrent jobs.

## 7\. Monitoring: How Do You Know Incremental Processing Is Working?

A job marked `SUCCEEDED` only tells you that the job completed according to its execution status. It does not prove that every expected source record reached the target.

Track metrics that describe the data itself:

*   Source records or files discovered.
    
*   New, updated, and deleted records processed.
    
*   Invalid and quarantined records.
    
*   Records written to the target.
    
*   Watermark or source offset before and after the run.
    
*   Processing duration and estimated resource consumption.
    
*   Reconciliation differences between source and target.
    

AWS Glue job metrics and Amazon CloudWatch can help monitor execution, while application-level metrics and reconciliation checks provide visibility into the data.

For example, if a source delivers 50,000 changed records but the pipeline writes only 47,000, the pipeline should investigate the difference instead of treating a successful job status as proof of correctness.

Do not assume source counts and target counts must always match exactly. Deduplication, filtering, and business rules may legitimately change counts. Define expected reconciliation rules for each pipeline.

## 8\. A Practical Production Checklist

Before putting an incremental pipeline into production, check the following:

*   \[ \] The initial load and subsequent incremental runs are both defined.
    
*   \[ \] The source provides a reliable way to identify changes.
    
*   \[ \] Late-arriving records are handled.
    
*   \[ \] Updates and deletes have explicit processing rules.
    
*   \[ \] Retrying a batch does not create duplicates or corrupt the target.
    
*   \[ \] The checkpoint advances only after the required commit conditions are satisfied.
    
*   \[ \] Failed and invalid records can be investigated and replayed.
    
*   \[ \] Schema changes are detected and handled deliberately.
    
*   \[ \] Source-to-target reconciliation is monitored.
    
*   \[ \] Backfills can run without accidentally advancing the normal production checkpoint.
    
*   \[ \] Concurrent runs cannot overwrite each other's progress.
    
*   \[ \] Processing volume, runtime, and cost are measured before and after the change.
    

## Final Thoughts

Incremental processing is not just about adding a timestamp filter to a PySpark DataFrame.

It is about deciding what has changed, identifying which changes have been processed successfully, and ensuring that failures do not leave the target in an inconsistent state.

AWS Glue bookmarks, application-level watermarks, CDC, and manifest-based ingestion can all be useful, but none of them removes the need to design the target writes and recovery behavior carefully.

Start with a small, measurable improvement. Identify a pipeline that repeatedly processes the same data, determine how changes can be tracked reliably, and compare the results against a full-refresh baseline.

The goal is not simply to process fewer records. It is to process fewer records **without sacrificing correctness or recoverability**.

That is what makes an incremental pipeline production-ready.

* * *

*What strategy do you use for incremental processing in your data pipelines—Glue job bookmarks, timestamps, CDC, or something else? I'd be interested to hear what has worked well in your environment.*