Imagine you have a data pipeline that processes 100 GB of data every morning.

The first day, everything works fine. A few weeks later, the dataset grows to 500 GB. After a few months, it reaches several terabytes.

But here is the interesting part: only a small portion of that data changes every day.

Your pipeline still reads everything.

The job takes longer, consumes more compute resources, and spends most of its time processing records that have already been processed before.

I have always found this to be one of the most important questions in data engineering:

If only 2% of the data has changed, why should we process 100% of it again?

This is where incremental data processing becomes useful.

In this article, we will explore how to design an incremental pipeline using Amazon S3, AWS Glue, and PySpark, including the problems that appear when the pipeline starts running in production.

1. Full Refresh vs. Incremental Processing

Let's start with a simple example.

Suppose an application exports customer records to Amazon S3 every day.

Your raw dataset contains 10 million records, but only 50,000 records are new or updated today.

With a full refresh, your pipeline might read and transform all 10 million records.

With incremental processing, it attempts to process only the records that have changed since the previous successful run.

These figures are illustrative, not measured benchmark results. Actual cost and runtime savings depend on file sizes, partition layout, transformation complexity, and the amount of data that must be scanned.

Incremental processing does not automatically make every pipeline faster. If your source contains millions of tiny files, for example, discovering and listing those files can remain expensive even when the number of new records is small.

Still, avoiding unnecessary work is a good starting point.

2. A Simple AWS Architecture

For this example, imagine that customer data arrives in Amazon S3 and AWS Glue transforms it into a curated dataset.

The flow looks like this:

Source Application
        |
        v
Amazon S3 - Raw Layer
        |
        v
AWS Glue + PySpark
        |
        +---- Read New or Changed Data
        |
        +---- Validate and Transform
        |
        v
Amazon S3 - Curated Layer
        |
        v
Downstream Analytics

The important part is deciding how the job knows which records are new.

There are several ways to do this:

• AWS Glue job bookmarks: Track processing progress for supported sources and transformations.

• Watermarks: Store a previously processed timestamp or another ordered change marker.

• CDC (Change Data Capture): Consume inserts, updates, and deletes from a change stream or database log.

• Manifest-based ingestion: Track which source files have been successfully processed.

These mechanisms solve related but different problems. A file bookmark does not automatically provide record-level change detection inside every file, and a timestamp watermark does not automatically handle deletes.

For our example, we will use a timestamp watermark to explain the core idea.

3. Build an Incremental Pipeline with PySpark

Assume the source provides customer records with the following columns:

customer_id
customer_name
email
updated_at

The updated_at field represents when the source record was last changed.

For the first run, the pipeline may need to process the entire available dataset. Subsequent runs can use the saved watermark to identify a smaller set of changes.

Step 1: Read the previous watermark

A watermark is a marker representing the last point in the source data that the pipeline has successfully processed.

For a real implementation, store this state durably in a system such as DynamoDB, a control table, or another suitable metadata store.

Here is a simplified example using a value supplied to the Glue job:

from datetime import datetime, timezone

previous_watermark = "2026-10-15T00:00:00Z"

watermark = datetime.fromisoformat(
    previous_watermark.replace("Z", "+00:00")
)

print(f"Previous watermark: {watermark}")

For an initial load, you need an explicit bootstrap strategy rather than assuming a previous watermark already exists.

Also, don't use the current clock time as the new watermark merely because the job started. The checkpoint must reflect data that has actually been processed successfully.

Step 2: Read and filter the source data

For this example, assume the source data is stored as Parquet files in S3 and the dataset fits the chosen scan strategy.

from pyspark.sql import functions as F

source_path = "s3://example-bucket/raw/customers/"

source_df = (
    spark.read
    .format("parquet")
    .load(source_path)
)

incremental_df = source_df.filter(
    F.col("updated_at") > F.lit(previous_watermark).cast("timestamp")
)

This demonstrates the basic filtering logic, but it is important to understand one limitation.

Filtering after reading Parquet files does not guarantee that Spark reads only the changed data from S3.

Spark may still need to discover and scan many files before applying the filter. For efficient incremental ingestion, the source layout must support it—for example, by partitioning or organizing data appropriately, reading only newly arrived files, or using a CDC source.

For production systems, validate timestamp types and timezone behavior, too. A watermark comparison is only reliable when the source's change marker has consistent semantics.

Step 3: Validate the incoming records

Incremental processing reduces the volume of work, but it does not remove the need for data quality checks.

For example, customer IDs should not be null.

valid_df = incremental_df.filter(
    F.col("customer_id").isNotNull()
)

invalid_df = incremental_df.filter(
    F.col("customer_id").isNull()
)

invalid_count = invalid_df.count()

print(f"Invalid records: {invalid_count}")

In a production pipeline, invalid records should normally be written to a quarantine location with enough information to investigate and replay them later.

Be careful with count(): it can trigger an additional Spark action and cause more computation. If the same DataFrame is used for several actions, consider whether persistence or a single aggregation is appropriate.

Also decide whether invalid records should block the entire batch or be quarantined while valid records continue. That decision should follow the data contract and business requirements.

Step 4: Write the processed data

For a simple append-only dataset, you can write the valid records to a curated S3 location:

target_path = "s3://example-bucket/curated/customers/"

(
    valid_df.write
    .format("parquet")
    .mode("append")
    .save(target_path)
)

This is fine as an illustration of writing new records, but it is not sufficient for a general customer table.

What if an existing customer's email address changes?

Appending the updated record may create two versions of the same customer. If the target must contain one current record per customer, use a storage format and write strategy that supports the required upsert behavior, such as an appropriately configured transactional table with MERGE support.

The target design should match the business requirement:

• Append-only events: append new events.

• Current-state records: apply inserts and updates using a stable key.

• Sources that support deletions: propagate delete events as well.

Never assume that an incremental source filter alone makes the target correct.

4. The Hard Part: When Should You Update the Watermark?

This is where a seemingly simple pipeline can lose data.

Imagine the job reads new records, writes some of them to the target, and then fails before it finishes.

If the pipeline advances its watermark too early, the next run may skip records that were never successfully written.

If the pipeline retries the batch, it may write records that were already committed during the previous attempt.

These are two different failure modes: missing data and duplicate data.

A safer design follows this sequence:

1. Determine the source range or batch to process.

2. Read and validate that batch.

3. Write the batch using an idempotent strategy.

4. Verify that the write completed according to the pipeline's requirements.

5. Advance the checkpoint only when the batch is safely committed.

However, the order alone does not guarantee exactly-once processing. A crash can occur after the target write succeeds but before the checkpoint is updated.

For that reason, the target write must also be safe to retry, or the system needs a coordinated commit mechanism.

For example, a deterministic batch ID, a deduplication key, and an idempotent merge can help make retries safe. The exact approach depends on the target storage format and the semantics of its write operations.

5. What About Records That Arrive Late?

Consider a record that has a source timestamp of October 14 but does not arrive until October 16.

If your pipeline has already advanced its watermark to October 15, a strict timestamp filter could miss that record.

There are several ways to handle this:

Option 1: Use a lookback window.

Read a small period before the last watermark and deduplicate or merge the results into the target.

This is simple and useful when the maximum lateness is known. It is not a complete solution if records can arrive arbitrarily late.

Option 2: Use a reliable change sequence.

A database log sequence number, CDC offset, or another source-defined change marker can be more reliable than application timestamps, provided the source guarantees the required ordering and retention.

Option 3: Reconcile periodically.

Compare source and target counts, keys, or other business-level checks to identify missing or inconsistent records.

Reconciliation is especially useful when source delivery and target ingestion are separate processes.

The right solution depends on how the source behaves. A pipeline that receives complete daily snapshots has different requirements from one that consumes a continuous database change stream.

6. What If the Source Record Is Updated?

Suppose customer C101 originally has this email:

customer_id: C101
email: old@example.com

Later, the source changes the email to:

customer_id: C101
email: new@example.com

An incremental pipeline should not blindly append the second record if the target represents the latest state of each customer.

Instead, it should identify the business key and apply the new version.

Conceptually, the operation is:

MERGE INTO target_customers AS target
USING staged_customer_changes AS source
ON target.customer_id = source.customer_id

WHEN MATCHED THEN
  UPDATE SET
    target.email = source.email

WHEN NOT MATCHED THEN
  INSERT (customer_id, email)
  VALUES (source.customer_id, source.email);

This is illustrative SQL. The exact syntax and supported behavior depend on the table format and query engine.

In practice, you should also define how to handle multiple updates to the same customer within a batch, deletes, schema evolution, and conflicting updates from concurrent jobs.

7. Monitoring: How Do You Know Incremental Processing Is Working?

A job marked SUCCEEDED only tells you that the job completed according to its execution status. It does not prove that every expected source record reached the target.

Track metrics that describe the data itself:

• Source records or files discovered.

• New, updated, and deleted records processed.

• Invalid and quarantined records.

• Records written to the target.

• Watermark or source offset before and after the run.

• Processing duration and estimated resource consumption.

• Reconciliation differences between source and target.

AWS Glue job metrics and Amazon CloudWatch can help monitor execution, while application-level metrics and reconciliation checks provide visibility into the data.

For example, if a source delivers 50,000 changed records but the pipeline writes only 47,000, the pipeline should investigate the difference instead of treating a successful job status as proof of correctness.

Do not assume source counts and target counts must always match exactly. Deduplication, filtering, and business rules may legitimately change counts. Define expected reconciliation rules for each pipeline.

8. A Practical Production Checklist

Before putting an incremental pipeline into production, check the following:

• [ ] The initial load and subsequent incremental runs are both defined.

• [ ] The source provides a reliable way to identify changes.

• [ ] Late-arriving records are handled.

• [ ] Updates and deletes have explicit processing rules.

• [ ] Retrying a batch does not create duplicates or corrupt the target.

• [ ] The checkpoint advances only after the required commit conditions are satisfied.

• [ ] Failed and invalid records can be investigated and replayed.

• [ ] Schema changes are detected and handled deliberately.

• [ ] Source-to-target reconciliation is monitored.

• [ ] Backfills can run without accidentally advancing the normal production checkpoint.

• [ ] Concurrent runs cannot overwrite each other's progress.

• [ ] Processing volume, runtime, and cost are measured before and after the change.

Final Thoughts

Incremental processing is not just about adding a timestamp filter to a PySpark DataFrame.

It is about deciding what has changed, identifying which changes have been processed successfully, and ensuring that failures do not leave the target in an inconsistent state.

AWS Glue bookmarks, application-level watermarks, CDC, and manifest-based ingestion can all be useful, but none of them removes the need to design the target writes and recovery behavior carefully.

Start with a small, measurable improvement. Identify a pipeline that repeatedly processes the same data, determine how changes can be tracked reliably, and compare the results against a full-refresh baseline.

The goal is not simply to process fewer records. It is to process fewer records without sacrificing correctness or recoverability.

That is what makes an incremental pipeline production-ready.

What strategy do you use for incremental processing in your data pipelines—Glue job bookmarks, timestamps, CDC, or something else? I'd be interested to hear what has worked well in your environment.

The confusing part is that both jobs might work perfectly when executed separately. The issue appears only when they run at the same time.

This is a common scenario to investigate when building data pipelines that use AWS Glue, Apache Spark, and Delta Lake.

In production, multiple jobs may read, insert, update, or delete records from the same table. When their operations overlap, Delta Lake's transaction mechanism must determine whether both changes can be committed safely.

Let's understand how these conflicts happen, how to identify their root causes, and what we can do to handle them.

What Is a Delta Lake Concurrency Issue?

Delta Lake uses a transaction log to track changes to a table. Instead of modifying a table without coordination, a write operation prepares its changes and attempts to commit them to the transaction log.

When multiple operations run simultaneously, their changes may conflict.

For example, imagine two AWS Glue jobs processing customer data.

• Job A deletes inactive customer records.

• Job B updates customer records with new information.

• Both jobs access overlapping data and attempt to commit their changes.

Depending on the operations, isolation settings, and Delta Lake version, one transaction may fail because its changes conflict with another committed transaction.

This is known as a concurrency conflict.

The important thing to understand is that concurrency conflicts are not necessarily bugs in your Spark code. They can be the result of multiple transactions attempting to change the same table at the same time.

Why Do Delta Lake Concurrency Issues Occur?

Let's look at the most common scenarios.

1. Multiple Glue Jobs Writing to the Same Delta Table

Consider a data pipeline where two Glue jobs update the same target table.

Job A processes today's customer updates, while Job B processes corrections from a previous batch.

Both jobs write to the same Delta table:

from delta.tables import DeltaTable

delta_table = DeltaTable.forPath(
    spark,
    "s3://my-data-lake/delta/customers/"
)

delta_table.update(
    condition="customer_status = 'ACTIVE'",
    set={"last_processed": "'2026-10-12'"}
)

If another job modifies overlapping data concurrently, the transaction may encounter a conflict.

The exact result depends on the changes made by each transaction and the concurrency behavior supported by the Delta Lake version being used.

How to address it:

• Avoid running conflicting writes simultaneously.

• Separate workloads when they can operate on independent data.

• Coordinate writers that must modify the same records.

• Investigate the exact exception before implementing retries.

2. Concurrent DELETE and UPDATE Operations

One particularly important scenario occurs when one job deletes records while another updates them.

Suppose Job A removes inactive customers:

DELETE FROM delta.`s3://my-data-lake/delta/customers/`
WHERE customer_status = 'INACTIVE';

At the same time, Job B updates customer records:

UPDATE delta.`s3://my-data-lake/delta/customers/`
SET customer_status = 'ACTIVE'
WHERE customer_id = 101;

If customer 101 is included in the delete operation's affected data, the operations may conflict.

This is especially worth investigating when a pipeline performs cleanup and data correction simultaneously.

How to address it:

Schedule cleanup after dependent updates complete, or redesign the workflow so that the two jobs operate on separate, well-defined sets of records.

Do not assume that every concurrent DELETE and UPDATE must fail. The outcome depends on the actual data overlap and the table's concurrency semantics.

3. Concurrent MERGE Operations

MERGE is frequently used in data engineering to implement upserts.

For example:

MERGE INTO delta.`s3://my-data-lake/delta/customers/` AS target
USING incoming_customers AS source
ON target.customer_id = source.customer_id

WHEN MATCHED THEN
    UPDATE SET
        target.customer_name = source.customer_name

WHEN NOT MATCHED THEN
    INSERT (
        customer_id,
        customer_name
    )
    VALUES (
        source.customer_id,
        source.customer_name
    );

Now imagine two Glue jobs running this MERGE operation with different source batches.

If both batches contain overlapping customer IDs, the jobs may attempt to change the same logical records. They may also affect overlapping data files or partitions.

Depending on the Delta Lake implementation and isolation behavior, one transaction may fail with a concurrency exception.

How to address it:

• Deduplicate source records before running MERGE.

• Ensure that the source contains a deterministic record for each business key.

• Avoid overlapping source batches where possible.

• Serialize operations that must update the same keys.

• Retry only after confirming that the failure is transient and the operation is safe to repeat.

Deduplication is particularly important because it also prevents ambiguous source data from causing separate MERGE errors.

4. Concurrent Jobs Modifying the Same Partitions

Partitioning can influence how much data a transaction needs to read or modify.

For example, suppose a Delta table is partitioned by processing date:

customers/
    processing_date=2026-10-10/
    processing_date=2026-10-11/
    processing_date=2026-10-12/

Job A processes October 12, while Job B processes October 11.

If the jobs truly modify separate partitions, the risk of conflicts may be lower. However, partition separation does not guarantee conflict-free execution in every situation.

For example, a broad DELETE, an UPDATE that touches multiple partitions, or a MERGE that reads and modifies overlapping data may still conflict with another transaction.

How to address it:

Design jobs to process clearly defined partitions and use partition filters where appropriate. Verify the actual read and write scope instead of assuming that partitioning alone prevents conflicts.

How to Identify the Root Cause

When a Glue job fails with a Delta Lake concurrency exception, I would investigate the following areas before changing the code.

Step 1: Check the Glue Job Logs

Open the failed job run and inspect its driver logs in Amazon CloudWatch.

Look for the full exception, including the exception class and the transaction operation involved.

Depending on the Delta Lake version, you may encounter errors with names such as:

• ConcurrentDeleteReadException

• ConcurrentDeleteDeleteException

• ConcurrentWriteException

• MetadataChangedException

These errors indicate different types of conflicts. Confirm the exact exception in your logs rather than assuming all concurrency errors have the same cause.

Step 2: Compare Job Execution Times

Check whether another Glue job was modifying the same Delta table when the failure occurred.

Compare:

• Job start and end times.

• Target table paths.

• Source batch identifiers.

• DELETE, UPDATE, and MERGE conditions.

• Affected partitions and business keys.

This helps determine whether the issue is related to concurrent access or another failure.

Step 3: Check the Delta Lake Version

AWS Glue version and Delta Lake version are important because supported features and transaction behavior can differ.

Confirm the libraries available to your Glue job and the configuration used to enable Delta Lake.

Do not assume that a Delta Lake feature available in a newer release is supported in your current environment.

Step 4: Review the Transaction Log

Delta Lake stores transaction information in the table's _delta_log directory.

For example:

s3://my-data-lake/delta/customers/
    _delta_log/
    part-00000-....parquet
    part-00001-....parquet

The transaction log can help you understand the sequence of committed changes and whether another transaction modified the table.

For production troubleshooting, use Delta Lake's supported history and diagnostic capabilities where available, rather than manually changing transaction-log files.

Practical Solutions for AWS Glue

Once the root cause is understood, choose a solution that matches the workload.

Solution 1: Prevent Overlapping Runs

If the same Glue job must not modify a Delta table concurrently, set its concurrency limit to one:

import boto3

glue = boto3.client("glue")

glue.update_job(
    JobName="customer-delta-job",
    ExecutionProperty={
        "MaxConcurrentRuns": 1
    }
)

This limits concurrent runs of that particular Glue job.

However, it does not prevent a different Glue job or external application from writing to the same Delta table. If several jobs share a target, coordinate their execution centrally.

Solution 2: Use AWS Step Functions to Coordinate Jobs

For pipelines with multiple dependent writers, AWS Step Functions can control the order of operations.

A workflow might look like this:

Start Pipeline
      |
      v
Process Incoming Data
      |
      v
Validate Source Records
      |
      v
Update Delta Table
      |
      v
Run Cleanup Operation
      |
      v
Validate Final Data
      |
      v
Complete

This approach is useful when cleanup must happen after an update or when multiple writes must be executed in a specific order.

Step Functions coordinates the workflow, but you must still consider other writers that operate outside that workflow.

Solution 3: Implement Safe Retries

Sometimes a transaction conflict is temporary, and retrying the operation after another transaction completes may succeed.

However, retries should not be added blindly.

A safer approach is to:

1. Identify the specific concurrency exception.

2. Confirm that retrying is appropriate for that exception.

3. Wait briefly before retrying.

4. Limit the number of attempts.

5. Record the failed attempt and the reason.

6. Validate the final table state.

For example, a simple retry structure might look like this:

import time

max_attempts = 3

for attempt in range(max_attempts):
    try:
        # Execute your Delta Lake write operation here.
        # Replace this comment with the actual operation.
        perform_delta_operation()
        break

    except Exception:
        if attempt == max_attempts - 1:
            raise

        time.sleep(2 ** attempt)

This is only a generic retry structure. In production, catch the specific retryable exception instead of catching every exception indiscriminately. Do not assume that a failed write is always safe to repeat.

Solution 4: Make Each Batch Deterministic

A reliable pipeline should know exactly which source records belong to each batch.

For example, if two jobs process the same customer ID from different source batches, they may attempt to update the same target record.

Use stable batch identifiers, deterministic deduplication rules, and clear ownership of the records each job is allowed to modify.

This reduces accidental overlap and makes failed batches easier to recover.

What I Would Avoid

A few shortcuts can make concurrency problems harder to diagnose:

• Do not retry every error. Some failures indicate persistent conflicts or incorrect logic.

• Do not disable concurrency controls without understanding the impact. Multiple writers may still conflict.

• Do not assume partitioning eliminates conflicts. Operations can read or modify data beyond the partition you expected.

• Do not manually delete files from a Delta table's storage directory. Delta Lake manages data files through its transaction log.

• Do not treat every concurrency exception as a data corruption issue. A failed transaction does not automatically mean that the table is corrupted.

Final Thoughts

Delta Lake concurrency issues can become challenging when multiple AWS Glue jobs update, delete, or merge records into the same table.

The solution is not always to reduce the number of jobs or increase Spark resources. In many cases, the real issue is how concurrent transactions interact with the same data.

Start by identifying the exact exception, comparing job execution times, and checking the affected records and partitions. Then decide whether you need execution coordination, safer batch boundaries, or a carefully controlled retry strategy.

A reliable data pipeline needs more than successful Spark execution. It also needs a clear strategy for managing concurrent changes to shared data.

Have you faced a ConcurrentDeleteReadException or another Delta Lake concurrency error in AWS Glue? Share the error and what caused it in your pipeline. Your experience may help other data engineers solve the same problem.

At first glance, they seem similar. Both help connect multiple tasks, manage execution order, handle failures, and coordinate complex processes.

But when we look at how they work and the problems they are designed to solve, the differences become clearer.

In this article, let's compare AWS Step Functions and Apache Airflow, understand their strengths, and explore when to choose one over the other.

What Is AWS Step Functions?

AWS Step Functions is a managed workflow orchestration service that coordinates AWS services and application tasks using state machines.

You define the workflow as a series of states. Each state can perform a task, make a decision, wait, run branches in parallel, or handle errors.

For example, consider a data pipeline that processes incoming files:

1. A file arrives in Amazon S3.

2. AWS Lambda validates the file.

3. AWS Glue transforms the data.

4. The workflow checks whether processing succeeded.

5. A notification is sent through Amazon SNS.

Step Functions can coordinate these operations, manage execution flow, and define retry and error-handling behavior.

One of its biggest advantages is that it integrates naturally with AWS services without requiring you to operate your own orchestration server.

What Is Apache Airflow?

Apache Airflow is an open-source platform for authoring, scheduling, and monitoring workflows programmatically.

Workflows are represented as Directed Acyclic Graphs (DAGs), typically written in Python. Each task represents an operation, and dependencies define the order in which tasks execute.

For example, a daily analytics pipeline might:

1. Extract data from multiple sources.

2. Run data-quality checks.

3. Transform datasets using Spark or SQL.

4. Load the results into a data warehouse.

5. Run validation queries.

6. Publish the final dataset for reporting.

Airflow is especially useful when workflows contain many dependent data-processing tasks and need scheduling, observability, backfills, and centralized pipeline management.

You can run Airflow yourself or use a managed offering such as Amazon Managed Workflows for Apache Airflow (Amazon MWAA).

Step Functions vs Airflow: What Is the Difference?

Although both tools orchestrate workflows, their design priorities are different.

Neither tool is universally better. The right choice depends on the workflow, operational requirements, and how much control you need over orchestration.

Use Case 1: When AWS Step Functions Makes More Sense

Imagine an event-driven pipeline in AWS.

Whenever a customer uploads a file to S3, the system needs to validate it, process it, and notify the relevant team.

A possible workflow is:

S3 Upload
    |
    v
EventBridge / Lambda
    |
    v
Step Functions
    |
    v
Validate File
    |
    v
Run AWS Glue Job
    |
    v
Check Result
   / \
Success Failure
   |      |
   v      v
Publish  Retry / Handle Error
   |
   v
Send Notification

Step Functions is a good fit here because the workflow is triggered by an event and coordinates several AWS services.

You can define retries for transient failures, branch based on job results, and track individual workflow executions.

This is particularly useful for event-driven data ingestion, file processing, application workflows, and automated recovery processes.

Use Case 2: When Apache Airflow Makes More Sense

Now consider a data engineering platform that processes data from several source systems every day.

The pipeline has dependencies between ingestion, transformation, data-quality checks, and reporting. Sometimes historical data must be reprocessed after a transformation changes.

An Airflow DAG could look like this:

Extract Customer Data     Extract Sales Data
          |                       |
          v                       v
       Validate                 Validate
          |                       |
          +-----------+-----------+
                      |
                      v
              Transform with Spark
                      |
                      v
                Data Quality
                      |
                      v
               Load Warehouse
                      |
                      v
              Publish Analytics

Airflow is a strong fit because you can express dependencies in Python, schedule recurring workflows, monitor task status, and manage historical reruns.

For data engineering teams managing dozens or hundreds of related DAGs, these capabilities can be especially valuable.

Airflow can also orchestrate AWS services, including Glue jobs, so using it does not mean abandoning AWS-native processing services.

Can We Use Both in the Same Architecture?

Yes. In some systems, using both is a sensible design.

For example, an enterprise data platform might use Airflow to schedule daily pipelines across multiple systems, while Step Functions manages a particular event-driven process that needs detailed branching and recovery.

Apache Airflow
      |
      v
Daily Data Pipeline
      |
      v
Trigger Step Functions
      |
      v
Validate and Process Files
      |
      v
Run AWS Glue
      |
      v
Return Workflow Status
      |
      v
Continue Airflow DAG

This is a conceptual example, not a mandatory architecture. The integration requires a suitable Airflow operator or API call, and the team must define how completion, failures, retries, and duplicate executions are handled.

However, there is an important consideration: don't introduce both tools just because you can.

If one service can orchestrate the entire workflow cleanly, adding another orchestrator may increase operational complexity, monitoring effort, and failure-handling responsibilities.

What About AWS Glue Workflows?

Since AWS Glue is commonly used for data engineering, it is also worth mentioning Glue Workflows.

AWS Glue Workflows can coordinate Glue jobs, crawlers, and triggers for ETL use cases. They may be sufficient when the workflow is primarily contained within Glue.

Step Functions becomes attractive when you need broader service orchestration, richer branching, and integration with services outside Glue.

Airflow is worth considering when you need a centralized platform for scheduling and managing pipelines across multiple systems.

How Do I Choose the Right Tool?

I would start by asking three questions.

1. What triggers the workflow?

If the process starts in response to an event and coordinates AWS services, Step Functions may be a natural fit. For recurring pipelines with schedules and data dependencies, Airflow is often a strong candidate.

2. How complex are the dependencies?

For application-style workflows with branching, waiting, and service-level error handling, Step Functions is well suited. For Python-defined DAGs, historical backfills, and a large collection of data pipelines, Airflow offers useful capabilities.

3. Who will operate the system?

Step Functions reduces the need to manage orchestration infrastructure. With Airflow, the team needs to account for the Airflow environment, workers, dependencies, upgrades, and operational monitoring, even when using a managed service.

Cost, execution duration, concurrency, service quotas, and workflow complexity should also be evaluated before making a production decision.

Final Thoughts

AWS Step Functions and Apache Airflow may appear to do the same thing because both coordinate tasks and manage workflow dependencies. But their strengths come from different design priorities.

Step Functions is particularly useful for managed, event-driven orchestration across AWS services. Airflow is particularly useful for scheduling and managing complex data pipelines through Python-defined DAGs.

For data engineers, the goal should not be to choose the most popular tool. It should be to choose the one that fits the problem without introducing unnecessary complexity.

My takeaway: Understand the workflow first, then choose the orchestration tool.

I'd love to hear from other data engineers: Do you prefer AWS Step Functions or Apache Airflow for your data pipelines, and what made you choose it?

I think many developers have done something similar while learning or building a quick prototype.

For example:

DATABASE_PASSWORD = "my_database_password"
PAYMENT_API_KEY = "my_payment_api_key"

It works locally. Your application connects to the database, calls the external API, and everything looks fine.

But what happens when you push that code to GitHub or deploy it to a production environment?

If those credentials end up in the wrong place, someone could use them to access your database or external services.

Moving secrets into a .env file is a useful step for local development, but production applications need a reliable way to manage credentials, control access, and handle rotation.

That is where HashiCorp Vault comes in.

In this article, we'll build a small example to understand how a Python backend can retrieve database credentials and API keys from Vault instead of keeping them in the source code.

What Is HashiCorp Vault?

HashiCorp Vault is a secrets management platform that helps applications securely manage sensitive information.

Some common examples include:

• Database usernames and passwords

• Third-party API keys

• Application credentials

• Certificates and encryption keys

Vault provides authentication mechanisms and policies that determine which identities can access particular secrets. It also supports dynamic secrets for supported systems, allowing credentials to be generated for a limited period instead of relying entirely on permanent passwords.

For this tutorial, we'll focus on a simpler use case: storing secrets in Vault and retrieving them from a Python application using the Key-Value (KV) secrets engine.

Understanding the Architecture

Before writing the code, let's look at how the process works.

1. An administrator stores application secrets in Vault.

2. The Python backend authenticates with Vault.

3. Vault verifies the application's identity and permissions.

4. The application requests a particular secret path.

5. Vault returns the permitted secret values.

6. The application uses those values to connect to a database or call an external API.

The important idea is that the application source code does not need to contain the actual database password or API key.

Step 1: Start Vault and Store the Secrets

We'll use a local development server to keep this example simple.

Start Vault:

vault server -dev

The development server prints a token that can be used for this local exercise. Keep it private and do not use it in production.

In another terminal, configure the connection details.

Linux or macOS:

export VAULT_ADDR="http://127.0.0.1:8200"
export VAULT_TOKEN="<your-development-token>"

Windows PowerShell:

$env:VAULT_ADDR = "http://127.0.0.1:8200"
$env:VAULT_TOKEN = "<your-development-token>"

The standard development setup enables a KV v2 secrets engine at the secret/ mount.

Let's store some sample credentials:

vault kv put secret/my-backend-app \
  database_username="app_user" \
  database_password="example-password" \
  external_api_key="example-api-key"

These values are for demonstration only.

We've now stored three values under the secret path:

secret/my-backend-app

The application will use this path to retrieve the values when it needs them.

Step 2: Create a Read-Only Policy

Storing secrets securely is important, but controlling access is equally important.

We don't want every application or developer to have permission to read, modify, or delete every secret in Vault.

Create a file named backend-read.hcl:

path "secret/data/my-backend-app" {
  capabilities = ["read"]
}

Why does the policy use secret/data/ instead of just secret/?

Because KV v2 uses a data API path for reading secret values. The CLI hides some of this detail, but the policy must use the correct API path.

Load the policy:

vault policy write backend-read backend-read.hcl

This policy grants read access to the specified secret. It does not grant write or delete permissions.

For this tutorial, we're using the development token to demonstrate the API call. The policy is an example of how access should be restricted; the development root token itself is not restricted by this policy.

In production, authenticate the application using a suitable method, such as AppRole or a platform-native identity mechanism, and associate the appropriate policy with that identity.

Never use the Vault root token as a permanent application credential.

Step 3: Connect Python to Vault

Now let's retrieve the secrets from Python.

First, install the Vault client library:

pip install hvac

The hvac library allows Python applications to interact with Vault's API.

Create a file named app.py:

import os
import hvac

vault_url = os.environ["VAULT_ADDR"]
vault_token = os.environ["VAULT_TOKEN"]

client = hvac.Client(
    url=vault_url,
    token=vault_token
)

if not client.is_authenticated():
    raise RuntimeError("Vault authentication failed")

response = client.secrets.kv.v2.read_secret_version(
    path="my-backend-app",
    mount_point="secret"
)

secrets = response["data"]["data"]

database_username = secrets["database_username"]
database_password = secrets["database_password"]
external_api_key = secrets["external_api_key"]

print("Secrets retrieved successfully")

Let's break down what's happening.

1. Create the Vault client

The application reads the Vault URL and token from environment variables rather than embedding them in the Python source code.

2. Check authentication

The application verifies whether the supplied credentials are accepted by Vault.

3. Request the secret

The read_secret_version() method retrieves the secret stored at secret/my-backend-app.

4. Extract the values

The returned response contains the secret data. We extract the database username, database password, and API key so the application can use them.

Notice that we print only a success message. Logging the full response could accidentally expose credentials.

For a remote production Vault server, use HTTPS with proper TLS certificate verification. Also, replace the development token with an appropriate application authentication method.

Step 4: Use the Retrieved Credentials

Retrieving a secret is only useful if the application can use it.

For example, a Python application using PostgreSQL could create a connection like this:

import os
import psycopg

connection = psycopg.connect(
    host=os.environ["DB_HOST"],
    dbname=os.environ["DB_NAME"],
    user=database_username,
    password=database_password
)

In this example, the database host and database name come from environment variables, while the sensitive database credentials come from Vault.

The same approach applies to external API calls. The application can pass external_api_key to the appropriate HTTP client when communicating with the service that issued the key.

One thing to remember: retrieving secrets securely doesn't automatically make every part of the application secure. You still need to protect logs, memory access, network connections, and the application's own authentication credentials.

Step 5: What Happens When an API Key Changes?

Let's say your external service issues a replacement API key.

Without centralized secrets management, you might have to find where the key is configured, update deployment settings, and redeploy the application.

With Vault, an authorized operator can update the secret:

vault kv put secret/my-backend-app \
  database_username="app_user" \
  database_password="example-password" \
  external_api_key="replacement-example-key"

KV v2 maintains secret versions, so this update creates a new version under the default configuration.

However, there's an important detail: updating a secret in Vault does not automatically replace a value that your application has already loaded into memory.

Your application needs a strategy for refreshing its credentials, such as re-reading the secret when appropriate or restarting the relevant service.

For systems that support dynamic credentials, Vault can also help manage short-lived credentials and their lifecycle.

Best Practices When Using Vault

A few practices are worth keeping in mind when moving from a local experiment to a real application.

• Follow least privilege: Give each application access only to the secrets it requires.

• Avoid permanent root tokens: Use an appropriate authentication method for the application.

• Use TLS: Protect connections to remote Vault servers and verify certificates.

• Keep secrets out of logs: Never print complete secret responses or credentials.

• Plan for rotation: Make sure the application can handle credential changes.

• Protect the Vault deployment: Secure its storage, authentication configuration, policies, and administrative access.

Also, remember that the Vault token is itself a sensitive credential. Moving an API key into Vault but exposing the token that can retrieve it would defeat much of the benefit.

Final Thoughts

The goal isn't simply to move passwords from one file to another. It's to separate sensitive credentials from application code and control how applications access them.

HashiCorp Vault provides a way to do that through centralized secret storage, authentication, and access policies.

We've walked through a small Python example: storing secrets in Vault, creating a read-only policy, retrieving the values, and using them in a backend application.

This is a starting point rather than a complete production deployment. The next steps are to explore application authentication, credential rotation, and dynamic secrets in more detail.

I'm learning more about HashiCorp Vault and how it fits into backend and cloud engineering workflows.

How do you currently manage application secrets in your projects?

Do you use environment variables, a cloud secrets manager, HashiCorp Vault, or a combination of these approaches?

I'd be interested to hear about your experience.

There are too many small files in S3.

A file that is only a few KB doesn't seem like a problem.

But what happens when you have thousands or even millions of those files?

That's when things can get interesting.

In one of my data pipelines, I faced an issue where an AWS Glue job was trying to read data from S3 but started failing with errors related to too many requests, throttling, and file read failures.

The root cause was not that the data was corrupted.

The real problem was the number of small files.

In this article, I'll explain what happened, why AWS Glue struggled to read the data, what the S3 requests had to do with it, and how we solved the problem.

The Architecture

The basic flow looked something like this:

Source → S3 → AWS Glue → Processing → Output

The data was being written into S3, and AWS Glue was responsible for reading those files and processing them.

Initially, everything looked normal.

Then we started seeing failures when the number of files increased.

The Problem: Too Many Small Files

The biggest issue was that the data was split into a very large number of small files.

For example, instead of having:

file-001.parquet   128 MB
file-002.parquet   128 MB
file-003.parquet   128 MB

we ended up with something closer to:

file-001.parquet   1 KB
file-002.parquet   2 KB
file-003.parquet   1 KB
file-004.parquet   3 KB
file-005.parquet   1 KB
...
...
thousands of files

Individually, these files were tiny.

Collectively, they became a problem.

Why?

Because the number of files matters, not just the total amount of data.

Suppose we have:

1 GB of data

There is a big difference between storing it as:

10 × 100 MB files

and:

1,000,000 × 1 KB files

The amount of data is approximately the same.

But the number of objects that have to be discovered, opened, requested, and processed is completely different.

What Was Happening Inside AWS Glue?

When AWS Glue reads data from S3, it doesn't simply make one request and magically receive the entire dataset.

There are multiple interactions involved in discovering and reading objects.

With a large number of files, the amount of S3 activity can increase significantly.

Conceptually, the process looks like this:

S3
 │
 ├── small-file-001
 ├── small-file-002
 ├── small-file-003
 ├── small-file-004
 ├── small-file-005
 ├── ...
 └── small-file-N
          │
          ▼
      AWS Glue
          │
          ▼
   Many read operations
          │
          ▼
   High request rate
          │
          ▼
      Throttling

This is where the problem started becoming visible in the Glue logs.

The Error in the Glue Logs

One of the important clues was the S3-related error.

The exact message can vary depending on the operation and implementation, but errors can include things such as:

AmazonS3Exception:
Slow Down
Status Code: 503

or errors indicating that a large number of requests were being made while Glue was attempting to read the objects.

A simplified example looks like:

com.amazonaws.services.s3.model.AmazonS3Exception:
Slow Down

Status Code: 503
Error Code: SlowDown

You may also see failures around reading particular files or partitions.

At first glance, it is tempting to think:

"Maybe one of the S3 files is corrupted."

But that wasn't the actual issue in our case.

The important clue was that the problem appeared when Glue was dealing with a very large number of small objects.

Understanding the Small Files Problem

This is a common problem in distributed data processing.

Imagine a restaurant with 100 customers.

If the customers arrive as 10 groups of 10 people, the restaurant can manage them reasonably well.

Now imagine the same 100 customers arriving individually, one after another.

The restaurant has to perform many more individual interactions.

The same basic idea applies here.

Instead of Glue dealing with:

100 large files

it may have to deal with:

100,000 small files

Even if the total data volume isn't huge.

Why Small Files Can Hurt Performance

There are several reasons.

1. More S3 requests

More objects generally means more object-related operations.

Instead of reading a few large objects, Glue needs to work with a much larger number of objects.

That increases request activity against S3.

2. More Spark tasks

AWS Glue uses Apache Spark under the hood for many workloads.

When Spark sees many input files, those files can translate into many input partitions/tasks.

For example:

S3
 │
 ├── file-1
 ├── file-2
 ├── file-3
 ├── ...
 └── file-10000
        │
        ▼
   Spark / Glue
        │
        ├── Task 1
        ├── Task 2
        ├── Task 3
        ├── ...
        └── Task N

Now Spark has additional scheduling and task-management overhead.

3. More metadata overhead

Even if each file contains very little actual data, the system still needs to discover and manage those objects.

So you can end up spending significant effort handling files rather than processing the actual data.

This is one reason why:

Small files can become a performance problem even when the total data size is small.

The Root Cause in Our Case

After looking at the data flow, the problem became much clearer.

The pipeline was producing a large number of very small files.

The Glue job then tried to read those files.

That resulted in a high number of S3 operations.

Eventually, the request rate became a problem and we started seeing throttling/read failures.

So the chain looked like this:

Too many small files
        ↓
Large number of objects to read
        ↓
Many S3 requests
        ↓
High request rate
        ↓
S3 throttling / Slow Down responses
        ↓
Glue read failures
        ↓
Glue job failure

That was the real issue.

The Solution

The most important lesson from this issue was:

Don't just try to make Glue handle more small files. Try to reduce the number of small files.

Instead of continuously increasing retries or resources, we focused on the actual problem.

We needed to combine the small files into larger files.

This process is commonly called:

• File compaction

• Small-file compaction

• File consolidation

The idea is simple:

Before

1 KB   2 KB   1 KB   3 KB   2 KB   1 KB
 │      │      │      │      │      │
 └──────┴──────┴──────┴──────┴──────┘
                  ↓
          Many small objects


After

        ┌───────────────────┐
        │    Larger file    │
        │      64 MB        │
        └───────────────────┘

        ┌───────────────────┐
        │    Larger file    │
        │      128 MB       │
        └───────────────────┘

Now Glue has significantly fewer files to deal with.

Using Spark to Compact the Files

If the data is being processed with Spark, one approach is to read the small files and write them back using a controlled number of partitions.

For example:

df = spark.read.parquet("s3://my-bucket/input/")

df = df.repartition(10)

df.write \
    .mode("overwrite") \
    .parquet("s3://my-bucket/compacted/")

The exact number of partitions should depend on the size of your dataset and workload.

Don't blindly use:

repartition(1)

for everything.

That forces all the data through a single partition and can create a bottleneck.

coalesce() vs repartition()

This is another important detail.

You may see code like:

df = df.coalesce(10)

or:

df = df.repartition(10)

They are not exactly the same.

coalesce()

coalesce() is generally useful when you want to reduce the number of partitions without a full shuffle.

For example:

df = df.coalesce(10)

This can be useful when reducing the number of output files.

repartition()

repartition() performs a shuffle and can redistribute the data across partitions.

df = df.repartition(10)

This can provide a more even distribution, but the shuffle itself has a cost.

So the choice depends on the workload.

An Important Point: coalesce() Is Not the Whole Solution

One thing I learned from this problem is that there is a difference between:

Reducing Spark partitions

and

Designing a proper file-compaction strategy.

If the source system is continuously producing thousands of tiny files, simply changing the number of Spark partitions in one downstream job may not completely solve the underlying problem.

You should also ask:

Why are these small files being created in the first place?

For example:

Source
  ↓
Many tiny writes
  ↓
S3
  ↓
Thousands of small files
  ↓
Glue

A better architecture tries to prevent or compact those files earlier:

Source
  ↓
Controlled writes
  ↓
Larger files
  ↓
S3
  ↓
Glue
  ↓
Processing

Another Option: Glue File Grouping

AWS Glue also provides mechanisms for grouping input files for processing.

For workloads with many small files, Glue's file grouping options can help reduce the number of Spark tasks created for individual files.

For example, depending on the Glue version and source configuration, you may see options such as:

groupFiles
groupSize

The important distinction is:

Grouping files for processing is not the same thing as physically combining the S3 objects.

If you use file grouping, the physical files in S3 still exist.

If you perform compaction, you actually create fewer, larger objects.

Both approaches can be useful, but they solve slightly different parts of the problem.

Don't Immediately Increase Retries

When you see an error such as:

Slow Down
503
TooManyRequests

your first thought might be:

"Let's increase the retry count."

Retries can help with transient failures.

But if the underlying problem is a huge number of small files generating a very high request rate, simply increasing retries may make the situation worse or prolong the job.

You could end up with:

Request
   ↓
Throttle
   ↓
Retry
   ↓
Throttle
   ↓
Retry
   ↓
Throttle
   ↓
Job eventually fails

The better approach is to reduce unnecessary request pressure where possible.

How I Would Troubleshoot This Today

If an AWS Glue job suddenly starts failing while reading from S3, I would check the following.

Step 1: Check the number of files

Don't only check the total size.

Look at:

Total objects
Average object size
Largest object
Smallest object
Number of objects per partition/prefix

For example:

Total data:       500 GB
Number of files:  8 million
Average size:     ~65 KB

That should immediately make you suspicious of a small-files problem.

Step 2: Check the Glue logs

Look for errors such as:

503 Slow Down
TooManyRequests
Read timeout
S3 request failures

Also look at when the failures occur.

If the job fails while discovering or reading a very large number of objects, the file count is worth investigating.

Step 3: Check the S3 layout

Look at the prefixes and partitions.

For example:

s3://bucket/raw/date=2026-09-25/

Then check how many objects exist underneath that prefix.

Sometimes one partition may contain an unexpectedly large number of files.

Step 4: Check how the files are generated

This is often the most important step.

Ask:

Why are we producing so many files?

Potential causes include:

• Too many Spark partitions

• Frequent small writes

• Streaming-style ingestion

• Poor partitioning strategy

• Multiple upstream jobs writing independently

• Very small batches

• Over-partitioned datasets

Fixing the producer can be much better than repeatedly fixing the consumer.

What File Size Should We Target?

There isn't one magic number that works for every workload.

A common approach for analytics workloads is to aim for reasonably sized files rather than thousands of tiny ones.

For example:

1 KB       ❌
10 KB      ❌
100 KB     ❌
1 MB       ⚠️
10 MB      ⚠️
64 MB      ✅
128 MB     ✅
256 MB     ✅

These are only illustrative ranges, not hard AWS requirements.

The right size depends on:

• Dataset size

• Query pattern

• Number of partitions

• Glue/Spark workload

• Compression

• Downstream consumers

• Frequency of writes

The goal isn't:

"Make every file exactly 128 MB."

The goal is:

Avoid unnecessarily large numbers of tiny files.

A Simple Before and After

Before

S3

file-001 → 1 KB
file-002 → 2 KB
file-003 → 1 KB
file-004 → 3 KB
...
file-100000 → 2 KB

          ↓

AWS Glue

Huge number of input files
          ↓
Many requests
          ↓
High request rate
          ↓
Slow Down / read failures

After

S3

part-001 → 128 MB
part-002 → 128 MB
part-003 → 128 MB
part-004 → 128 MB

          ↓

AWS Glue

Fewer input files
          ↓
Fewer file operations
          ↓
Less overhead
          ↓
More stable processing

The Bigger Lesson

This issue taught me something that is easy to overlook when working with cloud data platforms.

When we look at a dataset, we usually ask:

"How much data do we have?"

But in distributed systems, another question is equally important:

"How many objects are we dealing with?"

For example:

Dataset A

1 TB
1,000 files

Dataset B

1 TB
10,000,000 files

Both contain approximately the same amount of data.

But their processing characteristics can be very different.

The second dataset has a massive small-files problem.

Key Takeaways

Here are the main things I took away from this issue:

1. Small files are not always harmless

A few small files are fine.

Millions of small files can become a serious operational problem.

2. Look at file count, not only data volume

Always check:

Total data
+
Number of files
+
Average file size

3. Too many files can increase S3 request activity

When a distributed processing job has to work with a huge number of objects, request volume and metadata/task overhead can become significant.

4. Glue file grouping and compaction are different

Grouping can reduce processing/task overhead.

Compaction actually reduces the number of physical S3 objects.

5. Don't blindly increase retries

Retries are useful for transient failures, but they don't fix a poor file layout.

6. Fix the source when possible

If an upstream application is continuously generating tiny files, fixing the writing strategy is often better than repeatedly compensating downstream.

Final Thoughts

The interesting part of this issue was that the dataset itself wasn't necessarily huge.

The problem was how the data was stored.

A large number of tiny files created a completely different workload for AWS Glue and S3 than a smaller number of reasonably sized files.

Once we looked at the problem from that perspective, the solution became much clearer:

Too many small files
        ↓
Reduce the file count
        ↓
Compact / group files
        ↓
Fewer read operations
        ↓
More stable Glue processing

So, if your AWS Glue job suddenly starts throwing S3 read errors, don't immediately assume that the data is corrupted.

Check the number and size of the files first.

Sometimes the real problem isn't the data.

It's the number of files containing the data.

Tags

AWS AWS Glue Amazon S3 Apache Spark PySpark Data Engineering Big Data Cloud Computing ETL