Cloud Data Pipelines — AWS Glue, Azure Data Factory & GCP Dataflow Guide

In this tutorial, you'll learn about Cloud Data Pipelines. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Cloud Data Pipelines automate the movement and transformation of data between sources and destinations — handling ETL, real-time streaming, and schema evolution without managing servers.

What You'll Learn

You'll learn how to build batch Etl Pipelines with Glue and Data Factory, implement Stream Processing with Dataflow, and orchestrate complex data workflows with monitoring and error handling.

Why It Matters

Raw data in silos is useless. Pipelines clean, transform, and load data into warehouses where analytics tools and ML models can use it. DodaZIP uses Glue to process terabytes of compression logs daily, feeding usage analytics dashboards.

Real-World Use

A retail chain ingests sales data from 500 stores in different formats. Data Factory normalizes the schema, Glue crawlers catalog the tables, and Dataflow enriches the stream with weather data — all running automatically every 15 minutes.

Data Pipeline Architecture

flowchart LR
  A[Source: Databases] --> B[Ingestion Layer]
  C[Source: Streams] --> B
  D[Source: Files] --> B
  B --> E["Transform / Clean / Enrich"]
  E --> F["Data Lake: S3 / ADLS / GCS"]
  E --> G["Warehouse: Redshift / Synapse / BigQuery"]
  F --> H[Analytics & ML]
  G --> H
  style B fill:#48f,color:#fff
  style E fill:#f90,color:#fff

AWS Glue

Glue is a Serverless ETL service with a built-in data catalog and crawlers.

# Create a Glue database and crawler
aws glue create-database \
  --database-input '{"Name":"sales_db"}'

aws glue create-crawler \
  --name sales-crawler \
  --role arn:aws:iam::123456789012:role/GlueServiceRole \
  --database-name sales_db \
  --targets '{"S3Targets":[{"Path":"s3://raw-sales-data/"}]}'

# Start the crawler
aws glue start-crawler --name sales-crawler

# Glue ETL script (Spark-based)
import sys
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

sc = SparkContext()
glue_context = GlueContext(sc)
spark = glue_context.spark_session
job = Job(glue_context)

source_df = glue_context.create_dynamic_frame.from_catalog(
    database="sales_db", table_name="raw_sales"
)

transformed_df = source_df.drop_fields(["internal_id", "raw_json"]) \
    .rename_field("created_at", "event_date") \
    .filter(lambda row: row["amount"] > 0)

glue_context.write_dynamic_frame.from_catalog(
    frame=transformed_df,
    database="sales_db",
    table_name="clean_sales"
)

job.commit()

Azure Data Factory

Data Factory provides visual and code-based pipeline orchestration.

# Create a data factory
az datafactory create \
  --name dodatech-adf \
  --resource-group my-rg \
  --location eastus

# Create a linked service for source
az datafactory linked-service create \
  --factory-name dodatech-adf \
  --resource-group my-rg \
  --linked-service-name AzureBlobStorage \
  --properties '{"type":"AzureBlobStorage","typeProperties":{"connectionString":"DefaultEndpointsProtocol=https;AccountName=mystorage;"}}'

GCP Dataflow

Dataflow unifies batch and Stream Processing using Apache Beam.

import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions

options = PipelineOptions(
    project="my-project",
    region="us-central1",
    runner="DataflowRunner",
    temp_location="gs://my-bucket/temp/"
)

def parse_sale(line):
    fields = line.split(",")
    return {"product": fields[0], "amount": float(fields[1]), "date": fields[2]}

def format_record(record):
    return f"{record['date']},{record['product']},{record['amount']}"

with beam.Pipeline(options=options) as p:
    (p
     | "ReadFromGCS" >> beam.io.ReadFromText("gs://raw-sales-data/*.csv")
     | "ParseCSV" >> beam.Map(parse_sale)
     | "FilterInvalid" >> beam.Filter(lambda r: r["amount"] > 0)
     | "WriteToBigQuery" >> beam.io.WriteToBigQuery(
         table="my-project:sales.clean_orders",
         schema="date:DATE,product:STRING,amount:FLOAT",
         write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND
     ))

Pipeline Monitoring

# AWS Glue job monitoring
aws glue get-job-run \
  --job-name sales-etl-job \
  --run-id jr_abc123

# Check Data Factory pipeline runs
az datafactory pipeline-run query-by-factory \
  --factory-name dodatech-adf \
  --resource-group my-rg \
  --filters '[{"operand":"PipelineName","operator":"Equals","values":["SalesETL"]}]'

# List Dataflow jobs
gcloud dataflow jobs list \
  --region us-central1 \
  --filter="STATE=Running"

Common Errors

1. Schema changes breaking pipelines — A new column in source data can break strict schemas. Use schema evolution in Glue, Avro/Parquet formats, and add columns to the target first.
2. Not handling late-arriving data — Stream Processing assumes ordered events. Use watermarking in Dataflow and handle out-of-order records with windowing.
3. Small file problem — Millions of tiny files overwhelm the catalog and slow down reads. Use S3 batch operations to compact files before pipeline execution.
4. Missing error handling in ETL — One bad record can fail the entire batch. Add dead-letter queues or error tables to isolate failures.
5. Cold start delays in Serverless pipelines — Glue jobs take 1-5 minutes to start. For sub-minute latency, use streaming with Dataflow or Lambda.

Practice Questions

1. What is the difference between ETL and ELT? ETL transforms data before loading. ELT loads raw data first and transforms in the warehouse. ELT is faster for loading, ETL reduces warehouse compute.
2. How does Dataflow unify batch and streaming? Dataflow uses the same Beam API for both modes. Batch reads finite data, streaming reads unbounded data. Windowing and triggers work identically.
3. What is a Glue crawler and why is it useful? A crawler scans data sources, infers schemas, and populates the Glue Data Catalog. It automatically detects schema changes and new partitions.
4. How do you handle PII data in pipelines? Detect PII columns in the transform step, apply masking or tokenization, and ensure the data lake stores only anonymized data.
5. Challenge: Design a pipeline that ingests 10 million events/hour from IoT devices, enriches with device metadata, aggregates by device type, and loads to BigQuery. Handle duplicate events.

Mini Project

Build an end-to-end sales analytics pipeline:

• Raw CSV sales data in S3
• Glue crawler to catalog the data
• Glue ETL job cleans and transforms (drop nulls, fix date formats)
• Load parquet output to a data lake partitioned by date
• Query with Athena and visualize with QuickSight

FAQ

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.