F# Guide — Pipelines: Composable Data Processing

In this tutorial, you will learn about F# Guide. We cover key concepts, practical examples, and best practices to help you master this topic.

F# pipelines use the forward pipe operator (|>) to chain data transformations, creating readable, step-by-step sequences where data flows from one function to the next.

What You'll Learn

• Forward pipe operator (|>)
• Building data transformation pipelines
• Debugging pipelines
• Performance considerations
• Custom pipeline operators

Why It Matters

Pipelines make code read in the direction of data flow (left-to-right, top-to-bottom), matching how humans think about transformations. Durga Antivirus Pro uses pipelines for scan result processing.

Real-World Use

Etl Pipelines, data analysis, request processing middleware, and any sequence of data transformations.

flowchart LR
    A["Pipelines"] --> B["Pipe Operator"]
    B --> C["Transformations"]
    C --> D["Debugging"]
    D --> E["Custom Operators"]
    A:::current --> B
    style A fill:#2563eb,stroke:#2563eb,color:#fff
    style B fill:#dbeafe,stroke:#2563eb,color:#1e40af
    style C fill:#dbeafe,stroke:#2563eb,color:#1e40af
    style D fill:#dbeafe,stroke:#2563eb,color:#1e40af
    style E fill:#f1f5f9,stroke:#94a3b8,color:#64748b

Forward Pipe Operator

// Without pipe (nested, reads right-to-left)
let result = List.sum (List.map (fun x -> x * 2) (List.filter (fun x -> x > 0) [-1; 0; 1; 2]))

// With pipe (reads top-to-bottom)
let result =
    [-1; 0; 1; 2]
    |> List.filter (fun x -> x > 0)
    |> List.map (fun x -> x * 2)
    |> List.sum

Building Pipelines

let processData data =
    data
    |> Seq.filter (fun x -> x > 0)
    |> Seq.map (fun x -> x * x)
    |> Seq.sortByDescending id
    |> Seq.take 10
    |> Seq.toList

// Multi-step data processing
let analyzeText (text: string) =
    text
    |> fun s -> s.Split(' ')
    |> Array.filter (fun w -> w.Length > 3)
    |> Array.map (fun w -> w.ToLower())
    |> Array.countBy id
    |> Array.sortByDescending snd
    |> Array.take 5

Debugging Pipelines

// Inject debug steps
let result =
    data
    |> List.map (fun x -> x * 2)
    |> List.map (fun x -> printfn "After double: %d" x; x)
    |> List.filter (fun x -> x > 5)
    |> List.map (fun x -> printfn "After filter: %d" x; x)
    |> List.sum

// Or use a tap function
let tap f x = f x; x

let result =
    data
    |> List.map (fun x -> x * 2)
    |> tap (List.iter (printfn "Value: %d"))
    |> List.filter (fun x -> x > 5)
    |> List.sum

Custom Pipe Operators

// Pipe into two-track (Result) processing
let (>>=) result f = Result.bind f result

let processWithResult input =
    Ok input
    >>= (fun x -> if x > 0 then Ok (x * 2) else Error "negative")
    >>= (fun x -> if x < 100 then Ok x else Error "too large")

// Double-pipe for applying two functions
let (<|) f x = f x  // Backward pipe
let result = List.sum <| [1; 2; 3]

Pipeline with Side Effects

let saveToFile path data =
    System.IO.File.WriteAllText(path, data)
    data

let pipeline =
    getData()
    |> transformData
    |> validateData
    |> saveToFile "output.txt"
    |> sendNotification

Async Pipelines

let asyncPipeline input = async {
    let! data = fetchDataAsync input
    let processed = data |> transform |> validate
    let! saved = saveAsync processed
    return saved
}

// Chaining async operations
let result =
    asyncPipeline "input"
    |> Async.RunSynchronously

Common Mistakes

1. Over-piping simple expressions

Don't pipe everything. Simple add 5 3 is clearer than 3 |> add 5.

2. Breaking pipeline for side effects

If you need to debug, extract a named function rather than breaking the chain.

3. Long unreadable pipelines

Break long pipelines into named intermediate values or extract steps into functions.

4. Ignoring error handling in pipelines

Pipelines hide intermediate errors. Use Result types or add validation steps.

5. Performance from many intermediate collections

Each step can create intermediate collections. Use Seq for lazy evaluation where possible.

Practice Questions

1. What does the |> operator do? Passes the left operand as the last argument to the function on the right.

2. Why use pipelines instead of nested calls? Pipelines read in the natural order of data flow (left-to-right), unlike nested calls which read inside-out.

3. How do you debug a pipeline? Inject a tap function that logs intermediate values, or use printfn within a map step.

Challenge: Build a pipeline that reads a file, filters lines, transforms them, and writes results.

FAQ

{{< faq question="Is pipe operator slow?" >}} No. The pipe operator is inlined by the compiler and has zero runtime overhead. It's purely a readability feature. {{< /faq >}}

{{< faq question="Can I pipe into methods?" >}} Yes, but you need to use dot notation or wrap in a function: "hello" |> fun s -> s.ToUpper() or "hello" |> (fun s -> s.ToUpper()). {{< /faq >}}

{{< faq question="What is the reverse pipe?" >}} <| passes the right operand to the left function. Useful for avoiding parentheses: printfn "%d" <| 5 + 3. {{< /faq >}}

{{< faq question="Can I create custom pipeline steps?" >}} Yes. Any function of type 'T -> 'U can be used as a pipeline step. Named functions make the best steps. {{< /faq >}}

{{< faq question="How do I handle errors in pipelines?" >} Use Result types in your pipeline: Ok data >>= Process >>= validate. The >>= operator short-circuits on Error. {{< /faq >}}

Mini Project

Build an ETL pipeline for processing CSV data:

let processCsv (csv: string) =
    csv.Split('\n')
    |> Array.skip 1  // Skip header
    |> Array.map (fun line -> line.Split(','))
    |> Array.filter (fun fields -> fields.Length >= 3)
    |> Array.map (fun fields ->
        {| Name = fields.[0]; Age = int fields.[1]; Score = float fields.[2] |})
    |> Array.filter (fun r -> r.Score > 80.0)
    |> Array.sortByDescending (fun r -> r.Score)
    |> Array.take 10

What's Next

Now that you understand pipelines, explore function composition for building new functions.