Clojure Guide — Transducers: Composable Data Transformations

In this tutorial, you will learn about Clojure Guide. We cover key concepts, practical examples, and best practices to help you master this topic.

Transducers are composable transformation functions in Clojure that separate the what of data processing from the where, enabling the same transformation to apply to collections, streams, channels, and other data sources.

What You'll Learn

• What transducers are and why they matter
• Creating transducers with map, filter, take, drop
• Composing transducers for efficient pipelines
• Using transducers with channels and streams

Why It Matters

Transducers eliminate intermediate allocations and decouple transformation logic from data sources. A single transducer pipeline works with any collection, async channel, or observable. Durga Antivirus Pro uses transducers for efficient file scanning pipelines.

Real-World Use

Transducers Process log files, transform data streams, and build reusable transformation pipelines. Netflix uses Clojure transducers for data processing pipelines.

flowchart LR
    A["Transducers"] --> B["Core Transducers"]
    B --> C["Composition"]
    C --> D["with Collections"]
    D --> E["with Channels"]
    A:::current --> B
    style A fill:#2563eb,stroke:#2563eb,color:#fff
    style B fill:#dbeafe,stroke:#2563eb,color:#1e40af
    style C fill:#dbeafe,stroke:#2563eb,color:#1e40af
    style D fill:#dbeafe,stroke:#2563eb,color:#1e40af
    style E fill:#f1f5f9,stroke:#94a3b8,color:#64748b

What is a Transducer?

A transducer is a function that takes a reducing function and returns a new reducing function. It transforms the process of reduction without knowing what is being reduced.

;; A transducer
(def xf (filter even?))

;; Using with a collection
(transduce xf conj [1 2 3 4 5 6])
;; => [2 4 6]

Core Transducer Functions

(require '[clojure.core :as core])

;; map transducer
(def xf-map (map #(* % 2)))
(transduce xf-map conj [1 2 3])
;; => [2 4 6]

;; filter transducer
(def xf-filter (filter odd?))
(transduce xf-filter conj [1 2 3 4 5])
;; => [1 3 5]

;; take transducer
(def xf-take (take 3))
(transduce xf-take conj (range 100))
;; => [0 1 2]

Composing Transducers

Transducers compose left-to-right (unlike functions) and apply in order:

;; Compose transducers
(def xf (comp
          (filter odd?)
          (map #(* % 2))
          (take 5)))

(transduce xf conj (range 100))
;; => [2 6 10 14 18]

;; This processes elements one at a time
;; No intermediate collections created

Transducers with Collections

;; into - builds result collection
(into [] xf (range 10))
;; => [2 6 10 14 18]

;; sequence - lazy sequence
(seq xf (range 10))
;; => (2 6 10 14 18)

;; transduce - with explicit reducer
(transduce xf + 0 (range 10))
;; => 50

Transducers with Channels

Transducers work with core.async channels:

(require '[clojure.core.async :as async])

;; Channel with transducer
(let [ch (async/chan 10 (comp
                          (filter number?)
                          (map inc)))]
  (async/>!! ch 1)
  (async/>!! ch "a")
  (async/>!! ch 2)
  (println (async/<!! ch))  ; 2
  (println (async/<!! ch))) ; 3

Custom Transducers

(defn my-transducer
  "A transducer that skips every nth element"
  [n]
  (fn [rf]
    (let [counter (volatile! 0)]
      (fn
        ([] (rf))
        ([result] (rf result))
        ([result input]
         (vswap! counter inc)
         (if (zero? (mod @counter n))
           result
           (rf result input)))))))

(transduce (my-transducer 3) conj [1 2 3 4 5 6])
;; => [1 2 4 5]

Early Termination

;; Use reduced to signal early termination
(def xf (comp
          (filter odd?)
          (take-while #(< % 10))))

(transduce xf conj [1 2 5 8 11 13])
;; => [1 5 8]

Stateful Transducers

Some transducers like dedupe, partition-by, and distinct maintain internal state:

(transduce (partition-by identity) conj [1 1 2 2 2 3 1])
;; => [(1 1) (2 2 2) (3) (1)]

(transduce (dedupe) conj [1 1 2 2 3 3 1])
;; => [1 2 3 1]

Common Mistakes

1. Confusing comp order

Transducers compose left-to-right: (comp a b) applies a first, then b. This is opposite to (comp f g) for functions.

2. Using lazy seq where transducers shine

For one-off transformations, use transducers. Lazy seqs create intermediate allocations.

3. Stateful transducer reuse

Stateful transducers share state across uses. Create a new transducer instance for each use.

4. Forgetting completion arity

Custom transducers must handle the 0-arity (init) and 1-arity (completion) calls correctly.

5. Transducing infinite seqs without limiting

Always use take or take-while when transducing infinite sequences.

Practice Questions

1. What problem do transducers solve? They decouple transformation logic from input/output sources and eliminate intermediate collection allocations.

2. How is (comp a b) different for transducers? For transducers, (comp a b) applies transformation a before transformation b. This is the opposite of regular function composition.

3. Can transducers be stateful? Yes. dedupe, partition-by, and distinct are stateful transducers that maintain internal state across elements.

Challenge: Write a custom transducer that computes a running average of numeric input.

FAQ

{{< faq question="Are transducers faster than lazy sequences?" >}} Yes, transducers avoid creating intermediate lazy seq objects and process elements one at a time through the pipeline. This reduces GC pressure and improves throughput. {{< /faq >}}

{{< faq question="Can transducers work with maps?" >}} Yes, but you need map-specific transducers. Use (map-kv ...) or process entries as pairs. {{< /faq >}}

{{< faq question="Do transducers work in ClojureScript?" >}} Yes, transducers are a core Clojure feature available in both Clojure and ClojureScript. {{< /faq >}}

{{< faq question="What is the difference between transduce and reduce?" >}} transduce applies a transducer before the reducing function. reduce applies only the reducing function. Transduce enables composable transformations. {{< /faq >}}

{{< faq question="Can I convert a transducer to a function?" >}} Use (comp ...) to compose transducers into a single transformation. Use into [] or sequence to apply it to a collection. {{< /faq >}}

Mini Project

Create a log processing pipeline using transducers:

(def log-transducer
  (comp
    (filter #(re-find #"ERROR" %))
    (map #(re-matches #"(.*) - (.*)" %))
    (take 100)))

;; Process log file
(with-open [rdr (clojure.java.io/reader "app.log")]
  (into [] log-transducer (line-seq rdr)))

What's Next

Now that you understand transducers, explore core.async for asynchronous programming with channels and go blocks.