Fix GCP BigQuery Load Avro Errors

When working with GCP BigQuery, you may encounter a configuration error that prevents your data pipeline or messaging system from working. This guide explains the most common mistake with load avro and shows the exact fix.

A Common Mistake

Loading Avro files into BigQuery when the Avro schema contains incompatible types (e.g., UNION types), causing load failures.

The incorrect command:

bq load --source_format=AVRO my_project:my_dataset.my_table data.avro
# Avro schema has: {"name": "value", "type": ["null", "string"]}

Error output:

Error: Unsupported Avro type: UNION. BigQuery does not support Avro UNION types directly. UNION types must be represented as nullable fields by omitting the null type.

The Correct Approach

The right way to configure load avro in GCP BigQuery:

# Use Avro schema with nullable field directly:
# {"name": "value", "type": ["null", "string"]}
# Or better: use the non-null type and set default null
# {"name": "value", "type": "string", "default": null}
bq load --source_format=AVRO my_project:my_dataset.my_table data.avro

Successful result:

Loaded successfully.
Avro files with nullable fields (["null", "string"]) are supported when null is the first element of the UNION. BigQuery maps nullable UNION types to NULLABLE mode columns. Logical types (DATE, DECIMAL) in Avro are also supported.

How to Prevent This

Avro is the recommended format for BigQuery loads: faster than CSV/JSON, supports nested/repeated types, and schema evolution. Use nullable fields (["null", "type"]) not complex UNIONs. Avro supports compression (deflate, snappy). Avro loads are schema-on-read -- the Avro schema defines the table schema.

FAQ

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro. Secure your cloud with DodaTech.