decoder.tools

Parquet / Avro Header Decoder

Read Parquet and Avro file headers, schemas and metadata.

on the roadmapFile Format & Archivesfree · no sign-up

This bench is being wired up

The parquet-avro-decoder decoder is on the roadmap. The heavier binary and cryptographic parsers ship after the core text and token decoders — all of which are already live and working.

Browse the live decoders

Input is processed on your device. Nothing you paste is sent anywhere.

What is the Parquet / Avro Header Decoder?

A Parquet/Avro decoder reads the metadata sections of these two big-data file formats — Parquet's footer schema and row-group statistics, Avro's JSON schema header — to reveal a file's structure without scanning its data.

Parquet is columnar: values from the same column are stored together, which compresses far better and lets a query engine read only the columns it needs. The schema and per-row-group statistics live in a footer at the end of the file, immediately before the PAR1 magic, so metadata is read with a single seek to the tail.

Avro is row-oriented and self-describing in a different way: the writer's schema is embedded as JSON in the file header, followed by blocks of records separated by a sync marker. That design makes Avro good for streaming and for schema evolution, where readers reconcile their expected schema against the writer's.

In practice you reach for the metadata when a pipeline breaks. Parquet's row-group min/max statistics explain why predicate pushdown did or didn't skip data, and Avro's embedded schema explains why a consumer can't read what a producer wrote.

Parquet / Avro Header Decoder at a glance

Parquet magic
PAR1 at both start and end of file
Parquet layout
Columnar, row groups, footer holds the schema
Avro header
Obj1 magic plus a JSON schema
Avro layout
Row-oriented blocks with a 16-byte sync marker

How to use the Parquet / Avro Header Decoder

  1. 1Load the file — only the header and footer regions need to be read.
  2. 2Review the recovered schema: field names, types and nesting.
  3. 3For Parquet, check row-group counts and column statistics.

When you would reach for it

  • Confirming the schema of a Parquet file before writing a query against it.
  • Diagnosing an Avro schema-evolution mismatch between producer and consumer.
  • Checking row-group sizing when a Parquet dataset queries slowly.

Parquet / Avro Header Decoder — frequently asked questions

Last reviewed · questions or corrections: contact@decoder.tools