Apache Parquet

What it is

Apache Parquet is a file format for tabular data. A Parquet file holds records, arranged by column, compressed, with metadata describing what is inside. It is designed for one access pattern: read some columns from many records, filtered, as fast as possible, from storage that charges for bytes read.

It has become the default for analytical data in the same quiet way that JPEG became the default for photographs. Nobody argues about it much. Engines read it, warehouses export it, catalogs point at it, and object storage is full of it.

For an open agentic platform, Parquet matters because it is the level at which data stops being tied to a product. A table stored as Parquet in object storage can be read by any engine that implements the format, which is most of them. That property is what makes the layers above replaceable.

The problem it solves

Consider a table of one hundred million events with sixty columns, and a question that needs three of those columns for events from one day.

Stored as rows, whether in CSV, JSON, or a row-oriented binary format, answering that question requires reading essentially the whole file. Every record must be examined to check its date, and reading a record means reading all sixty of its fields. Compression helps with size and not with the fundamental problem: the bytes you need are interleaved with the bytes you do not.

Stored as Parquet, the same question reads three columns rather than sixty, and within those columns skips every block whose recorded value range cannot contain the target date. It is common for this to reduce bytes read by two orders of magnitude. On object storage, where you pay per byte and per request, that is a direct cost reduction as well as a speed one.

Anatomy of a Parquet file

The structure is worth knowing because almost every performance question about Parquet is answered by it.

A file is divided into row groups. A row group is a horizontal slice: some number of complete records, typically tens or hundreds of megabytes worth. Row groups are the unit of parallelism, because two readers can process two row groups independently.

Within a row group, each column is stored as a column chunk. All values of that column, for the records in that row group, sit contiguously. This is where the columnar benefit comes from.

Each column chunk is divided into pages, which are the unit of compression and encoding. A page is typically around a megabyte before compression.

At the end of the file sits the footer, holding the schema, the location of every column chunk, and statistics: minimum and maximum values, null counts, and distinct counts where available. The footer is read first, which is why a reader can decide what to fetch before fetching anything substantial.

That last detail is the one that matters most on object storage. A reader issues a small request for the footer, works out exactly which byte ranges it needs, and then issues requests for only those ranges. The file can be a gigabyte and the read can touch ten megabytes.

How skipping actually works

Parquet supports skipping at several levels, and understanding which level is doing the work explains why two superficially similar tables perform very differently.

Column pruning

The simplest and most reliable. If a query needs three columns, the reader fetches three column chunks and ignores the rest. This works always, requires no tuning, and is why selecting only the columns you need is good advice rather than pedantry.

Row group skipping

Each row group records minimum and maximum values per column. If a filter asks for a date in March and a row group's date range is entirely in January, the reader skips it without reading any data. This is effective in proportion to how well the data is ordered. If records are sorted by date, most row groups can be skipped. If dates are randomly distributed, every row group spans the whole range and nothing is skipped.

Page level skipping

Newer readers and writers support page-level statistics and indexes, which apply the same idea at finer granularity. The benefit again depends on ordering.

Bloom filters

For high-cardinality equality filters, such as looking up a specific identifier, minimum and maximum statistics are useless because nearly every row group's range contains the value. Bloom filters answer the question of whether a specific value might be present, and are worth enabling on columns that are frequently looked up by exact value.

Encoding and compression

Parquet applies two layers of size reduction, and they are frequently confused.

Encoding exploits structure in the values themselves. Dictionary encoding replaces repeated values with small integers pointing at a dictionary, which is enormously effective for low-cardinality string columns such as country or status. Run-length encoding collapses repeated consecutive values. Delta encoding stores differences rather than absolute values, which suits sorted numbers and timestamps.

Compression then applies a general-purpose algorithm to the encoded pages. The common choices are Snappy, which is fast and moderately effective, and Zstandard, which compresses better at some cost in speed and is now often the better default because storage and network usually dominate.

Because encoding happens first, the compression algorithm sees data that is already highly regular, which is why Parquet files are so much smaller than the same data compressed as CSV. It also means encoding choice matters more than compression choice for well-structured columns.

Schema and evolution

Every Parquet file carries its own schema in the footer: column names, types, nesting, and whether each field is optional or required. This makes files self-describing, which is why a file written five years ago is still readable without external information.

Parquet supports nested structures directly, including lists and maps, using a definition and repetition scheme that encodes nesting without storing the structure per record. This is genuinely clever and mostly invisible in practice.

Where Parquet stops is evolution across files. A file has one schema, fixed at write time. If the table gains a column, new files have it and old files do not. If a column is renamed, nothing connects the old name to the new one, because names are the only identity Parquet has.

This is precisely the gap table formats fill. Iceberg assigns stable identifiers to columns and maps them onto whatever the underlying files call them, which is what makes safe renaming and reordering possible over a collection of files written across years. Parquet on its own is a file format, and schema evolution is a table concern.

Writing Parquet well

Reading Parquet requires no expertise. Writing it well requires a handful of decisions that are easy to get wrong and expensive to fix later.

• File size. Aim for files in the hundreds of megabytes. Much smaller and metadata overhead dominates. Much larger and parallelism suffers, and a single bad file becomes an expensive rewrite.
• Row group size. Typically one hundred to five hundred megabytes. Too small wastes the statistics mechanism; too large reduces skipping precision.
• Sort by the primary filter column. Usually a timestamp or a tenant identifier. This is the single most valuable write-time choice.
• Use Zstandard unless you have measured otherwise. The size difference is usually worth the additional CPU.
• Keep dictionary encoding on for low-cardinality strings. Most writers do this automatically until a dictionary grows past a threshold, at which point they fall back, sometimes silently.
• Enable bloom filters on identifier columns you look up by value. They cost a little space and remove a class of full scans.
• Choose types deliberately. Storing numbers as strings, or timestamps as strings, gives up encoding, statistics, and skipping all at once. This is the most common avoidable mistake in the format.

Partitioning and layout

How files are arranged in storage matters nearly as much as what is inside them, and the two decisions interact.

The traditional approach is directory partitioning: writing files into paths that encode a column value, such as a date directory per day. A reader that understands the convention can eliminate whole directories before listing them. This works, and it carries three well-known problems.

The first is that the partition column becomes part of the physical layout, so changing the partitioning scheme means rewriting the table. The second is that queries must filter on the partition column in exactly the form the layout expects, so a filter on a timestamp does not eliminate date directories unless the query engine is clever enough to derive one from the other. The third is skew: partitioning by a column with uneven distribution produces one directory with ten files and another with ten thousand.

Choosing partition granularity is the recurring judgment call. Too coarse and every query scans too much. Too fine and you manufacture the small files problem deliberately. A reasonable heuristic is to target partitions that hold at least a few hundred megabytes, and to partition on the column that appears in nearly every query rather than the one with the most distinct values.

Within a partition, sort order does the remaining work. Partitioning eliminates directories; sorting and statistics eliminate row groups inside the surviving files. Teams often tune one and ignore the other, then conclude the format is slow. Both together are what produce the order-of-magnitude differences people associate with well-run lakehouse tables.

Table formats improve on all of this by tracking partition values in metadata rather than in paths, which decouples the logical partitioning from the physical layout and allows it to change without a rewrite. That is a table-format feature, not a Parquet one, and it is one of the clearest arguments for not managing Parquet directories by hand.

Where it matters for agents

Agents interact with Parquet through queries, never directly, but three of its properties shape agentic behavior noticeably.

Exploratory queries are affordable

Agents explore. They check what a column contains, count distinct values, sample rows, and look at a related table before committing to an approach. On a row format each of those is a full scan. On well-written Parquet most are cheap. The difference determines whether exploratory behavior is a feature or a cost problem.

Wide tables stop being a penalty

Analytical tables often have many columns, most irrelevant to any given question. Because column pruning is automatic, an agent asking about three fields does not pay for the other fifty-seven. This makes wide, descriptive tables a reasonable design for agent consumption, where they would have been a problem in a row-oriented world.

Statistics answer some questions without reading data

A question such as what date range this table covers, or how many nulls are in this column, can sometimes be answered from footer metadata alone. Tools that expose this cheaply give agents a fast way to orient before issuing an expensive query.

Cost is predictable enough to bound

Because bytes read is largely a function of columns selected and partitions touched, a tool can estimate the cost of a query before running it and refuse or warn when an agent proposes something unbounded. That check is far easier to build over a columnar format with statistics than over an opaque store, and it is one of the more effective guardrails available for agent data access.

The small files problem

This is the failure mode that catches almost every team eventually, and it is worth understanding before it happens rather than after.

Streaming or frequent batch writes produce many small files. A pipeline writing every minute produces 1,440 files a day and half a million a year. Each file has a footer to read, a request to issue, and metadata to track. Query planning slows down, then query execution slows down, then the catalog itself slows down.

The symptom is distinctive: a table that is not especially large becomes slow to query, and the slowness is in planning rather than in scanning. Listing the files takes longer than reading them.

The fix is compaction: periodically rewriting many small files into fewer large ones, ideally sorted while you are at it. Table formats provide this as a maintenance operation, which is one of several reasons why Parquet in a table format is a different proposition from Parquet in a directory. Directories of Parquet files have no mechanism for compacting safely while readers are active. Table formats do.

Gotchas worth knowing

• Files are immutable. Updating a single record means rewriting the file that contains it. Parquet has no in-place update. Anything resembling a mutation is provided by the layer above.
• Statistics can be missing or wrong. Some writers omit them, and some historically wrote incorrect statistics for certain types, which readers then defensively ignore. If skipping is not happening, checking whether statistics exist is the first diagnostic.
• Timestamp semantics vary. Precision and time zone interpretation have changed across format versions and differ between writers. This is the most common source of values that are correct in one engine and off by hours in another.
• Decimal handling differs between implementations. Worth testing rather than assuming, especially for financial data.
• Very wide tables have real metadata cost. Thousands of columns means a large footer, read on every access.
• Compression is per page, not per file. A file is not one compressed blob, which is what allows partial reads and why you cannot judge read cost from file size alone.

Alternatives and when they apply

Parquet is not the only columnar format, and there are cases where something else fits better.

ORC solves the same problem with a different design and remains common in Hadoop-derived environments. Choose it when the surrounding ecosystem already uses it, not on technical grounds alone.

Arrow IPC files store Arrow buffers directly. They are fast to read because no decoding is required and much larger because nothing is compressed. Reasonable for short-lived intermediates, not for storage.

Row formats such as Avro remain the right choice for streaming and for record-at-a-time processing, where you want whole records and write throughput matters more than scan efficiency. Many pipelines use Avro on the way in and Parquet at rest, which is a sensible division rather than a compromise.

Plain CSV and JSON remain the right answer for interchange with systems and people that need something universally readable, and the wrong answer for anything an agent will query repeatedly.

How it scores on openness

• Replaceable. Strong. Many independent implementations, and no single vendor controls the format.
• Inspectable. Strong. The specification is public and tooling exists to dump metadata and inspect content.
• Portable. Strong. A file is self-describing and readable anywhere, indefinitely.
• Bounded. Not applicable. Files carry no access model; permissions live in the catalog and the storage system.
• Grounded. Partial. Files carry names and types but no meaning. Semantics belong further up.
• Auditable. Partial. Files do not record who read them, though they do preserve exactly what was written.

What it is not

Parquet is not a table. A directory of Parquet files has no atomic commits, no snapshot history, and no safe concurrent writes. Two writers to the same directory can produce a state readers observe as partially updated. Table formats exist to add exactly these properties.

Parquet is not a database. There is no index in the ordinary sense, no transaction, and no query engine. It is a file that engines read.

Parquet is not an in-memory format. Data must be decoded before it can be computed on, and the decoded form is usually Arrow. Treating them as competitors leads to choosing between two things that are meant to be used together.