Apache Polaris

What it is

Apache Polaris is an open-source catalog for Apache Iceberg tables. It implements the Iceberg REST catalog protocol, which means any engine or client that speaks that protocol can use it without a Polaris-specific driver.

Its job is to answer questions about tables rather than to store data. Which tables exist. Where is the current metadata for this one. May this caller read it. Here are temporary credentials scoped to exactly that. Commit this new version of the table, but only if the version I started from is still current.

Described that way it sounds like plumbing, and it is. It is also the single most important governance decision in the data layer, because every read and write passes through it.

What a catalog actually does

Catalog is an overloaded word. In the lakehouse sense it has a specific, small set of responsibilities.

Naming

It maps a human-usable name to a physical location. A query referring to a table by name gets resolved to the current metadata file, wherever that lives. Without this, every query would need a storage path, and moving data would break every consumer.

Atomic commit

It performs the compare-and-swap that makes a table write atomic. This is the operation that makes concurrent writers safe, and it is the reason a catalog cannot be a passive index. It must be able to serialize conflicting updates.

Authorization

It decides whether a caller may perform an operation before performing it. This is where read and write permission actually lives in a lakehouse, rather than in the query engine, which is easy to bypass, or in storage permissions alone, which are too coarse.

Credential handling

It provides the caller with credentials to reach the underlying storage, scoped and short-lived, rather than expecting the caller to hold long-lived storage keys.

Discovery

It lists what exists so that consumers, including agents, can find out what is available without being told in advance.

Why the catalog is the control point

There are three plausible places to enforce access to lakehouse data, and only one of them works well.

In the query engine. Attractive because engines already understand tables and columns. Weak because anything that can read the storage directly bypasses it, and because a second engine means a second copy of the rules that will drift from the first.

In storage permissions. Attractive because it cannot be bypassed. Weak because storage permissions operate on paths and prefixes rather than tables, columns, and rows, so expressing anything nuanced requires contorting the physical layout to match the access model.

In the catalog. Every reader must consult the catalog to find the current metadata, so it sits in the path by construction. It understands tables rather than paths. And because it can vend credentials rather than assume the caller has them, it can be the only holder of the underlying storage keys.

That last property is what turns the catalog from a convenient place to enforce policy into the correct place. If callers cannot reach storage without the catalog, then catalog policy is not advisory.

Credential vending

This deserves its own treatment because it is the mechanism that makes the rest work, and because it is often turned off during setup and never turned back on.

Without credential vending, a client that wants to read a table asks the catalog for the metadata location and then uses its own storage credentials to fetch the files. Those credentials are typically long-lived, broad, and stored in configuration. Every tool that reads data has a copy. Revocation means finding every copy.

With credential vending, the client authenticates to the catalog, the catalog authorizes the operation, and the catalog returns temporary credentials scoped to just the storage locations that table occupies, valid for a short window. The client never holds general storage access.

The properties this produces are worth listing plainly, because they map directly onto what an agentic system needs:

• One place to revoke. Removing a grant in the catalog takes effect as soon as outstanding credentials expire.
• Blast radius bounded by time. A leaked credential is useful for minutes rather than indefinitely.
• Blast radius bounded by scope. A credential for one table cannot read another.
• One place to observe. Every access request is a catalog request, so usage is visible without instrumenting every client.
• No secrets in agent configuration. Which removes an entire category of exposure, since agent configuration tends to be copied, shared, and occasionally committed.

There is one honest limitation to note. Vended credentials are usually scoped at the storage prefix level, which means they are naturally table-shaped rather than column-shaped or row-shaped. Finer-grained controls, such as masking a column or restricting rows to a region, are enforced by the engine or by exposing a restricted view as its own table. An architecture that needs row-level restriction for agents should design for it explicitly rather than assuming the catalog delivers it, and the view-as-a-table approach has the advantage that it also works for clients the engine does not mediate.

Access control model

Polaris organizes access around catalogs, namespaces, and tables, with roles granting privileges over those objects. Principals are assigned roles, and roles carry privileges such as reading a table, writing to it, or managing a namespace.

Two design points matter more than the specifics.

The first is that privileges are hierarchical. A grant at namespace level covers the tables within it, which keeps the number of individual grants manageable as table counts grow. Systems that require a grant per table become unmanageable at a few hundred tables and then get worked around with a wildcard grant that defeats the purpose.

The second is that the principal is a first-class concept, distinct from a human user. An agent, a service, or a pipeline can be its own principal with its own roles. This is what makes it possible to say precisely what one agent may reach, and to answer questions about its access without inference. When agent access is instead implemented by giving the agent a human's credentials, both properties disappear at once: the agent inherits everything that person can do, and the record shows the person rather than the agent.

Federation and multiple engines

Most organizations do not have one query engine. They have the one the data team uses, the one that arrived with a product, and the one someone adopted for a specific workload. Multi-engine access is the normal case rather than an advanced scenario.

Because Polaris implements the Iceberg REST protocol, engines that support that protocol connect without engine-specific catalog work. That list includes several widely used open engines, and the specifics are worth checking against current documentation rather than a summary, since support matures continuously.

The architectural consequence is the one that matters for this site: the catalog stops being a property of an engine and becomes a shared service. Governance is expressed once and observed by everything, rather than expressed per engine and enforced inconsistently.

What a request looks like end to end

Walking one read through the system makes the division of labor concrete, and it explains why several of the gotchas later on are gotchas.

1. The client authenticates. An engine, notebook, or agent tool presents a credential to the catalog and receives a token. The identity established here is the identity everything else is judged against, which is why a shared agent account collapses the whole chain into one indistinguishable actor.
2. The client asks to load a table. It names a namespace and table rather than a storage path.
3. The catalog authorizes. It checks whether this principal holds a role granting read on that object, directly or through the namespace above it. A denial happens here, before anything is read.
4. The catalog returns metadata and credentials. The current metadata file location, plus temporary credentials scoped to the storage prefix that table occupies.
5. The client reads metadata and plans. It walks the manifest structure, eliminates files that cannot match the filter, and produces a list of byte ranges to fetch.
6. The client reads data directly from storage. Using the vended credentials. Data does not pass through the catalog, which is why the catalog does not become a throughput bottleneck.

Two things follow from step six. The catalog is on the control path and not the data path, so it needs to be highly available and low latency but not high bandwidth. And the credentials handed out in step four are the real enforcement, which is why disabling vending quietly removes the enforcement while leaving the authorization check in place as a formality that anything can route around.

Federation has a second meaning worth separating from multi-engine access. Some catalogs can present tables that live in another catalog, so a single endpoint exposes a wider set than it owns. This is useful when an organization has accumulated several catalogs and wants one place for consumers to look, and it is worth being clear-eyed about: federation makes discovery easier and does not merge the governance models underneath. Permissions are still enforced by whichever catalog owns the table, so a federated view can list things a caller cannot actually read. For agents that is acceptable, and only if the listing is filtered by what the caller can reach, since an agent that repeatedly discovers tables it cannot open wastes steps and reports confusing failures.

Project status and governance

Polaris entered the Apache Incubator after being contributed as open source, and graduated to a top-level Apache project in early 2026. Graduation is a meaningful signal rather than a formality: it indicates the Apache Software Foundation judged the project to have a sufficiently diverse community and a working governance process to sustain itself independently of any single contributing company.

For an architecture that intends to remain open, that governance question is as important as the feature list. A catalog is a long-lived dependency sitting at the centre of the data layer. Whether it is stewarded by a foundation with a public process, or by a company that could change direction, is a legitimate part of the evaluation.

Where it matters for agents

The natural boundary for agent access

Agents should not hold storage credentials. Routing their data access through a catalog that authenticates them, authorizes the operation, and vends short-lived credentials turns a diffuse security problem into a single administrative surface.

Discovery without a hardcoded list

An agent asking what tables it can see gets a list filtered by its own permissions. This is better than maintaining a list in a prompt, which drifts, and better than exposing everything, which is unsafe. The catalog is already the authority, so the answer is current by construction.

Access records that mean something

Because every access passes through the catalog, catalog logs are a record of what agents actually read. This is a substantial part of what makes the auditable property achievable in the data layer.

Per-agent scoping

Different agents can be different principals with different grants. A customer-facing agent and an internal analysis agent need very different access, and expressing that in the catalog is far more reliable than expressing it in instructions.

Delegated user identity

The stronger pattern, where the platform supports it, is for an agent acting on behalf of a person to receive access scoped to that person rather than to a general agent account. Then an agent cannot become a way to see data the requester could not see directly, which is one of the more common quiet governance failures in agentic deployments.

Deployment considerations

A catalog is on the critical path for every query, which makes a few operational properties non-negotiable.

• Availability. If the catalog is down, no table can be read. It needs the availability of a core service, not of an internal tool.
• Latency. Every query starts with catalog calls. Placing it far from the engines that use it adds a fixed cost to everything.
• Durable backing store. The catalog holds the pointer that defines table state. Its persistence layer needs backup and recovery treated seriously.
• Identity integration. Connecting it to existing identity infrastructure is what makes per-user and per-agent principals practical rather than an administrative burden.
• Audit retention. Catalog logs are the access record. Retaining them for as long as you might need to answer a question about past access is a policy decision worth making explicitly.

Adopting it into an existing setup

Very few teams start here. The usual situation is tables already existing under some other arrangement, with credentials already distributed, and a working setup nobody wants to break. A staged path works better than a migration project.

1. Stand it up alongside, read-only. Register existing tables and point one non-critical consumer at the catalog. This validates protocol support in your engines before anything depends on it.
2. Model principals before grants. Decide what the identities are: which services, which pipelines, which agents, and whether agents act as themselves or on behalf of users. Grants written before this question is settled tend to be rewritten.
3. Turn on credential vending for new consumers first. New tools should never receive direct storage credentials, even while old ones still have them. This stops the problem growing while you shrink it.
4. Move agent access first among existing consumers. Agents are the newest, least entangled, and highest-risk consumers, which makes them both the easiest to move and the most valuable to move.
5. Retire direct storage credentials one consumer at a time. Each retirement is small. The list is what takes time, which is the argument for not letting it grow further.
6. Then narrow the grants. Start with roughly what people already had, then reduce based on what the access records show is actually used. Narrowing against evidence is achievable; narrowing against speculation produces a stream of access requests and eventually a wildcard grant.

Gotchas worth knowing

• Credential vending disabled during setup. The most common configuration mistake. It is easier to get things working with direct storage credentials, and the temporary arrangement becomes permanent.
• One shared agent principal. Convenient and it collapses every access record into one identity, removing the ability to attribute anything.
• Catalog treated as a metadata search product. Business glossaries and data discovery tools are also called catalogs. This is a different thing. Both can be useful, and only one is in the read path.
• Permissions granted broadly to unblock work. Understandable and, once done, rarely revisited. Broad grants tend to outlive the reason for them.
• Protocol support assumed rather than verified. Clients vary in which parts of the REST specification they implement, particularly around writes and newer operations. Verify the operations you need.

How it scores on openness

• Replaceable. Strong. Because it implements a published protocol, another conformant catalog can take its place, and clients do not change.
• Inspectable. Strong. Open source, with an openly specified protocol.
• Portable. Good. Table metadata and data are portable already; catalog state such as grants requires deliberate migration.
• Bounded. Strong. This is the component whose entire purpose is expressing and enforcing boundaries.
• Grounded. Partial. It knows tables and schemas, not business meaning.
• Auditable. Strong for access. Catalog logs record who asked for what, which is the hardest part to reconstruct otherwise.

What it is not

Polaris is not a query engine. It does not read data or run SQL. It tells engines where tables are and whether they may be read.

Polaris is not a semantic layer. It carries table and column metadata, and it does not define what net revenue means or which of three similar tables is authoritative. That work sits above it.

Polaris is not a general-purpose permission system for an organization. It governs lakehouse tables. Application permissions, document access, and everything else belong elsewhere, and an agent operating across several domains will be subject to several such systems rather than one.