Design a Multi-Tenant Data Analysis Platform

Problem Overview

Design a multi-tenant data analysis platform, in the spirit of Palantir's own Foundry: tenants ingest datasets, run transforms that derive new datasets from them, and analysts query the results. The users are government and enterprise customers, which changes what the interview is about.

This is not the throughput-and-sharding design you may have drilled elsewhere. The scoping is explicit and it reshapes the whole conversation:

• Scale is not the point. You are not asked to serve a billion QPS. Say this out loud early so you do not spend your interview on Kafka partitioning that no one asked for.
• The permission model is the hard part. Government and enterprise tenants have deeply layered access rules: classification levels, need-to-know, per-column and per-row restrictions, and cross-tenant isolation that must never leak.
• Compliance, audit logs, and data lineage are first-class features, not afterthoughts you bolt on at the end. They are the product.
• The interviewer assumes you will get things wrong. The refrain to design against is "Assume mistakes will happen. How do we detect and recover?" A bad permission grant, a broken transform, a leaked dataset: your design has to make each of those detectable and reversible.

This walkthrough follows the Interview Framework and is sized for a 45 to 60 minute round.

Phase 1: Requirements

Functional Requirements

1. Tenant workspaces. Each tenant (a government agency or enterprise) works inside an isolated project space. A user in one tenant must never see another tenant's data or metadata.
2. Datasets and transforms. Tenants ingest source datasets and run transforms that read one or more datasets and produce a derived dataset. This forms a directed acyclic graph of data.
3. Fine-grained permissions. Access is controlled down to the dataset, column, and row level, layered on top of classification markings (for example, a security label a user must be cleared for). Permission is enforced at query time, not just at the storage boundary.
4. Audit logging. Every access and every mutation, who read what, who changed a permission, who ran which transform, is recorded in a tamper-evident, append-only log.
5. Data lineage. For any dataset you can walk upstream to its sources and downstream to everything derived from it, including which transform and which transform version produced each edge.
6. Detection and recovery. When something goes wrong, a wrong grant, a bad transform output, a dataset shared too widely, you can detect the blast radius through lineage and audit, and recover by rolling back to a prior dataset version, revoking access, and replaying transforms.

Non-Functional Requirements

The priorities here are unusual, so name them deliberately:

• Correctness and isolation over throughput. A permission check that is fast but occasionally wrong is a failure. Cross-tenant isolation is the one property that can never bend.
• Auditability. The audit log must be complete and tamper-evident. If an operator cannot prove who saw a record, the compliance story collapses.
• Recoverability. Datasets are versioned and immutable, so any state the platform was ever in can be reconstructed and any bad change can be undone.
• Lineage completeness. Lineage cannot be best-effort. If an edge is missing, the blast-radius analysis that recovery depends on is wrong.
• Availability. Standard enterprise SaaS availability is fine. There is no consumer-scale fleet hammering the system.

Capacity, Briefly

Keep this to a sentence or two, only to justify a simple shape. Assume thousands of tenants, each with thousands to millions of rows per dataset and a modest number of analysts. Metadata (datasets, versions, permissions, lineage edges) fits comfortably in a relational database. Dataset bytes live in object storage. Query volume is human-driven and small. The design effort goes into the control plane, not the data plane.

Phase 2: Data Model

Core Entities

Tenant(
  tenant_id UUID PK,
  name TEXT
)

Principal(                          -- a user or a service account
  principal_id UUID PK,
  tenant_id UUID FK,
  type ENUM('user','service'),
  email TEXT
)

Group(                              -- roles / need-to-know groups
  group_id UUID PK,
  tenant_id UUID FK,
  name TEXT
)

GroupMembership(                    -- which principals belong to which groups
  group_id UUID FK,
  principal_id UUID FK
)

Dataset(
  dataset_id UUID PK,
  tenant_id UUID FK,
  name TEXT,
  current_version_id UUID,          -- pointer to the latest immutable version
  created_by UUID
)

DatasetVersion(                     -- immutable snapshot; datasets are append-only in time
  version_id UUID PK,
  dataset_id UUID FK,
  version_number INT,
  storage_uri TEXT,                 -- object-storage location of this snapshot
  markings UUID[],                  -- dataset-level classification labels
  column_schema JSONB,              -- column definitions and the markings on each column
  produced_by_transform_run UUID,   -- NULL for a raw ingested version
  content_hash CHAR(64),            -- integrity + dedup
  created_at TIMESTAMP
)

Transform(
  transform_id UUID PK,
  tenant_id UUID FK,
  name TEXT,
  code_ref TEXT,                    -- versioned reference to the transform logic
  code_version TEXT
)

TransformRun(
  run_id UUID PK,
  transform_id UUID FK,
  code_version TEXT,
  input_version_ids UUID[],         -- exact versions consumed
  output_version_id UUID,           -- exact version produced
  status ENUM('running','succeeded','failed'),
  started_at TIMESTAMP,
  finished_at TIMESTAMP
)

LineageEdge(                        -- explicit graph: which version derived from which
  edge_id UUID PK,
  upstream_version_id UUID FK,
  downstream_version_id UUID FK,
  transform_run_id UUID FK
)

Marking(                            -- a classification label, e.g. a clearance level or handling caveat
  marking_id UUID PK,
  tenant_id UUID FK,
  name TEXT
)

MarkingAssignment(                  -- which principals or groups are cleared for which markings
  marking_id UUID FK,
  principal_type ENUM(,),
  principal_id UUID
)

AccessGrant(                        
  grant_id UUID PK,
  tenant_id UUID FK,
  dataset_id UUID FK,
  principal_type ENUM(,),
  principal_id UUID,
  action ENUM(,,,),
  allowed_columns TEXT[] ,      
  row_predicate TEXT ,          
  granted_by UUID,
  created_at ,
  expires_at  
)

AuditEvent(                         
  event_id UUID PK,
  tenant_id UUID FK,
  actor_principal_id UUID,
  action TEXT,                      
  resource_type TEXT,
  resource_id UUID,
  resource_version_id UUID ,
  decision ENUM(,),
  context JSONB,                    
  prev_hash (),               
  event_hash (),
  created_at 
)

The Decisions That Matter

Datasets are immutable and versioned. A transform never overwrites a dataset in place. It produces a new DatasetVersion, and the Dataset row simply advances its pointer. This is the backbone of recovery: rolling back a bad transform is repointing to the previous version, and the old state is never lost. It is also what makes lineage exact rather than approximate, because every edge references specific versions.

Lineage is a first-class graph, captured at execution time. Every TransformRun records the exact input versions it read and the output version it wrote, and materializes LineageEdge rows. You do not reconstruct lineage by parsing code later; you record it as a byproduct of running the transform, which is the only way to keep it complete. [Source: darkinterview.com]

Permissions and markings are two orthogonal layers. An AccessGrant is an ACL entry: it lets a principal (a user or a group) perform an action on a dataset, optionally narrowed to an allowed_columns set and a row_predicate. Markings are a separate, data-carried classification: columns and whole datasets are labeled, principals are cleared for those labels through MarkingAssignment, and you may only see data whose markings you hold. Access is the intersection of the two. This split is how you model government-style need-to-know: a grant can let the analytics group read a dataset, while markings still filter each analyst to the rows and columns their clearance permits. Neither layer alone is sufficient.

Phase 3: API Design

REST over HTTPS fits the control plane cleanly; this is metadata management plus a policy-checked query path. Dataset bytes are read through the query engine, which applies permissions, not fetched directly from storage by clients. [Source: darkinterview.com]

Datasets and Transforms

POST /v1/datasets                          -- register a dataset, ingest first version
POST /v1/transforms/{id}:run               -- run a transform; records a TransformRun + lineage
GET  /v1/datasets/{id}/versions            -- list immutable versions

Query (permission-enforced)

POST /v1/datasets/{id}:query
Body: { "columns": ["region","amount"], "filter": "amount > 1000", "version_id": "..." }

The query service resolves the caller's effective permission, injects the row predicate and column mask from any matching grant, checks the caller's markings against the version's column markings, and only then reads bytes. Every call emits an audit event with the columns actually touched.

Permissions

POST   /v1/grants                          -- create an AccessGrant
DELETE /v1/grants/{id}                     -- revoke; takes effect on the next check
GET    /v1/datasets/{id}/access-review     -- who can currently read this, and via which grant

Lineage

GET /v1/versions/{id}/lineage?direction=downstream&depth=all
Response: { "nodes": [ ...dataset versions... ], "edges": [ ...transform runs... ] }

Audit and Recovery

GET  /v1/audit?resource_id=...&actor=...&start=...&end=...     -- query the append-only log
POST /v1/datasets/{id}:rollback                                -- repoint to a prior version
POST /v1/transforms/{id}:replay                                -- re-run from a chosen input version

Phase 4: High-Level Design

Architecture

The system splits into a control plane (what is allowed, where data came from, what happened) and a thin data plane (running transforms, serving reads). The interesting logic is almost entirely in the control plane. [Source: darkinterview.com]

Read Flow (walk this carefully)

1. An analyst queries a dataset version through the API gateway, which authenticates them.
2. The query service asks the policy engine for a decision: given this principal, their groups, and their markings, what can they read on this version? The engine returns an allow/deny plus the row predicate and column mask to apply.
3. The query service reads the immutable version from object storage, applies the predicate and mask so the analyst sees only permitted rows and columns, and returns the result.
4. It emits an audit event recording the actor, the version, the exact columns touched, and the decision. The event is chained to the previous one by hash.

Write Flow (transforms)

1. A transform run reads specific input versions, executes its logic, and writes a new output version to object storage. It never mutates an input.
2. It records a TransformRun and the LineageEdge rows connecting input versions to the output version.
3. It emits an audit event. The dataset pointer advances to the new version only after the run succeeds.

Policy Engine (the core)

Answering "can principal P read version V, and what may they see" means:

1. Build P's identity: their principal_id, the groups they belong to (GroupMembership), and the markings they hold (MarkingAssignment).
2. Grant check. Collect unexpired AccessGrant rows on V's dataset with the read action that match P or any of P's groups. No matching grant means deny.
3. Marking check. If the dataset carries a marking P does not hold, deny the read outright; any individual column whose marking P lacks is filtered out of the result.
4. Combine. The effective column set is the union of the matching grants' allowed_columns, then restricted to columns whose markings P holds; the effective row filter is the disjunction of their row_predicates. Read V, apply both, and return only what remains.

Cache decisions per (principal, version, query shape) and invalidate on any grant or marking change touching that principal or resource. Because grants and markings are tenant-scoped, invalidation stays local to one tenant.

Phase 5: Compliance, Detection, and Recovery

This is where the interview actually lives. Reframe the usual "scaling" phase around the interviewer's refrain: mistakes will happen, so detect and recover.

Tamper-Evident Audit

The audit log is append-only and each event stores the hash of the previous event, forming a hash chain per tenant. Periodically the head hash is anchored somewhere write-once (a separate append-only store or a signed checkpoint). To verify integrity, recompute the chain and compare against the anchors. Any deletion or edit of a past event breaks the chain from that point forward and is detectable.

• Writes acknowledge only after the event is durably appended.
• Operators can read the log but cannot silently rewrite it; an attempt shows up as a chain break.
• The log records denials as well as allows, so a burst of denied cross-tenant reads is itself a detectable signal.

Data Lineage as the Detection Tool

Lineage is not documentation; it is how you compute a blast radius. When something goes wrong with a dataset version, walk the lineage graph downstream to find every version derived from it, transitively, along with the transform runs on each edge. That set is exactly what a mistake could have contaminated.

• Bad source or bad transform: downstream lineage tells you every derived dataset that must be recomputed or quarantined.
• Improper access: the audit log tells you who read a version; lineage tells you which other versions expose the same underlying data, so a leak's reach is the union of both.

Detect and Recover: the Failure Playbook

Have a crisp answer for each mistake the interviewer will raise:

Because every dataset version is immutable, recovery almost never means reconstructing lost data. It means repointing to a good version and, where access was the problem, revoking and measuring exposure. That is the property immutability buys you, and it is worth saying plainly. [Source: darkinterview.com]

Permissions at Enterprise Complexity

Government and enterprise access rules layer several mechanisms, and a strong answer keeps them distinct:

• Role/group grants for coarse access ("the analytics group can read this dataset").
• Row-level predicates for data-dependent scoping ("only rows where region is EU").
• Column-level scoping through a grant's allowed-column set ("read everything except the SSN column").
• Classification markings as an orthogonal layer: data carries markings, principals are cleared for them, and even with a grant you only see data whose markings you hold.

Deny wins on conflict, unmatched means denied, and grants expire. Column markings travel with the schema through transforms, so a derived dataset does not silently launder a restricted column into an unmarked one; propagating markings along lineage edges is what prevents that. [Source: darkinterview.com]

On Scale (kept short on purpose)

If pushed, give the growth path in one breath and move on: the metadata database can take read replicas and, far later, partition by tenant_id since tenants never query across each other; object storage absorbs dataset bytes natively; the audit store is append-only and partitions by tenant and time. None of this is the focus, and saying so is the right instinct here.

Common Pitfalls

Interview Checklist

Requirements

• Said out loud that scale is not the focus; permissions, audit, and lineage are
• Named correctness, isolation, auditability, and recoverability as the binding NFRs
• Framed the whole design around "mistakes happen: detect and recover"

Data Model

• Immutable, versioned datasets with a pointer to the current version
• Lineage as explicit edges tied to exact input/output versions and transform runs
• Two orthogonal permission layers: ACL grants (role, row predicate, allowed columns) and data markings the principal must be cleared for
• Every row tenant-scoped, including grants, lineage, and audit

API Design [Source: darkinterview.com]

• Permission-enforced query path that applies row/column masking
• Lineage traversal, access review, audit query, rollback, and replay endpoints

High-Level Design

• Policy engine as a distinct decision point; query service as the enforcement point
• Read flow emits an audit event with the columns actually touched
• Transforms write new versions and record lineage as a byproduct

Detection & Recovery

• Tamper-evident audit via a per-tenant hash chain plus anchoring
• Lineage used to compute blast radius downstream of a bad version
• A concrete recover playbook per failure: rollback, revoke, replay, measure exposure
• Marking propagation along lineage so derived datasets do not launder sensitive columns

Summary