Staging ingest — files to governed rows

Teams keep living data in spreadsheets: project exports, fee schedules, hours reports. The staging ingest layer turns those files into governed rows your apps can trust — every file deduplicated, every row validated against a confirmed template, every decision audited, and every load accounted for.

Turning it on

Staging is approved per tenant by a platform super admin:

1. A tenant admin opens Data Flows → Staging → Ingest row storage and clicks Request staging.
2. A super admin approves it under Admin → Staging approvals, setting the retention window — how long staged row contents are kept before being purged (the audit ledger is kept).
3. The tenant admin picks a row-storage database (below). From then on files can land, and the tenant's templates, runs and bindings are active.

All staging settings live in one place, Data Flows → Staging. Documents → Sources only shows, on one line, what the current project uses: staging state, row-storage database, the project's schema and the retention window. Configure in Data Flows → Staging jumps to the settings, and Back returns you to Sources.

Where staged rows live

Staged row contents are sensitive, so they are stored in a staging database the tenant owns — a completely separate database from AgentData's own registry, registered under Data Flows → Staging (your own PostgreSQL, on-prem or cloud, or a shared hosted staging database granted by the platform). The tenant holds admin rights on that database; AgentData creates and manages only its own ingest tables inside it, and keeps nothing there but your rows.

• Per-project isolation — inside one staging database, every project gets its own schema (ing_p<project> on PostgreSQL), so projects never share tables. A project can also point at a different staging database entirely (Data Flows → Staging → Ingest row storage).
• Versioned structure — everything AgentData creates in your staging database comes from one versioned schema management file (full create scripts per staging table). The applied version is recorded in the database itself (ing_meta), so upgrades re-apply the file exactly once.
• One-click migration, build then load — change the tenant default or a project's placement and tick Migrate existing rows now. Phase 1 builds the structure in the destination database from the schema file; phase 2 loads the existing rows run by run (copy → repoint → empty the old location), executed as a temporary migration flow you can watch in Monitoring. Migration plan (.sql) downloads the full reviewable script first: the structure SQL per destination plus the export/load statements per run. Nothing is lost mid-move — each run always knows exactly where its rows are, and reviewer decisions (who approved/rejected, when) travel with the rows.
• Fail closed — until a storage database is selected, files can land but nothing stages: AgentData refuses to fall back to its own registry for row contents.
• The super admin's approval queue shows each tenant's storage placement (database, engine, location) — never its credentials, which stay encrypted.

Managed staging databases

Besides registering your own database, AgentData can provision one for you on its hosted server — a real, separate database next to (never inside) the platform registry, owned by a dedicated login created just for it:

• Instances — a staging instance is a database server the super admin registers (Admin → Staging approvals → Staging instances) from an ordinary PostgreSQL connection: pick one of your existing connections, open the regular Add-Connection form, or enter/edit the server details directly (AWS RDS/Aurora, Azure, Cloud SQL, self-hosted — any reachable PostgreSQL). Each instance row offers Test (verifies the saved connection and records the result), Edit (shows the full saved connection, editable), Deactivate (no new placements; existing databases keep working) and Delete. The built-in hosted server is pre-registered. Several tenants can share one instance — each gets its own dedicated database on it.
• Placement at approval — approving a tenant also places its storage: the super admin picks the instance and the database name (prefilled stg_t<tenant>), and that dedicated database is created and assigned on the spot. A storage the tenant already chose is never overridden. There is no shared multi-tenant staging database: one tenant's rows never share a database with another's — until a database is placed, nothing can stage (fail closed).
• Per-tenant management — each tenant row in the approval queue shows its staging name and database and offers three actions: Disable (staging pauses, every setting remains), Edit (move to a new dedicated database or one of the tenant's related databases — its old one or an empty managed one, never another tenant's — migrating rows along and optionally dropping the old managed database; the dialog also shows the database's owner login, password and URL, with password change), and Delete (the managed database and all its staged data are destroyed; the approval and settings remain).
• Inside the database — the tenant holds the owner (root) login: full connection details are visible to the tenant admin, the password is changeable, and they have admin rights inside the database (tables, stored procedures, sensitive data). Placement itself — creating or switching databases — is the platform admin's call, not the tenant's.
• Project schemas — inside your database every project maps to a schema. The default scope is named after your tenant, and a new project gets one automatically from the first letters of its name. Data Flows → Staging → Project schemas lists every project → schema mapping, with the current project marked; projects can't be removed from the list, but you can rename each schema — or point several projects at one. Renaming moves all existing data: on PostgreSQL the schema is renamed in place; otherwise rows are copied over and the old ingest objects are dropped.
• Full credentials, visible — for your own (dedicated or self-registered) database, Connection shows the host, port, database, user, password and the full ready-to-paste connection URL. On a managed database you can change the password any time (or generate a fresh one); the stored connection follows automatically.
• Reset, bounded — the super admin can reset the password of a managed database only. A database you registered with your own credentials is opaque to the platform: no reset, no credential access. Credentials of a shared multi-tenant staging database are never revealed to granted tenants.
• Isolation — every managed database revokes public connect rights; one tenant's login cannot reach another tenant's database or the platform registry.

Watching sources

Everything about watching files lives in Documents → Sources. Credentials never do: they stay in Connectors, and a binding simply picks an existing connection.

1. Add the connection once in Connectors → Add source: Local folder, Amazon S3 or On-Prem Connector for folders, SharePoint (directory/tenant ID, client ID, client secret of an Entra app with Sites.Read.All) or Google Drive (OAuth client ID, client secret, refresh token for drive.readonly). Secrets are stored encrypted and never shown again; rotating them in Connectors applies on the next sweep.
2. Create a binding in Documents → Sources → Add watched source: pick the kind and the connection, then the location — subfolder (folders), site + list ID (SharePoint list), library ID + folder path (SharePoint library) or folder ID (Google Drive) — the file pattern (*.xlsx, report-*.csv), the templates allowed to match, the poll interval and, for folders, whether to move picked-up files to processed/.

Kind	What is picked up
Local folder	Excel/CSV files on a folder connection
SharePoint list	the list's rows, snapshotted whenever they change (Microsoft Graph delta)
SharePoint library folder	Excel/CSV files in a document library folder
Google Drive folder	files in a Drive folder; Google Sheets arrive as Excel

Bindings are swept automatically on the scheduler, or on demand with Scan now; every sweep writes a line to Monitoring (files staged, duplicates skipped, or the error). Restrict a binding to specific templates so an unexpected layout never loads silently.

Upload file in the section header (next to Refresh) lands a single Excel/CSV into the same pipeline — handy for one-off files and first templates.

Templates

A template is a confirmed column mapping, versioned and matched by the file's column set (order doesn't matter):

• A file whose columns exactly match a confirmed template is validated immediately.
• A file with added or missing columns (drift) or a new layout stops as needs mapping and appears under Templates to approve — nothing is guessed.
• A file missing a column the template marks required (or its business key) is rejected as a whole, with the reason shown on the run.
• A template marked trusted auto-approves rows that validate cleanly — right for recurring, well-understood feeds.

Approving a template

Templates to approve and Templates are laid out like Documents → Templates: the list on one side, the selected file or template on the other.

Select a file under Templates to approve. The table preview shows the file as it was read: sheet tabs, the detected header row (change it if the export has a title block) and the first rows. Use this sheet and header row restarts the proposal from that view.

Below it, the mapping uses the same source → target arrows as the flow designer:

• Drag a file column onto a target column to connect it. Remove an arrow to leave the column out.
• Click an arrow to add a field transform (trim, upper, lower, to_number, to_date, round, …).
• Draw several columns into one target, or use Add calculated column (for example = {hours} * {fee}), to create a calculated column.

The Columns table sets each landing column's name, type, required and business key. JSON view shows the same mapping as text for export or import.

Transforms and calculated columns run in the template's flow, so the landing table keeps the raw values and the business table gets the shaped ones. Later edits are made in that flow.

For a drifted file there are three choices:

• Accept as new version — the edited mapping becomes the next version of the template (an added column becomes a new column in its tables).
• Alias to existing columns — the file renamed columns: map each incoming column to the existing one it replaces; the template keeps its structure.
• Ignore this file — the run is closed as ignored, with your reason kept.

Either confirmation stages the file's rows right away.

Excel support includes multi-sheet workbooks, merged cells (a merged title above the header is recognised as a banner), header rows that don't start at the top, hidden rows/columns, and formula cells (the stored value is used — file code never runs). CSV files may use , ; tab or | as the delimiter; it is detected automatically. Values that are not well-formed numbers in a number column (for example 1.200,50) are kept as written and flagged, never reinterpreted.

You can preview any CSV/Excel file the same way from Documents → Documents — selecting it opens the table view instead of the page viewer.

Template tables and load mode

A template whose target is staging owns three objects in your staging database (project schema): <kind>_<template>_landing (every row with its row_state and error_desc), the business table <kind>_<template> and the view <kind>_<template>_errors. Select a template under Templates to see its column mapping, these tables with row counts and Open flow, which opens the template's landing → final flow in the flow designer. Load mode — Incremental (upsert by business key, the default) or Snapshot (replace) — is set there too and applied to that flow. Export downloads the template definition as JSON.

Row review

Every staged row lands in exactly one state — ready, error, approved, rejected, loaded or duplicate — with a stable error code and a human-readable suggestion (bad number, missing required value, unknown status label, duplicate key…). Reviewers approve or reject row by row; a fix supplied with an approval is re-validated first, so approval can never bypass validation.

In Documents → Sources → Runs, open a run's rows, filter by state, read each row's error_desc, and Approve, Fix… (edit the values, then Approve with fix) or Reject. A file with both clean and error rows shows as partial. Tenant admins take these decisions in the app; integrated apps take the same decisions over the API.

Loading and the ledger

Load approved releases approved rows to the template's target — writes to a connected SaaS system stay dry-run until explicitly run live by a tenant that allows it. Loads are keyed and re-runnable: a second load with nothing newly approved writes nothing. The per-run ledger accounts for every row in the file; totals can be reconciled against a claimed report.

For app builders

The whole pipeline is exposed to external apps through two doors, both scoped to the tenant of the API key:

• REST — /api/ingest/v1: land files, stage, list/match/confirm templates, page rows (rejected included), approve/reject with fixes, load, ledger, reconcile. Keys need read for reads and the explicit ingest scope for decisions.
• Management views for an app that runs the review itself:
  • GET /templates/pending — files waiting for a template decision (headers, column profile, proposed mapping, drift).
  • POST /templates/{run_id}/approve with mode new_version (+ mapping), alias (+ aliases {incoming: existing}) or ignore (+ reason).
  • GET /templates/{id}/errors and GET /templates/{id}/table — paged rows (offset, limit) of the errors view and the business table; GET /templates/{id}/tables for names and counts.
• MCP tools — the same operations for agents (ingest_land_file, ingest_stage, ingest_pending_templates, ingest_approve_template, ingest_template_errors, ingest_template_table, ingest_rows, ingest_decide_row, ingest_load, …).

Typical pattern: a review screen inside your own product (for example a monday.com board view) drives template approval and row review against AgentData, which remains the governed system of record.

Try it

A five-minute walkthrough on a tenant with approved staging:

1. Connectors → Add source → Local folder (or SharePoint / Google Drive) — create the connection.
2. Documents → Sources → Add watched source — pick the connection, pattern *.xlsx, poll every 5 minutes.
3. Drop an export (for example a projects sheet with Project, Person, Status, Hours, Fee) into the folder and click Scan now — or use Upload file.
4. Under Templates to approve, check the preview, set Project as business key, mark Hours required, click the Person arrow and pick upper, add a calculated column total = = {hours} * {fee}, and confirm. The run shows its rows: clean rows ready, problems as error with their error_desc.
5. Open Runs → View rows, fix one error row with Fix…, reject another, approve the rest, then Load approved.
6. Select the template under Templates: the landing table holds every row with its state (raw values), the errors view the remaining problems, and the business table the flow-written rows, with person upper-cased and total calculated. Open flow to edit the shaping later.
7. Drop the next week's export with an extra Region column: it appears under Templates to approve as drift — Accept as new version creates v2 and adds the column.
8. Monitoring shows every binding sweep.

What AI sees

AI assistance over ingest works on column profiles and coded issues only — header names, value shapes, types and error codes. Row values never reach a model. Staged row contents live only in the tenant's own staging database (isolated per project) and are purged after the retention window set at approval; the ledger and audit trail remain.