Natural Language Data Agent

A conversational agent that turns plain-English questions into verified SQL queries, pipeline diagnostics, and team visibility, grounded in the same versioned contracts engineers already maintain.

The Problem

Data teams maintain detailed documentation: schema contracts, pipeline definitions, lineage graphs, and domain READMEs. But that knowledge is rarely accessible to the people who need it most.

Analysts spend time tracking down the right table. Engineers context-switch during incidents to piece together upstream dependencies. Engineering managers lose visibility into what shipped and what is at risk.

The agent was built to serve all three audiences from a single interface, without requiring any of them to know where the documentation lives or how to navigate it.

How It Works

Every message goes through a structured pipeline: intent is classified first, then the right documentation is retrieved and validated, and finally a response is generated using a template matched to the query type.

flowchart LR
    A([User Message]) --> B[Classify Intent]
    B --> C[Search Documentation]
    C --> D[Read & Validate]
    D --> E[Generate Response]
    E --> F([Deliver to User])

Classifying intent before searching is the key architectural decision. It ensures the agent retrieves only the documentation relevant to the question, keeping responses accurate and the context window focused.

Intent Classification

The agent handles six query types, each mapped to a dedicated search strategy and response template.

flowchart TD
    A[User Message] --> B{Greeting?}
    B -- Yes --> C[Capabilities Overview]
    B -- No --> D{Discover available tables?}
    D -- Yes --> E[Table Discovery]
    D -- No --> F{Schema or column details?}
    F -- Yes --> G[Schema Lookup]
    F -- No --> H{Generate a query?}
    H -- Yes --> I[SQL Generation]
    H -- No --> J{Pipeline or lineage?}
    J -- Yes --> K[ETL Lineage]
    J -- No --> L[Fallback: Ask for Clarification]

Routing before retrieval means the agent never searches blindly. Each query type targets the subset of documentation most likely to contain the answer.

For Analysts: Self-Serve Data Access

Analysts often know what they want to understand but not how to get the data. The agent handles schema discovery and SQL generation so analysts can go from question to query without involving engineering.

sequenceDiagram
    actor Analyst
    participant Agent as Data Agent
    participant Docs as Documentation Layer
    participant SQL as SQL Generator

    Analyst->>Agent: Plain-English question
    Agent->>Docs: Search schema documentation
    Docs-->>Agent: Relevant contracts and metadata
    Agent->>Agent: Validate schema, table, date columns
    Agent->>SQL: Generate SQL with verified schema
    SQL-->>Agent: Query and explanation
    Agent-->>Analyst: SQL · column descriptions · data freshness · source citations

A pre-generation validation step catches the most common failure modes: wrong table references, incorrect date granularity, and mismatched column types, before any SQL reaches the analyst.

Safe Execution

The agent supports optional query execution but always shows a cost estimate first. No query runs without explicit confirmation.

sequenceDiagram
    actor Analyst
    participant Agent as Data Agent
    participant Warehouse as Data Warehouse

    Analyst->>Agent: Confirmed question
    Agent->>Warehouse: Dry-run (estimate only)
    Warehouse-->>Agent: Bytes scanned and validation result
    Agent-->>Analyst: "Query scans ~2.3 GB. Run it?"

    Analyst->>Agent: "Yes"
    Agent->>Warehouse: Execute with row limit
    Warehouse-->>Agent: Results
    Agent-->>Analyst: Formatted results and summary

At enterprise data scale, a single unconstrained query can scan terabytes unexpectedly. Treating execution as opt-in aligns with how cost-conscious teams already think about ad-hoc queries.

For Engineers: Pipeline Debugging

When a pipeline fails or data looks wrong, the first questions are always the same: which upstream table is the source, when did it last refresh, and what transformation sits between raw ingestion and the final output?

Previously, answering those questions meant cross-referencing model definitions, checking scheduler logs, and tracing dependencies manually across multiple systems. The agent consolidates that into a single query.

sequenceDiagram
    actor Engineer
    participant Agent as Data Agent
    participant Docs as Documentation Layer

    Engineer->>Agent: "When did this table last refresh and what feeds it?"
    Agent->>Docs: Search lineage sections in documentation
    Docs-->>Agent: Upstream sources, schedule, last run status
    Agent-->>Engineer: Upstream dependencies · refresh cadence · last successful run · owner contact

A typical lineage response surfaces:

• Upstream sources: which raw or staging tables feed the model
• Refresh schedule: how frequently the pipeline runs and the expected SLA
• Last known run: timestamp of the most recent successful load
• Owner: the team or individual responsible for the pipeline

This matters most during incidents. An engineer can confirm within seconds whether the problem is in the transformation logic or whether an upstream dependency simply has not refreshed yet. That distinction alone reduces on-call triage time significantly.

For Engineering Managers: Work Visibility and PR Review

The agent is not limited to data queries. Because it already has access to the repositories that hold contracts, pipeline code, and documentation, it can serve engineering managers as a lightweight work tracking and code review tool without any additional integration.

Tracking Work in Progress

sequenceDiagram
    actor EM as Engineering Manager
    participant Agent as Data Agent
    participant Repo as Repository Layer

    EM->>Agent: "What pipeline changes went out this week?"
    Agent->>Repo: Search recent commits and merged pull requests
    Repo-->>Agent: Merged changes, authors, linked tickets
    Agent-->>EM: Summary of changes · owners · scope of impact

Rather than navigating pull request lists or interrupting the team for status updates, the EM gets a structured summary: what changed, who owns it, and which tables or domains are affected.

PR Review Assistance

sequenceDiagram
    actor EM as Engineering Manager
    participant Agent as Data Agent
    participant Repo as Repository Layer

    EM->>Agent: "Review the open PR for the orders pipeline"
    Agent->>Repo: Fetch PR diff and related contract files
    Repo-->>Agent: Code changes and existing schema contracts
    Agent->>Agent: Compare changes against contracts and conventions
    Agent-->>EM: Change summary · schema impact · potential risks · suggested questions

The agent accelerates the first pass of a review: understanding what the PR is trying to do, whether it aligns with the existing contract, and which questions are worth raising. This is particularly useful for EMs reviewing work across multiple domains where keeping the full context of every table and pipeline in memory is not realistic.

Graceful Degradation

When documentation is incomplete or the question is ambiguous, the agent responds with what it knows and asks for clarification, rather than guessing.

flowchart TD
    A[Classify Intent] --> B{Documentation found?}
    B -- Yes --> C{Sufficient data?}
    C -- Yes --> D[Generate Response with Citations]
    C -- Partial --> E[Respond with Available Info and Note Gaps]
    C -- No --> F[Inform User and Suggest Next Steps]
    B -- No --> G[Broaden Search or Ask Clarification]
    F --> G

Every response includes source citations so users can verify the information and flag documentation that needs updating.

Why GitHub as the Knowledge Source

Data contracts and schema documentation live in many places: wikis, spreadsheets, internal portals, Slack threads. Most of these are stale within weeks of a schema change.

Source control is different. When a table is modified, the contract file gets updated as part of the same pull request that changes the pipeline. The documentation in GitHub reflects the actual deployed state of the data, not what someone intended to write six months ago.

Grounding the agent in GitHub provided three concrete guarantees:

• Accuracy: schema definitions match what is in production, because they go through the same review process as the code
• Ownership: every contract file has a clear team owner, making it straightforward to surface the right contact when documentation is incomplete
• Auditability: users can follow the citation link and see the exact file and version the agent used to generate a response

Indexing a wiki or a metadata portal would have introduced a synchronisation lag and a trust problem. Using source control as the single source of truth removed that ambiguity entirely.

Extensibility: Built for Atomic Teams

The agent is designed around a collection registry: a configuration file that maps domain names to their repository locations. Adding a new domain to the agent is a configuration change, not an engineering project.

flowchart TD
    A[Collection Registry] --> B[Domain A: Repo 1]
    A --> C[Domain B: Repo 2]
    A --> D[Domain C: Repo 3]
    A --> E[Domain N: Repo N]

    B --> F[Agent]
    C --> F
    D --> F
    E --> F

Any team that follows the same documentation framework, versioned contracts, a domain README, and consistent schema definitions, can onboard onto the agent in two ways:

Add to the shared collection — the team registers their repository in the central collection registry. From that point, the agent can answer questions about their domain alongside every other onboarded team. No code changes required.

Fork and configure — teams that want a dedicated agent for their domain can fork the base configuration, point it at their own repositories, and tune the intent classification and response templates to match their specific workflows. The core pipeline stays the same; only the knowledge source and templates change.

This model means the agent scales horizontally across an organisation. As more teams adopt the same contract framework, the agent becomes more capable without any changes to the underlying system.

Impact

The most significant shift was making the documentation that engineers already maintain directly useful to everyone who depends on it, without any extra tooling or process.

January 2026