Background
On our domain data platform team, bringing a new data source into the data lake means writing a PySpark transformation, wiring up an Airflow DAG, generating policy files for dev, stage, and prod, checking everything against our Lake Naming Standards, and opening a PR. It's the same sequence every time. For our 2026 Hackathon, we built Pipeline Build Agent — a conversational AI agent that does all of that from a single Slack message, taking a task that used to take hours and reducing it to minutes.
The cost of repetition
Every time a product team needed logs from a new S3 bucket in the data lake, an engineer had to sit down and manually walk through the same steps: open the templates, infer the schema, look up naming conventions, fill in the right bucket paths and table names, generate the files, and open a PR. The work wasn't hard — it was just time-consuming and easy to get wrong.
The frustrating part was that we already had everything we needed to do it automatically. We had a Lake Naming Standards knowledge base, a well-defined file structure, and templates for every artifact. The knowledge existed; it just wasn't being applied without a human in the loop. That gap — between "we know exactly how to do this" and "it still takes 2–4 hours each time" — was the problem we wanted to solve.
We also had a secondary problem: compliance. Our naming standards aren't optional. Pipelines that don't follow them fail review. Engineers who were unfamiliar with the standards or in a hurry would sometimes generate files that needed to be fixed before the PR could merge, adding more back-and-forth.
Why we chose an AI agent
Our first instinct was to build a script or CLI wizard that walked engineers through the required inputs. But that approach hits a wall quickly: pipeline creation isn't a form you fill out — it's a conversation. You might not know the output table name upfront, or whether to include all fields, or what to name the pipeline. You figure those things out by talking through them.
We evaluated a few options:
- A web form — forces all inputs upfront, doesn't handle optional or derived fields well
- A CLI script — automates generation but still needs a human to drive every step
- An AI agent — can ask clarifying questions, infer schema from S3 automatically, and handle the full workflow end-to-end
The agent approach also let us meet engineers where they already are. Our team lives in Slack. Wrapping the agent in a FastAPI service gave us Slack, REST API, Swagger UI, and GitHub Actions as entry points — all with the same underlying conversation model.
We built the system on our internal go-agent-python SDK and deployed it on Katana (GoDaddy's internal continuous delivery platform).
How it works
The agent uses a multi-agent architecture where a "manager agent" classifies intent and routes to one of two "specialist agents".
User (Slack / REST / Swagger / GHA)
↓
FastAPI runner.py
↓
Manager Agent ──→ Pipeline Creation Agent
──→ Column Addition Agent
The Pipeline Creation Agent handles new pipelines. It checks S3 schemas, generates all required files, and opens a PR. The Column Addition Agent handles modifications. It finds the relevant pipeline, propagates the change to downstream tables, and opens a PR.
Each specialist has a focused set of tools:
| Agent | Key tools |
|---|---|
| Pipeline Creation | check_s3_schema, create_pyspark_job, create_dag, create_policy_files, create_pull_request |
| Column Addition | find_pipeline_by_table, get_related_tables, read_pipeline_script, write_modified_pipeline_script, create_pull_request |
Both specialists also have access to GoKnowb (GoDaddy's internal AI knowledge base), which queries our Lake Naming Standards knowledge base.
The knowledge base-first pattern
Early on we ran into a naming compliance problem. The agent would generate code, then check it against our standards, find violations, and need to rewrite and re-push. This produced messy PRs with multiple corrective commits.
The fix was to flip the order. Every specialist now queries the naming standards knowledge base before generating any code. The standards land in the LLM's context first, so pipeline names, table names, database names, and file paths are compliant from the first generation. The update_pr_files tool exists as a safety net but is rarely needed.
A full session
The following conversation illustrates what a pipeline creation conversation looks like end-to-end:
👤 User: I need logs from s3://example-data/raw-logs/app-logs/ processed
🤖 Agent: ✅ Found schema:
• timestamp (datetime)
• user_id (string)
• event_type (string)
• value (integer)
Do you need all fields or want to exclude any?
👤 User: All fields
🤖 Agent: What should we name this pipeline?
Suggestion: app_logs_pipeline
👤 User: app_logs_pipeline
🤖 Agent: ✅ Created:
- PySpark job: myteam/app_logs_pipeline/src/pyspark/app_logs_pipeline_job.py
- DAG: myteam/app_logs_pipeline/app_logs_pipeline.py
- Policy files (dev/stage/prod + pipeline)
Create a pull request?
👤 User: yes
🤖 Agent: ✅ PR ready:
https://github.com/my-repo/demo-piplines/pull/123Six files committed, one PR opened, naming-standards compliant. The generated file structure looks like this:
myteam/<pipeline_name>/
├── <pipeline_name>.py # Airflow DAG
└── src/
├── pyspark/
│ └── <pipeline_name>_job.py # PySpark transformation
└── policies/
├── environment.dev_private.yaml
├── environment.stage.yaml
├── environment.prod.yaml
└── <pipeline_name>.yamlThe PySpark job reads from S3, writes to the Glue catalog with Hive-style partitioning (snap_date=YYYY-MM-DD), and integrates with Switchboard. The policy files are pre-populated with the correct AWS account numbers per environment. Conversation state is managed via a conversation_id so the agent can reference earlier answers (like the pipeline name) without asking again.
What we got wrong first
Keeping the PR clean was harder than expected. Without a strict rule, the agent would sometimes call create_pull_request more than once per session, opening duplicate PRs. We solved this by making "one PR per run" a hard invariant. create_pull_request is called exactly once, and update_pr_files is the only path for any corrections. That single constraint cleaned up most of the edge cases.
Results and what's next
Since deploying on Katana, our team has been using the agent to handle routine pipeline requests through Slack. What used to take a few hours of manual work (including looking up standards and wiring up templates) now takes a few minutes of conversation.
We learned a few general lessons from this project:
- Treat internal standards as context to inject early, not guardrails to check at the end.
- Separate agents with narrow responsibilities are easier to maintain and debug than one large agent trying to handle everything.
- For tasks with optional or derived inputs, a conversational interface is a better fit than a form.
We're planning to add Redshift and S3 CSV export as pipeline destinations next, along with pre-PR syntax validation for generated DAGs and PySpark code.
If you're interested in working on problems like this, we're hiring — check out our open engineering roles at GoDaddy Careers.
