WEEK1
LLMs, calling an LLM API, rate limits (TPM/RPM)
Build
An LLM client with retries, timeouts and a rate limiter; every call logged to a table; a domain config loader.
Pass this test
A burst of 50 calls finishes with no rate-limit errors.
18-week course roadmap for working data engineers
Turn your data skills into AI engineering skills.
You already move, clean and model data. Learn to add retrieval, agents, guardrails and production serving by building one real product, one module a week.
0weeks
0products built
0industries
0per week
Why this roadmap
The AI work inside companies is mostly data engineering.
Parsing documents, chunking, embedding, keeping an index fresh, controlling who sees what, and proving answers are right: that is your day job, with new tools. This course closes the gap without months of theory. Each week has one topic and one build, and nothing is thrown away.
Learn by building
One product, DataPilot, grows a module each week. You never start from a blank page.
Skills that transfer
Industry details live in config “packs”, so the same code runs on Lending, Retail and Telecom.
Production habits
Tracing, evaluation, access control, load tests and failure drills are part of the plan.
After 18 weeks you can
- Turn PDFs, scans and messy text into clean, searchable, governed data
- Build RAG that answers with citations using vector, keyword and hybrid search
- Build agents that query a warehouse safely and diagnose pipeline failures
- Control access by role and keep customers away from internal data
- Test, trace and debug AI systems like any production service
- Serve and monitor on Databricks, and sit a Generative AI Engineer exam
How it works
One routine. Eighteen weeks.
About 10 hours a week. Only the topic changes.
Mon1.5 h
Learn
Watch or read the week's free course module.
Tue1.5 h
Try it
Reproduce the course example in a notebook.
Wed1.5 h
Build
Turn the notebook into a product module.
Thu1.5 h
Test
Run the week's test; fix until it passes.
Fri1.5 h
Swap domain
Rerun on another industry. From week 10, add a 45-min incident drill.
Sat2 h
Write up
Record results and what broke when you swapped.
Sun1 h
Design
One system-design interview question.
Fall behind? Shorten Monday's course time, never Thursday's test or Friday's domain swap. A module that works on two domains beats a finished video course.
Domain packs
A pack is a folder with everything industry-specific: warehouse schema, sample data, documents, catalog, brand voice, compliance rules and test questions. The one rule: if switching domains needs a code change, that's a bug. Move it into the pack.
Where each domain is used
1–4Lendingswap: Retail
5–8Retailswap: Lending + Telecom
9–12Telecomswap: Lending + Retail
13–14Retailswap: Lending + Telecom
15–18All threeoptional 4th pack
The roadmap
Five phases from first prompt to production.
Pick a phase to see what you learn, build and test each week.
Phase 1: Foundations
Primary domain · LendingLearn how LLMs work and turn messy documents into clean, searchable data.
WEEK2
Structured outputs, tool calling, chain-of-thought
Build
A document extractor that turns text into validated JSON, with a direct vs step-by-step comparison.
Pass this test
10 sample documents load, bad ones are rejected, accuracy recorded.
WEEK3
Embeddings and semantic search
Build
A warehouse built from a schema, plus search over column descriptions and cosine similarity written by hand.
Pass this test
“Where is the payment date?” finds the right column.
WEEK4
Parsing unstructured data, OpenCV, document loaders
Build
Ingestion of internal and customer documents into tables, with scan clean-up before OCR.
Pass this test
20 PDFs ingested two ways; OCR accuracy compared before and after clean-up.
Phase 2: Ask the Docs
Primary domain · RetailBuild a retrieval-augmented (RAG) assistant that answers from company documents with citations.
WEEK5
RAG fundamentals, chunking, LangChain core
Build
Four chunking strategies compared, including the recursive splitter.
Pass this test
10 questions pick a winning strategy.
WEEK6
Vector databases
Build
A pgvector store with domain and audience filters.
Pass this test
Top-5 results are relevant; customer queries never return internal chunks.
WEEK7
RAG app, BM25 hybrid search, MMR
Build
Ask the Docs with page citations, keyword + vector fusion and diversity-aware retrieval.
Pass this test
10 answers cite the right file and page.
WEEK8
Orchestrating AI data pipelines
Build
Incremental re-embedding by hash with an Airflow DAG per domain.
Pass this test
Re-running skips unchanged documents.
Phase 3: Agents
Primary domain · TelecomBuild agents that use tools: query the warehouse, diagnose failures, respect roles, and measure quality.
WEEK9
Agents, text-to-SQL, ReAct, tracing
Build
Ask the Data: a LangGraph text-to-SQL agent (read-only, timeouts, self-correction) with end-to-end tracing.
Pass this test
15 questions answered correctly; every request has a complete trace.
WEEK10
Agents with tools and retrieval
Build
Triage a Failure: classify seeded pipeline failures and suggest fixes from similar past incidents.
Pass this test
10 seeded failures classified correctly.
WEEK11
Persona routing, auth roles, memory, human approval, MCP
Build
A persona-aware router, per-user rate limits, memory, approval before large queries, and an MCP server.
Pass this test
Each role reaches only its allowed capabilities and data.
WEEK12
Evaluating LLM apps; RLHF and LoRA concepts
Build
An evaluation harness for every capability, plus a small LoRA fine-tuning notebook.
Pass this test
One command produces a full quality report.
Phase 4: Customers & Marketing
Primary domain · RetailPut agents in front of customers safely, and add generative content with compliance and human sign-off.
WEEK13
Customer-facing agents, guardrails, prompt injection, handoff
Build
A public Customer Assistant on a separate endpoint, scoped to the logged-in customer, with input/output checks and human handoff.
Pass this test
20 customer questions answered; 15 attack prompts all refused.
WEEK14
Segmentation, content generation, compliance, approval
Build
Marketing Studio: pick a segment, write the brief and ad copy, optionally create images, check compliance, queue for approval.
Pass this test
A full campaign passes compliance; a rule-breaking draft is blocked.
Phase 5: Production & Certification
Primary domain · All threeGovern, serve, monitor and stress-test the platform, then certify.
WEEK15
Databricks data prep and governance
Build
Chunks in Delta tables under Unity Catalog, Vector Search, PII masking before embedding, grants per persona.
Pass this test
No raw PII in the index; grants enforced per role.
WEEK16
MLflow, Model Serving, Databricks Apps
Build
Agents logged and registered, served on Model Serving, with versioned prompts and an internal app.
Pass this test
The endpoint answers every capability with traces.
WEEK17
Monitoring, containers, load, spike and failure testing
Build
Custom scorers, AI Gateway limits, Docker Compose, Locust load tests, failure tests, circuit breaker and fallback model.
Pass this test
Under a 10x spike, error rate and latency stay within targets.
WEEK18
Production readiness and the exam
Build
The production-readiness gate, a runbook, final evaluation, and the certification exam.
Pass this test
Every gate item is checked.
Free learning material
What to watch and read, week by week.
Tip: click any resource title to open it in a new tab. Some courses need a free account.
Sunday design hour books: Designing Data-Intensive Applications (Kleppmann), weeks 1–9; AI Engineering (Chip Huyen), weeks 10–18.
What you will build
One platform. Six capabilities.
DataPilot serves three audiences: the data team (everything), business users (answers and campaigns, never raw personal data) and customers (public info and their own account only). The customer assistant runs on a separate public endpoint, so it is never one prompt injection away from internal data.
RAG
Ask the Docs
Answers questions over company documents with page citations, kept fresh by a daily pipeline.
RAGpgvectorBM25Airflow
Text-to-SQL
Ask the Data
Turns a plain-language business question into safe, read-only SQL and returns an answer or chart.
LangGraphSQLTracing
Reliability
Triage a Failure
Reads a failed pipeline log, classifies the failure, finds similar past incidents and suggests a fix.
AgentsEmbeddingsAirflow
Guarded agent
Customer Assistant
A public assistant that answers from customer-safe documents and the customer's own account, and hands off to a human when unsure.
GuardrailsAuthRed teaming
Generative AI
Marketing Studio
Picks a segment from the warehouse, writes the brief and ad copy, checks compliance, and waits for human approval.
LLMsOpenCVCompliance
Access control
Persona Router
Decides which capabilities and data each user can reach: data team, business user or customer.
RolesMCPRow-level security
How the product grows
Wk 4
Extract fields, search metadata, ingest PDFs
Wk 8
Ask the Docs with citations, fresh daily
Wk 12
Ask the Data, Triage, router, MCP, evals
Wk 14
Customer Assistant + Marketing Studio
Wk 18
Governed, served, load-tested, certified
Industry impact
How industry solves real problems with it.
The same capabilities apply across sectors. Only the data, documents and rules change.
Problem
Analysts are the bottleneck for every portfolio question.
How it helps
Ask the Data lets loan officers ask “which loans are 30+ days past due by branch?” and get an answer without waiting.
Problem
Borrowers call to ask about balances and payment dates.
How it helps
The Customer Assistant answers from public policies and the customer's own loan record, and hands off when unsure.
Problem
Ads and emails must carry required disclosures.
How it helps
Marketing Studio checks every draft against compliance rules and blocks it until a person approves.
Problem
Policy documents are long and scattered.
How it helps
Ask the Docs answers with the exact page it came from, which auditors can verify.
Problem
Returns and order questions flood support.
How it helps
The Customer Assistant answers return policy and order status, and refuses to reveal other customers' data.
Problem
Merchandising can't easily see why products come back.
How it helps
Ask the Data answers “which products had the highest returns last quarter?” with a chart.
Problem
Win-back campaigns take days to prepare.
How it helps
Marketing Studio selects the segment, writes ad variants per channel, and queues them for approval.
Problem
Supplier invoices arrive as PDFs and scans.
How it helps
The extractor and OpenCV clean-up turn them into validated, searchable data.
Problem
Plan and bill questions drive call volume.
How it helps
The Customer Assistant explains plans and bills using the customer's own account data.
Problem
Billing pipelines fail overnight and teams scramble.
How it helps
Triage a Failure classifies the log and suggests a fix from similar past incidents.
Problem
Churn needs targeted offers.
How it helps
Marketing Studio builds a segment and an offer campaign with compliance checks.
Problem
Technical manuals are hard to search.
How it helps
Ask the Docs gives cited answers to field and support teams.
Shared governance
One place decides who sees which data, so a new feature never opens a new leak.
Shared quality checks
One evaluation harness and tracing setup cover every capability.
Faster rollout
A new industry is a new pack of config, documents and test questions: hours, not weeks.
Production ready
Built to run, not just to demo.
The product isn't “delivered” until every item passes.
- Login and role-based access for every persona
- Rate limits per user, per key and per provider
- Timeouts on every external call
- Retries with jitter, fallback model, circuit breaker
- Prompt-injection and PII guardrails on public endpoints
- Failure tests: database down, slow LLM, tool errors, bad input
- Spike test at 10x users within latency and error targets
- End-to-end tracing of every request
- Monitoring and alerts on errors, latency and cost
- Evaluation tests run automatically before every release
- A runbook for tracing, reproducing and fixing failures
The incident drill
From week 10, every Friday you break something on purpose and fix it the same way:
- Trace the failed request
- Locate retrieval, tool or model fault
- Reproduce by replaying the trace
- Fix the root cause
- Add a test so it never returns
What about a million users?
If 10% of 1 million users ask 5 questions a day:
500Kquestions / day
~60per second at 10x peak
~11Mtokens / minute
The bottleneck is the LLM provider, not the web servers. Fixes: gateway quotas, autoscaling servers with Redis memory, an LLM gateway with fallback and caching, queues for slow work, and a protected warehouse.
Certify and interview
Finish with a credential and eighteen design answers.
Databricks Generative AI Engineer Associate
45 scored questions, 90 minutes, $200, online proctored. The product you build is your hands-on prep for all six sections: design, data preparation, application development, deployment, governance, and evaluation and monitoring.
Sunday system-design track
1Bronze/Silver/Gold batch pipeline2CDC with SCD Type 23Real-time pipeline with Kafka4Data quality and reconciliation5Backfilling years of history6Governed multi-team platform7Ingesting millions of PDFs8Orchestrating 500 pipelines9Tracing an agent platform10RAG over 10 million documents11Multi-tenant agent platform12LLM evaluation and regression testing13Public support agent with guardrails14AI marketing system with approval15PII-safe retrieval16Serving 1 million users17Incident response for AI systems18Mock interview: present the product in 10 minutes
FAQ
Questions, answered.
Who is this for?
Working data engineers comfortable with SQL, Python and pipelines who want to add AI skills. No prior machine-learning experience is needed.
How much time do I need?
About 10 hours a week for 18 weeks: 1.5 hours each weekday, 2 hours on Saturday and 1 hour on Sunday.
Does it cost money?
All learning material is free. You pay for LLM API usage, which stays low with small models while building. The optional exam costs $200.
Do I need Databricks?
Weeks 1–14 run locally with Postgres, Airflow and Python. Databricks appears in weeks 15–18 for governance, serving and the exam.
Which LLM provider should I use?
Any. The provider lives in configuration, so you can use OpenAI, Gemini, Claude or an open-source model through Ollama.
What if I fall behind?
Skip course videos, never the build. Always run Thursday's test and Friday's domain swap.
Ready to build your first AI data product?
Start with week 1: an LLM client with retries, rate limiting and call logging.
Open the roadmap