LLM Applications

Core SystemsModule 3 · Required

Build an LLM feature with clear inputs, validated outputs, and an evaluation that reveals where it succeeds and fails.

Frame a task

Define context and a useful output contract.

Validate responses

Use structured outputs and handle invalid results.

Evaluate behavior

Run test cases and inspect failures and limitations.

LLM Application Fundamentals

build an LLM feature with a clear contract and measurable behavior.

Before you start: Software Engineering for AI; use deep learning's transformer introduction for background.

Learn

• Tokens, context limits, message roles, prompts, and selecting a suitable model for a task.
• Hosted model calls or local inference; configuration, timeouts, retries, and response handling.
• Structured outputs and schema validation; valid structure alone does not establish factual correctness.
• Embeddings, retrieval, and tool calling at a conceptual level. Full RAG and agent implementations belong to electives.
• Build a fixed evaluation set, inspect errors, and compare quality, latency, and cost or local resource use.
• Handle sensitive inputs deliberately and recognize untrusted instructions in supplied content.

Resources

video

Karpathy: Intro to Large Language Models

Required. Focus on training/inference, model behavior, and limitations.

video

Karpathy: How I use LLMs

Required. Selected examples of task framing and practical use. Apply one pattern and evaluate the result.

article

Hugging Face: Structured Outputs

Required. Study JSON schemas and validation. Use one model/provider that supports your chosen setup.

article

Anthropic: Demystifying evals for AI agents

Required. Use the introductory evaluation concepts: test inputs, success criteria, grading, and regression checks. Advanced agent sections support the Agents and Tool-Based Workflows elective.

Practice

• Build a small extraction or classification feature with a documented input/output schema.
• Create a fixed set of representative, ambiguous, and invalid inputs with expected behavior.
• Compare two prompts or model configurations; record task accuracy, invalid outputs, latency, and token/cost information where available.
• Handle a timeout, malformed response, or unavailable model without presenting it as a successful result.

Expected output

A working LLM feature, validated response handling, an evaluation script and results, and an explanation of failures. A local model or Academy-provided endpoint can support the task; compare local resource use when no per-call pricing applies.

Workload

Use the provisional estimates where supplied. Other workloads remain pending pilot. Include implementation, debugging, review, and revision.

Finish this module

Keep the required outputs and explain your decisions. Diagnose a relevant failure during review and revise incomplete evidence after feedback. Resource completion supports progress; the stated practical requirements determine completion.