应用工程：从调一次 API 到带 evals 的生产系统

• Claude Cookbooks — Anthropic freeDone with Claude Cookbooks — Anthropic

  repo — Runnable notebooks for the calls you will actually make — tool use, JSON output, prompt caching, batches — each a working example rather than a description.

• Let's build the GPT Tokenizer — Andrej Karpathy freeDone with Let's build the GPT Tokenizer — Andrej Karpathy

  video · 15 min · only 0:00–14:56 — What a token is, in a live tokenizer: why the same text costs different amounts on different models, and why models stumble on spelling and arithmetic.

  Play 0:00–14:56 of “Let's build the GPT Tokenizer — Andrej Karpathy”Loads the YouTube player (youtube-nocookie.com) when you press it

复现项目: A client that knows what it costs — Call one model API with streaming, one tool and a JSON schema; log tokens and latency per call, and work out the cost of a thousand requests.

• Prompting best practices — Anthropic freeDone with Prompting best practices — Anthropic

  docs — The model maker's reference, rewritten with each release so it fits the models you will call: instructions, examples, structure, output format, tools. Start with the techniques for all models.

复现项目: Improve one prompt with a test set, not by feel — Fifty cases with expected outputs, a baseline score, then each prompt change measured as a diff — keep the ones that move the number.

• Structured outputs — Anthropic freeDone with Structured outputs — Anthropic

  docs — 模型侧 JSON schema 约束和 strict 工具参数的官方说明：支持哪些 schema 特性、编译后的 grammar 怎么缓存、和工具调用怎么配合——先弄清供应商保证了什么，再决定自己在哪一层兜底。

• Instructor freeDone with Instructor

  repo — 把 schema 校验、失败重试、多供应商统一成一个薄层的开源实现；读它的校验与重试环，等于拿到自己写降级链时的参考答案。

复现项目: 一个带校验与降级链的抽取器 — 对一批真实输入用 JSON schema 约束抽取字段；schema 特性不支持时退到 prompt 约束加本地校验，再退到重试。记每一级的命中率、失败率和每条成本。

• What are embeddings? — Vicki Boykis freeDone with What are embeddings? — Vicki Boykis

  book — A short free book that goes from one-hot vectors to transformer embeddings with the engineering around them — the one explanation practitioners keep linking.

复现项目: Semantic search over your own documents — Embed a few thousand of your own documents into a vector index, then compare its top ten against keyword search on twenty queries you label.

• Introducing Contextual Retrieval — Anthropic freeDone with Introducing Contextual Retrieval — Anthropic

  article — Hybrid search (embeddings plus BM25), reranking and chunk context, each with a measured drop in retrieval failures — design with numbers.

复现项目: Hybrid retrieval with a labelled query set — BM25 plus vectors fused with reciprocal rank fusion, a reranker, and 50 hand-labelled queries; report recall@k and nDCG against keyword search alone.

• Building effective agents — Anthropic freeDone with Building effective agents — Anthropic

  article — Workflows versus agents, and when you need neither: the handful of patterns these systems are built from, named by a lab that ships them.

• Hugging Face Agents Course freeDone with Hugging Face Agents Course

  course — Hands-on: tool calling, a ReAct-style loop and multi-agent setups in code you run — the gap between reading about agents and writing one.

• Effective context engineering for AI agents — Anthropic freeDone with Effective context engineering for AI agents — Anthropic

  article — Context is the agent loop's most expensive variable: when to trim, compress, take notes or hand off to a sub-agent, sorted by problem — the checklist for your own loop's context budget.

复现项目: Write the agent loop yourself, then evaluate it — Messages API tool use with a step cap, tool-error retries and context trimming; score it on 30 real tasks for tool choice, argument correctness and steps taken.

• Model Context Protocol — introduction and spec freeDone with Model Context Protocol — introduction and spec

  docs — The protocol itself — tools, resources, transports — straight from the spec rather than from a framework's wrapper around it.

复现项目: An MCP server for an API you know, measured by an agent — Expose five real operations as tools, then run an agent on 30 tasks and count how often it picks the right tool with the right arguments; rewrite descriptions until that number moves.

• AI Evals: Everything You Need to Know — Hamel Husain and Shreya Shankar freeDone with AI Evals: Everything You Need to Know — Hamel Husain and Shreya Shankar

  article — Start from error analysis on real traces, not from a metric; then judges you check against people — the practice behind 'evaluation frameworks' in these postings.

• promptfoo freeDone with promptfoo

  repo — Runs an eval set as CI — cases in YAML, assertions by rule or by judge, red when the score drops — turning one-off eval studies into a regression suite. Open source; part of OpenAI since March 2026.

• Inspect freeDone with Inspect

  repo — An evaluation institute's open-source framework: a clean task / dataset / solver / scorer split, with agent tasks and tool use first-class — the shape to grow a reusable harness in.

复现项目: An eval harness that gates a change — Label 100 real outputs by hand, build an LLM judge and measure its agreement with you, and make CI fail when a prompt or model change drops the score.

• Langfuse freeDone with Langfuse

  repo — 开源、可自托管进 k3s 的 trace / 会话 / 评分平台。价值不在看板，而在把生产 trace 变成能抽样、能打分、能回灌进评测集的数据——error analysis 的入口就在这里。

复现项目: 给一个在线 LLM 功能接 trace，再从 trace 回到评测集 — 给一个生产里的 LLM 功能每次调用记一条 trace（输入摘要、prompt 版本、模型、token、延迟、成本、用户有没有改译文）；一个月后抽 100 条做 error analysis，把失败模式写成新的评测用例。

• Prompt caching — Anthropic freeDone with Prompt caching — Anthropic

  docs — 供应商原生省钱手段里最大的一项：哪些前缀能缓存、各模型的最小可缓存长度、写入与命中的价格倍率，权威数字只在这页。读完就能算出共用一段长系统提示的调用该省多少。

复现项目: 给一个共用系统提示的生产调用加 prompt caching，拿账单验证 — 把共用的系统提示和术语表放进缓存前缀，跑一周，对比缓存命中 token 占比、每千次调用成本和 p50 延迟；批量回填类任务改走 Batch API，再比一次。

• OWASP Top 10 for LLM Applications 2025 freeDone with OWASP Top 10 for LLM Applications 2025

  docs — 十类风险一张清单——prompt injection、输出处理不当、过度代理、向量库弱点、无上限消耗。给自己的 agent 或 MCP server 做威胁建模时逐条过一遍，不会漏掉整类。

• My Lethal Trifecta talk at the Bay Area AI Security Meetup — Simon Willison freeDone with My Lethal Trifecta talk at the Bay Area AI Security Meetup — Simon Willison

  article — 一条判定规则就够记：私有数据、不可信内容、对外发送三者同在，数据就能被偷走。设计工具权限边界时先问「拿掉哪一个」，比堆检测器可靠。

复现项目: 给 ImageStep MCP 做一轮 prompt injection 红队 — 用 promptfoo 的 red team 对 MCP server 的 7 个工具跑注入用例（藏在文件名、URL 页面、工具返回里的指令），数 agent 越过权限边界或把数据发到外部的次数；修完再跑一轮。

• The TypeScript Handbook freeDone with The TypeScript Handbook

  docs — The language's own reference, short enough to read in an evening — enough to read and change the web and SDK code these roles ship to customers.

复现项目: Type an SDK for an API you use — Write a small typed client for a real API — request and response types, errors as values, one streaming endpoint — and publish it with its tests.