Anthropic adds eval-design and hillclimbing guidance to Claude Code's claude-api skill

What changed

Source summary

Anthropic has updated the claude-api skill in Claude Code with eval-design and hillclimbing guidance, applied through its build-eval and hillclimb commands: /claude-api build-eval walks you through generating an evaluation set for a specific problem inside your codebase, interviewing you and pausing for you to approve the examples and the grader it proposes, and /claude-api hillclimb then uses that evaluation to iteratively improve your application (for example tuning prompts or effort settings) one change at a time — splitting the evaluation at random into a training set it can read and a held-out test set it never sees, to guard against overfitting. Run claude update first to get the latest version of the commands in the skill shipped inside Claude Code. The source post is dated 28 September 2026; it does not establish when the skill update shipped or when the commands first launched, and states no plan, pricing, region or rollout restrictions.

Why it matters to your work

Practical interpretation

If your engineering team uses Claude Code to build AI-powered features, the updated claude-api skill's guidance for its build-eval and hillclimb commands helps you test and improve them with more confidence: /claude-api build-eval interviews you to generate an evaluation set for a specific problem and gets your approval on the examples and grading approach before the baseline evaluation, and /claude-api hillclimb then tunes your application against that evaluation one change at a time (for example prompts or effort settings), holding back a test set it never sees so you can tell whether a change genuinely helped or just overfit the examples.

When this could help

Useful when you want to measure and improve an AI feature's quality systematically — for example tuning prompts, effort settings, tool descriptions or harness code — instead of changing things by trial and error with no way to tell if a change genuinely helped.

The claude-api skill, updated with eval-design and hillclimbing guidance, uses its build-eval and hillclimb commands to give you a guided way to measure and improve an AI feature. /claude-api build-eval interviews you to build an evaluation set for a specific problem inside your codebase, and pauses for you to approve the examples and the grader before it runs the baseline evaluation. /claude-api hillclimb then asks what you want to optimise (for example performance, or cost while performance holds) ; you choose the surfaces it may change. Hillclimb splits the evaluation at random into a training set it can read and a held-out test set it never sees, and iterates against that split one change at a time to guard against overfitting.

How to get started

1. Run `claude update` to get the latest Claude Code, which ships the claude-api skill's build-eval and hillclimb commands.

   Provider guidance ↗

2. Run `/claude-api build-eval` to generate an evaluation set for a particular problem inside your codebase. Claude interviews you, builds the eval, and pauses at specific points for you to approve the examples it has chosen and the grader it proposes before moving on.

   Provider guidance ↗

3. Run `/claude-api hillclimb` once you have an evaluation. Claude asks what you want to optimise, such as performance or cost while performance holds. Choose the surfaces it may change, including your system prompt, skills or instruction files, tool descriptions, model choice, effort and other API parameters, or harness code.

   Provider guidance ↗

Build it into your routine

SCALA suggested workflow

1. Before trusting the grader

   Read a sample of scored transcripts yourself before believing the evaluator — scoring failures are one of the most common ways an evaluation is misconfigured.

2. Watching for overfitting

   Watch the train and test results. If train scores improve while test scores stay flat, Claude suspects overfitting and reverts the patch.

3. Before merging a change

   If hillclimb reports that a gain is within noise, it recommends against merging. Review that finding before deciding whether to keep the change.

4. Before running hillclimb

   Hillclimb asks what you want to optimise. You choose the surfaces it may change; step 3 lists the options.

What to check

• Hillclimbing tunes parameters such as effort or prompts against an evaluation set; the source stresses splitting cases into a train set the hillclimber may read and a held-out test set it never sees, specifically to catch overfitting. The source describes this split as part of hillclimb. Source ↗
• Hillclimb leaves your code at the version that did best on the held-out test set for your goal, and reports the test result against the baseline with confidence intervals. Check those intervals yourself, and treat a reported gain that's within noise as a reason not to merge. Source ↗
• If the baseline evaluation already scores about 95% or higher, the skill warns that there isn't much headroom left and suggests aiming hillclimb at cost or latency rather than quality. Source ↗
• 28 September 2026 is the source post date, not a confirmed first-launch date for the commands or a separately stated ship date for the skill update. Source ↗