hillclimb: a cheaper Claude API feature
Your costliest Claude API feature: a test set first, then cheaper models and lower effort step by step while accuracy holds.
1.7K saves on X, as of Posted
Read this guide: https://claude.dev/blog/automating-eval-design-and-hillclimbing/
Then list every place in this project that calls the Claude API, estimate which one costs the most, and let me pick one. Use build-eval from the claude-api skill to build a test set for it, and tell me roughly what it will cost in API fees before you run anything.
Once I've approved the test cases and the grading, run hillclimb on a new branch. The goal is to keep accuracy and cut cost. You can try lower effort and cheaper models, including Sonnet 5.5. The report should spell out three things:
1. What changed at each step, and how accuracy and cost per call moved.
2. Which cheaper setups got things wrong, and where.
3. Which sentences were removed from or added to the prompt.
Show me the report first. Don't merge anything until I confirm.Next Send it to Claude Code in the project that calls the Claude API.
Anthropic published a guide where they had Claude tune an AI feature round by round, and the same job ended up costing about 1/5 as much, with higher accuracy. You can have Claude Code follow it and put any feature in your product that calls the Claude API through the same process, with two commands.
It takes three steps. First, cut the extra steps and contradictory rules from the prompt. Then step down to cheaper models and lower effort, one level at a time. Finally, add the rules that were missing. Every step is checked against a set of cases it has never seen, and any change that lowers accuracy gets rolled back. In the end, accuracy went from 78.6% to 90.5%. The new Sonnet 5.5 is close to Opus 5.5 in capability but cheaper and faster, so if you want to use it, measure it the same way first.
Send this prompt to Claude Code.
Part of a playbook
Quota and modelsQuota runs out early: effort and models per job, the cache timed, API costs cut.OpenShare this tip
