Anthropic's Skill-Creator Tool Receives Major Upgrade with New Evaluation System

Anthropic has updated its Skill-creator tool, a core component for developing custom AI capabilities within its Claude models. The update introduces an evaluation system designed to improve the reliability, performance, and accuracy of generated skills.

The Skill-creator acts as an official generator, allowing users to verbally describe desired functionalities and then convert them into executable skills for Anthropic's AI agents. The latest enhancements address previous limitations where generated skills operated as "black boxes" without clear performance metrics or trigger mechanisms.

Key Enhancements to Skill-Creator

The updated Skill-creator introduces four primary new capabilities:

1. Evaluation System: Provides direct feedback on whether a skill functions as intended after execution.

2. Benchmarking: Quantifies performance metrics such as pass rate, execution time, and token usage.

3. Multi-agent Parallel Testing: Enables independent testing of skills in clean environments, supporting A/B blind evaluations and preventing cross-contamination of results.

4. Description Tuning: Automatically refines skill descriptions to ensure accurate triggering, activating when appropriate and preventing unintended activations.

Previously, users lacked mechanisms to assess the quality or trigger logic of generated skills. The integration of a comprehensive evaluation system aims to guide development and optimization. Users can update their agents by sending a specific command to integrate the new Skill-creator version.

Practical Application and Optimization

An example demonstrates the new tool's capabilities by creating a skill to generate text transcripts from video links, including bilingual options for foreign-language content. After initial generation, the tool facilitates refinement, addressing issues like text formatting and readability.

A significant feature is the ability to optimize skill descriptions to prevent conflicts, particularly when multiple skills share similar trigger conditions. The evaluation system generates test queries, including edge cases, to determine correct triggering behavior. Users can review and adjust these judgments via a web interface.

The system then initiates an optimization loop, running up to five iterations. Each iteration tests and evaluates the skill, with progress reported periodically. The results are presented in a table, showing trigger success (green check) or failure (red cross) across various queries and description versions. The system divides samples into training and test sets (60% training, 40% test) to prevent overfitting and automatically writes the optimal description back to the skill's markdown file. Anthropic's internal tests on six document-related skills showed an increased trigger rate in five after optimization with the new Skill-creator.

Performance Evaluation and Benchmarking

Beyond trigger accuracy, the updated Skill-creator assesses a skill's performance on actual tasks. For the video transcription skill, the system reads the skill file, identifies core processes, and designs test scenarios with quantitative acceptance criteria. It then launches multiple independent sub-agents for parallel testing.

This parallel execution in clean environments, each with its own token count and time metrics, prevents context contamination that could skew results in sequential testing. The system also prepares quantitative scoring scripts, automatically checking output formats.

Evaluation results are presented on a dedicated page with "output" and "benchmark" tabs. The output tab displays test case results and allows users to provide feedback, which is stored for future skill improvements. The benchmark tab offers quantitative comparisons, such as a 100% pass rate with the skill versus 9% without, alongside token usage differences.

Anthropic cited an example where the evaluation process identified and helped resolve an error in a PDF processing skill that struggled with precise text placement in tables. The evaluation results are stored locally, enabling iterative improvement by focusing on previously identified issues during subsequent optimization cycles.

Skill Categorization and Future Outlook

The update supports two main types of skills:

• Capability Enhancement: Teaches AI models to perform tasks they are not inherently proficient at, often involving techniques beyond simple prompts.

• Coding Preference (Workflow): Guides AI models to follow specific rules or processes, such as organizing meeting minutes into a fixed format or generating weekly reports from disparate data sources.

Evaluation approaches differ for these types. Capability enhancement skills are assessed via A/B testing to determine if they remain necessary after model updates. Coding preference skills are evaluated on their adherence to specified processes and steps.

The enhanced Skill-creator aims to bring rigorous software development practices, including testing, benchmarking, and iterative improvement, to the AI skill creation process. This development is expected to foster further growth and refinement within the AI agent ecosystem.