JevCode技能建议
从 Nous Research 的 Hermes 目录中的 182 个技能中，为代理回合选择最多一个技能：一个 TypeSafe 请求会对所有技能进行排名，并判断该回合是否真的需要某个技能；第二个请求会正确读取排名前三的技能，并可以拒绝所有选项。获胜技能的名称会被写入代理的 sy
本文由机器翻译自 en，未经人工校对，仅供快速参考。
智能体通过截断并加载所有技能到系统消息中来选择技能，这会增加成本、降低技能选择的性能，并导致会话其余部分的上下文退化。我们通过每轮使用两个 TypeSafe 请求来解决这个问题：一个用于对技能进行排名，另一个用于验证选择，从而将错误的技能加载减少了一半以上。
拥有大量技能库的智能体几乎在没有任何信息的情况下做出选择。技能库以索引形式呈现：每个技能一行，描述被截断，以免完整文本挤占对话空间。此处使用的智能体框架 Hermes 默认将其截断为 60 个字符。例如，在该宽度下，编辑 .pptx 文件的技能与撰写它们的技能看起来几乎相同。如果要求制作一个演示文稿，智能体可能会加载错误的技能。在完全没有合适技能的轮次中，它仍可能加载其中一个，因为名称列表会诱导猜测。
本 cookbook 保留描述不变，采用渐进式披露的方式，先低成本地读取全部 182 个技能，然后详细读取其中三个。在决定是否加载任何技能之前，会先发起两个 TypeSafe 请求。第一个请求根据用户的当前轮次对技能库中的每个技能进行排名，并回答该轮次是否确实需要技能。第二个请求仅重新读取排名前三的技能，此时每个技能都附带其完整描述及指令的开头部分，并且可以拒绝所有这三个技能。
获胜技能的名称会被添加到该轮智能体系统提示中的一行额外内容中：
<skill_relevance>
Relevant to the current request: pptx-author. Ignore this if it does not fit what the user
actually asked for.
</skill_relevance>代理保留了其完整的索引和自身的判断，而这一行代码仅告知它首先查看哪个条目。名册本身永远不会改变，因此针对它的任何前缀缓存仍然有效。在对 claude-haiku-4-5-20251001 发出的 488 次请求中，使用了来自 Hermes 名册的技能：
| 节点 | 说明 | 所属分组 |
|---|---|---|
| 仅使用名册的代理 | 16.8% | 9.8% |
| 带有 TypeSafe 建议的代理 | 7.3% | 4.0% |
| 代理被给予正确答案 | 2.5% | 1.2% |
第三行表明犯错的底线并非为零，因为即使代理获得了正确的技能，它也不总是加载该技能，且无论选择方法多么优秀，都无法突破这一限制。
最终你会得到一个 suggest() 函数，它最多返回一个技能名称，一个用于包装系统提示中的 suggestion_block()，以及生成上述表格的测试框架，准备好指向你自己的名册。
流向：LR
| 从 | 条件 | 到 |
|---|---|---|
C1 |
调用 1 - 快速浏览所有 182 个技能 | 调用 1 - 快速浏览所有 182 个技能 |
Q1 |
选择：哪个技能合适？/ 全部 182 个，每行一个 | 调用 1 - 快速浏览所有 182 个技能 |
N1 |
Nouls：是否真的需要技能？/ · 基于其内容行动？/ · 遵循书面步骤？/ · 还是仅仅交谈？ | 调用 1 - 快速浏览所有 182 个技能 |
C2 |
调用 2 - 仔细阅读这 3 个 | 调用 2 - 仔细阅读这 3 个 |
Q2 |
选择：这 3 个中哪个？/ 现在使用真实细节 | 调用 2 - 仔细阅读这 3 个 |
N2 |
Nouls：每一个是否 / 真正做到了？ | 调用 2 - 仔细阅读这 3 个 |
| 节点 | 说明 | 所属分组 |
|---|---|---|
Q1 |
— | N1 |
Q2 |
— | N2 |
C1 |
前 3 名 | C2 |
C1 |
没有 / 适用 | STOP |
C2 |
没有合适的 | STOP |
C2 |
获胜者 | OUT |
设置
- 安装 TypeSafe 客户端、Anthropic 客户端以及共享的 cookbook 辅助工具。
- 设置 TypeSafe API 密钥，以及用于被测代理的 Anthropic 密钥。
pip install anthropic matplotlib ipython "typesafe-sdk>=0.5.7" cooksafe --extra-index-url https://pypi.typesafe.ai/
export TYPESAFE_API_KEY=your-key-here
export ANTHROPIC_API_KEY=your-key-here注意： 以下代码块是一个脚本，按顺序排列。若要跟随操作，请按照所示顺序将它们放入单个文件中。
缓存结果
JsonCache 会根据输入键值保存每次调用的结果，因此重新运行时会重现以下数值，而不是调用任何 API。删除 json_cache.json 即可进行实时运行。本次发布的运行使用了 jev-1.12 和 claude-haiku-4-5-20251001，渲染时间为 2026-07-31。
import json
import os
from collections import defaultdict
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from time import perf_counter
import anthropic
import matplotlib
import matplotlib.pyplot as plt
from matplotlib.ticker import PercentFormatter
from cooksafe import JsonCache, make_playground_link
from IPython.display import Markdown, display
from typesafe_sdk import Choice, Noul, TypeSafeClient
matplotlib.use("Agg") # headless render
TYPESAFE_MODEL = "jev-1.12"
AGENT_MODEL = (
"claude-haiku-4-5-20251001" # the agent under test, pinned so scores are stable
)
SHORTLIST = 3 # candidates carried from the first request into the second
EXCERPT_CHARS = (
700 # SKILL.md characters each candidate brings; the roster file stores 1600
)
GATE_THRESHOLD = (
0.30 # mean of the three request nouls, below which nothing is suggested
)
FITS_THRESHOLD = (
0.30 # a shortlist whose best "does this fit" noul is under this is dropped
)
WORKERS = 8 # small pool: enough to keep a live run to minutes, gentle on rate limits
assert EXCERPT_CHARS <= 1600, (
"the shipped roster file stores 1600 body characters per skill"
)
client = TypeSafeClient(
api_key=os.environ.get(
"TYPESAFE_API_KEY", "cache-only"
), # keyless kernels replay the cache
base_url=os.environ.get("TYPESAFE_ENDPOINT"),
timeout=120.0,
)
agent = anthropic.Anthropic(api_key=os.environ.get("ANTHROPIC_API_KEY", "cache-only"))
json_cache = JsonCache(Path("json_cache.json"))第一步：加载角色列表
hermes_roster.json 文件包含了 NousResearch/hermes-agent（MIT 许可证）在某个固定提交版本中的 182 项技能。每条记录包含技能的名称和类别、索引中显示的描述、完整描述，以及其 SKILL.md 文件的开头部分。
下方的索引及其上方的说明均从 Hermes 复制而来。
ROSTER = json.loads(Path("hermes_roster.json").read_text(encoding="utf-8"))
BY_NAME = {skill["name"]: skill for skill in ROSTER}
# Verbatim from hermes-agent agent/prompt_builder.py:build_skills_system_prompt.
PREAMBLE = (
"## Skills (mandatory)\n"
"Before replying, scan the skills below. If a skill matches or is even partially relevant "
"to your task, you MUST load it with skill_view(name) and follow its instructions. "
"Err on the side of loading — it is always better to have context you don't need "
"than to miss critical steps, pitfalls, or established workflows. "
"Skills contain specialized knowledge — API endpoints, tool-specific commands, "
"and proven workflows that outperform general-purpose approaches. Load the skill "
"even if you think you could handle the task with basic tools like web_search or terminal. "
"Skills also encode the user's preferred approach, conventions, and quality standards "
"for tasks like code review, planning, and testing — load them even for tasks you "
"already know how to do, because the skill defines how it should be done here.\n"
"Whenever the user asks you to configure, set up, install, enable, disable, modify, "
"or troubleshoot Hermes Agent itself — its CLI, config, models, providers, tools, "
"skills, voice, gateway, plugins, or any feature — load the `hermes-agent` skill "
"first. It has the actual commands (e.g. `hermes config set …`, `hermes tools`, "
"`hermes setup`) so you don't have to guess or invent workarounds.\n"
"If a skill has issues, fix it with skill_manage(action='patch').\n"
"After difficult/iterative tasks, offer to save as a skill. "
"If a skill you loaded was missing steps, had wrong commands, or needed "
"pitfalls you discovered, update it before finishing.\n"
"\n"
)
FOOTER = "\n\nOnly proceed without loading a skill if genuinely none are relevant to the task."
IDENTITY = (
"You are Hermes, a capable AI assistant with access to tools and a library "
"of skills. You help the user with coding, research, and everyday tasks.\n\n"
)
def render_index() -> str:
"""The body of <available_skills>: skills grouped by category, both sorted by name."""
by_category = defaultdict(list)
for skill in ROSTER:
by_category[skill["category"]].append(skill)
lines = []
for category in sorted(by_category):
lines.append(f" {category}:")
for skill in sorted(by_category[category], key=lambda s: s["name"]):
lines.append(f" - {skill['name']}: {skill['description']}")
return "\n".join(lines)
CATALOG_PROMPT = (
IDENTITY
+ PREAMBLE
+ "<available_skills>\n"
+ render_index()
+ "\n</available_skills>"
+ FOOTER
)
widths = [len(skill["description"]) for skill in ROSTER]
print(f"{len(ROSTER)} skills in {len({s['category'] for s in ROSTER})} categories")
print(f"roster prompt: {len(CATALOG_PROMPT):,} characters")
print(
f"index description: {sum(widths) / len(widths):.0f} characters on average, "
f"{max(widths)} at most"
)
print("\none category, as the agent reads it:")
index_lines = render_index().splitlines()
start = index_lines.index(" apple:")
end = next(
i
for i in range(start + 1, len(index_lines))
if not index_lines[i].startswith(" ")
)
print("\n".join(index_lines[start:end]))182 skills in 33 categories
roster prompt: 16,089 characters
index description: 54 characters on average, 60 at most
one category, as the agent reads it:
apple:
- apple-notes: Manage Apple Notes via memo CLI: create, search, edit.
- apple-reminders: Apple Reminders via remindctl: add, list, complete.
- findmy: Track Apple devices/AirTags via FindMy.app on macOS.
- imessage: Send and receive iMessages/SMS via the imsg CLI on macOS.步骤 2：对代理自身进行评分
requests.json 包含 488 个单轮请求，其中 315 个请求恰好被一个技能覆盖，其余 173 个请求未被任何技能覆盖。
这些被覆盖的请求是由 Claude Sonnet 5 根据每个技能各自的 SKILL.md 编写的，因此标签是可靠的，且这些请求比用户发送的请求更容易。
那 173 个未被覆盖的请求均旨在惩罚猜测行为：包括 85 个日常请求、42 个无技能服务的技术问题（例如 解释什么是单子），以及 46 个要求提供技能列表中不存在的特定功能的请求，例如在仅覆盖 X 且无其他技能的技能列表中要求 将此发布到 Mastodon。
评分仅读取代理的首次响应。这两个数字均为错误率，因此数值越低越好：
- 错误加载：在被覆盖的请求中，首次
skill_view调用未加载对应覆盖技能的占比。如果某轮对话未加载任何技能，则计为未命中。 - 不必要的加载：在未被覆盖的请求中，代理调用
skill_view的占比。
REQUESTS = json.loads(Path("requests.json").read_text(encoding="utf-8"))
POSITIVES = [p for p in REQUESTS if p["gold"]]
NEGATIVES = [p for p in REQUESTS if not p["gold"]]
print(
f"{len(REQUESTS)} requests: {len(POSITIVES)} covered by a skill "
f"({len({p['gold'] for p in POSITIVES})} distinct skills), {len(NEGATIVES)} covered by none"
)
print(f"\ncovered [{POSITIVES[0]['gold']}] {POSITIVES[0]['text']}")
print(f"uncovered {NEGATIVES[0]['text']}")488 requests: 315 covered by a skill (171 distinct skills), 173 covered by none
covered [1password] I've got a config.yaml with `{{ op://app-prod/db/password }}` placeholders in it — can you set up my project to pull the real values in at runtime instead of hardcoding them?
uncovered Add these three cards to our Trello backlog.建议内容位于系统提示词中独立的区块，置于角色列表之后而非包含在列表内部，因此角色列表文本在每一轮对话中保持一致，以维持前缀缓存的效果。
该代理仅配备一套最小化工具集，其中包括 skill_view，用于通过自由文本名称加载技能。名称必须与技能完全匹配，才能正确加载。
# Verbatim from hermes-agent tools/skills_tool.py:SKILL_VIEW_SCHEMA.
SKILL_VIEW_DESCRIPTION = (
"Skills allow for loading information about specific tasks and workflows, as "
"well as scripts and templates. Load a skill's full content or access its "
"linked files (references, templates, scripts). First call returns SKILL.md "
"content plus a 'linked_files' dict showing available references/templates/"
"scripts. To access those, call again with file_path parameter."
)
TOOLS = [
{
"name": "skill_view",
"description": SKILL_VIEW_DESCRIPTION,
"input_schema": {
"type": "object",
"properties": {
"name": {"type": "string", "description": "The skill name."}
},
"required": ["name"],
},
},
{
"name": "terminal",
"description": "Run a shell command on the user's machine and return its output.",
"input_schema": {
"type": "object",
"properties": {"command": {"type": "string"}},
"required": ["command"],
},
},
{
"name": "read_file",
"description": "Read a file from the user's filesystem.",
"input_schema": {
"type": "object",
"properties": {"path": {"type": "string"}},
"required": ["path"],
},
},
{
"name": "web_search",
"description": "Search the web and return result snippets.",
"input_schema": {
"type": "object",
"properties": {"query": {"type": "string"}},
"required": ["query"],
},
},
]
@json_cache
def run_turn(model: str, arm: str, request: str, suggestion: str) -> dict:
"""One measured turn. ``arm`` is in the key so each arm samples independently."""
system = [
{"type": "text", "text": CATALOG_PROMPT, "cache_control": {"type": "ephemeral"}}
]
if suggestion:
system.append({"type": "text", "text": suggestion}) # after the breakpoint
response = agent.messages.create(
model=model,
max_tokens=1024,
system=system,
tools=TOOLS,
messages=[{"role": "user", "content": request}],
)
usage = response.usage
return {
"loaded": [
str(block.input.get("name", ""))
for block in response.content
if block.type == "tool_use" and block.name == "skill_view"
],
"input_tokens": usage.input_tokens or 0,
"output_tokens": usage.output_tokens or 0,
}
def summarise(turns: dict[str, dict]) -> dict[str, float]:
"""Two failure rates: wrong loads on covered requests, needless ones on uncovered."""
hits = [turns[p["text"]]["loaded"][:1] == [p["gold"]] for p in POSITIVES]
over = [bool(turns[p["text"]]["loaded"]) for p in NEGATIVES]
return {
# both metrics are errors, so the two columns read the same direction
"wrong_load": 1 - sum(hits) / len(hits),
"needless_load": sum(over) / len(over),
}
def run_arm(arm: str, suggestions: dict[str, str]) -> dict[str, dict]:
"""One measured turn per request, in a small pool. 488 calls."""
texts = [request["text"] for request in REQUESTS]
with ThreadPoolExecutor(max_workers=WORKERS) as pool:
turns = pool.map(
lambda t: run_turn(AGENT_MODEL, arm, t, suggestions.get(t, "")), texts
)
return dict(zip(texts, turns))代理首先仅凭其角色列表运行，这是其当前的工作方式。它的两个错误率构成了其余食谱所衡量的基线。
baseline = run_arm("baseline", {})
base_scores = summarise(baseline)
print(
f"wrong loads {base_scores['wrong_load']:.1%} ({len(POSITIVES)} covered requests)"
)
print(
f"needless loads {base_scores['needless_load']:.1%} ({len(NEGATIVES)} uncovered requests)"
)
# where the wrong loads land: a neighbour of the right skill, or somewhere unrelated?
misses = [
(p["gold"], baseline[p["text"]]["loaded"][0])
for p in POSITIVES
if baseline[p["text"]]["loaded"] and baseline[p["text"]]["loaded"][0] != p["gold"]
]
same_category = sum(
1
for gold, got in misses
if got in BY_NAME and BY_NAME[got]["category"] == BY_NAME[gold]["category"]
)
print(
f"\nof {len(misses)} wrong first picks, {same_category} came from the right skill's own "
f"category"
)wrong loads 16.8% (315 covered requests)
needless loads 9.8% (173 uncovered requests)
of 36 wrong first picks, 10 came from the right skill's own category错误的负载落入正确技能所属类别的频率远高于随机概率所预期的水平，因此难点在于区分少数几个外观相似的技能。代理已经大致在正确的范围内进行搜索。
步骤 3：对整个技能库进行排序
一个请求包含两种类型的问题：
which是一个Choice问题， 针对全部 182 个技能名称，并以索引描述作为每个选项的判定标准（与代理自身接收到的文本相同）。其概率即为排序依据。- 三个关于该请求的
Noul问题，如下所示，分别从不同角度询问是否希望采取操作而非提供解释。prose_suffices则从相反的角度进行计数。它们的平均值决定是否需要提出任何建议，若低于 0.30 则不提出任何建议。
这两部分通过一次请求发出，因此排序和检查仅消耗一次往返时间。
编写这三个问题以询问是否希望执行操作。关于主题内容的问题无法区分“解释什么是 monad”与需要调用技能的操作，因为两者都属于软件范畴。
一个 Choice 问题足以舒适地容纳如此规模的技能库。如果规模稍大几倍，则需将其拆分为多个块，分别对每个块进行排序，然后对胜出者再次运行此简短列表筛选步骤。
CHOICE_INSTRUCTIONS = (
"Which of these skills, if any, is the right one to load to help with the "
"user's latest request?"
)
GATE_QUESTIONS = {
"acts_on_user_system": (
"Is the assistant being asked to act on the user's files, accounts, devices, "
"or online services, rather than only to explain or advise?"
),
"would_follow_documented_procedure": (
"Would a careful expert answering this consult a specific documented procedure "
"or set of commands, rather than answering from general understanding?"
),
"prose_suffices": (
"Could a knowledgeable generalist fully satisfy this request in prose, with "
"no tools, no documentation, and no access to the user's files or accounts?"
),
}
INVERTED = {"prose_suffices"} # a yes here points away from needing a skill
def build_state(request: str) -> dict:
return {"request": request, "recent_context": ""}
@json_cache
def rank_wide(request: str) -> dict:
"""Request 1: rank all 182 skills, and score the request for whether a skill applies."""
questions = {
"which": Choice(
instructions=CHOICE_INSTRUCTIONS,
criteria={skill["name"]: skill["description"] for skill in ROSTER},
)
}
for key, text in GATE_QUESTIONS.items():
questions[f"gate::{key}"] = Noul(instructions=text)
started = perf_counter()
response = client.system_one(
state=build_state(request), questions=questions, model=TYPESAFE_MODEL
)
ranked = sorted(
response.answers["which"].probabilities.items(), key=lambda kv: -kv[1]
)
values = {
key.removeprefix("gate::"): answer.noul
for key, answer in response.answers.items()
if key.startswith("gate::")
}
oriented = [(1.0 - v) if k in INVERTED else v for k, v in values.items()]
return {
"ranked": ranked[
:12
], # more than any shortlist needs, and keeps the cache small
"gate": sum(oriented) / len(oriented),
"values": values,
"seconds": round(perf_counter() - started, 2),
"input_tokens": response.usage.input_tokens or 0,
"output_tokens": response.usage.output_tokens or 0,
}
DEMO = [
"Can you save this recipe as a new note in my 'Recipes' folder in Notes.app so it syncs"
" to my phone? Just write it up in whatever editor pops up.",
"Can you put together a pitch deck skeleton (cover, situation overview, comps, precedent"
" transactions, DCF, LBO) as a .pptx, using our firm-template.pptx for branding and"
" footnoting each valuation number back to the cell it came from in the model?",
"Post this announcement to my Mastodon account.",
]
for request in DEMO:
wide = rank_wide(request)
verdict = "suggest" if wide["gate"] >= GATE_THRESHOLD else "stay quiet"
print(f'"{request[:78]}"')
print(f" needs a skill {wide['gate']:.2f} -> {verdict} ({wide['seconds']}s)")
for name, probability in wide["ranked"][:SHORTLIST]:
print(f" {probability:.3f} {name:<38}{BY_NAME[name]['description']}")
print()"Can you save this recipe as a new note in my 'Recipes' folder in Notes.app so "
needs a skill 0.75 -> suggest (0.31s)
0.990 apple-notes Manage Apple Notes via memo CLI: create, search, edit.
0.010 computer-use Drive the user's desktop in the background — clicking, ty...
0.000 concept-diagrams Generate flat, minimal educational SVG visuals as HTML.
"Can you put together a pitch deck skeleton (cover, situation overview, comps, "
needs a skill 0.76 -> suggest (0.16s)
0.700 powerpoint Create, read, edit .pptx decks, slides, notes, templates.
0.300 pptx-author Build PowerPoint decks headless with python-pptx.
0.000 chroma Embedding database for RAG and semantic search.
"Post this announcement to my Mastodon account."
needs a skill 0.78 -> suggest (0.16s)
0.550 xurl X/Twitter via xurl CLI: raw post search, posting, DM, media.
0.140 computer-use Drive the user's desktop in the background — clicking, ty...
0.080 openhands Delegate coding to OpenHands CLI (model-agnostic, LiteLLM).Notes.app 的请求含义明确，其首选选项是正确的。任何排序操作都无法挽救 Mastodon 选项：这三个问题表明需要一项技能，因为向账户发布内容是一种操作，而针对 X 的发布技能与针对 Mastodon 的缺失技能相比，前者无论如何都会胜出。
接下来是演示文稿（deck）的处理。两个领先者都是 .pptx 技能，在 60 个字符的限制下，宽泛的 Choice 问题将编辑技能置于创作技能之前，但这与请求创作演示文稿的意图不符。
步骤 4：重新排序前三名
三个选项为完整描述以及每个技能各自的 SKILL.md 开头留出了空间，因此第二个请求将相同的问题应用于更有力的证据：
which是一个针对候选列表的Choice问题，以上述较长的文本作为每个选项的判断标准。fits::{name}是针对每个候选者的单个Noul问题：该技能是否执行了请求中指定的具体操作？每个问题独立回答，因此它们可能都返回低分，如果候选列表中最高分低于 0.30，则该列表将被完全丢弃。
RERANK_INSTRUCTIONS = (
"Exactly one of these skills is the right one to load for the user's latest "
"request. Which one? Read what each actually does, not just its name."
)
def rerank_criteria(names: tuple[str, ...], excerpt: int) -> dict[str, str]:
return {
name: f"{BY_NAME[name]['description_full']} — {BY_NAME[name]['body'][:excerpt]}"
for name in names
}
def rerank_questions(names: tuple[str, ...], excerpt: int) -> dict:
questions = {
"which": Choice(
instructions=RERANK_INSTRUCTIONS, criteria=rerank_criteria(names, excerpt)
)
}
for name in names:
questions[f"fits::{name}"] = Noul(
instructions=(
f"Does the skill '{name}' do the specific thing the user's request asks "
f"for? It is described as: {BY_NAME[name]['description_full']}"
)
)
return questions
@json_cache
def rerank(request: str, names: tuple[str, ...], excerpt: int) -> dict:
"""Request 2: the same Choice over a shortlist, plus one absolute noul per candidate."""
started = perf_counter()
response = client.system_one(
state=build_state(request),
questions=rerank_questions(names, excerpt),
model=TYPESAFE_MODEL,
)
return {
"winner": response.answers["which"].choice,
"fits": {
key.removeprefix("fits::"): answer.noul
for key, answer in response.answers.items()
if key.startswith("fits::")
},
"seconds": round(perf_counter() - started, 2),
"input_tokens": response.usage.input_tokens or 0,
"output_tokens": response.usage.output_tokens or 0,
}
for request in DEMO:
wide = rank_wide(request)
if wide["gate"] < GATE_THRESHOLD:
print(f'"{request[:78]}"\n scored too low, nothing suggested\n')
continue
shortlist = tuple(name for name, _ in wide["ranked"][:SHORTLIST])
result = rerank(request, shortlist, EXCERPT_CHARS)
best = max(result["fits"].values())
verdict = result["winner"] if best >= FITS_THRESHOLD else "nothing fits"
print(f'"{request[:78]}"')
print(f" was {shortlist[0]} -> {verdict} ({result['seconds']}s)")
for name in shortlist:
print(f" fits {result['fits'][name]:.2f} {name}")
print()"Can you save this recipe as a new note in my 'Recipes' folder in Notes.app so "
was apple-notes -> apple-notes (0.12s)
fits 0.60 apple-notes
fits 0.54 computer-use
fits 0.01 concept-diagrams
"Can you put together a pitch deck skeleton (cover, situation overview, comps, "
was powerpoint -> pptx-author (0.09s)
fits 0.73 powerpoint
fits 0.38 pptx-author
fits 0.02 chroma
"Post this announcement to my Mastodon account."
was xurl -> xurl (0.09s)
fits 0.56 xurl
fits 0.38 computer-use
fits 0.05 openhands两个 .pptx 技能各自独立处理其对应的文本：当请求涉及演示文稿时，会路由到创作技能。
fits 无核（nouls）与 Choice 在此存在分歧：无核对编辑技能给出了更高的评分，而 Choice 则选择了创作技能。它们正在决定不同的事项。Choice 确定的是哪个技能，而无核确定的是是否要输出任何内容。
Mastodon 请求通过了两项检查：其最佳 fits 无核得分高于 0.30，因此该配方建议对关于 Mastodon 的请求使用 X 技能。大多数类似请求都会被捕获。第二遍筛选只能拒绝宽泛排名所传递的结果，而在此例中，结果是三个接近但未达标的候选项。
以下函数展示了完整的配方：包含两个请求和两个阈值，最多返回一个技能名称。
要将其指向你自己的技能列表，请替换 hermes_roster.json。上述每个问题都从该文件中读取 name、description、description_full 和 body，且其他部分均不感知 Hermes。
def suggest(request: str) -> tuple[str, ...]:
"""At most one skill name for a request, or () for "nothing here applies"."""
wide = rank_wide(request)
if wide["gate"] < GATE_THRESHOLD:
return ()
shortlist = tuple(name for name, _ in wide["ranked"][:SHORTLIST])
result = rerank(request, shortlist, EXCERPT_CHARS)
if max(result["fits"].values()) < FITS_THRESHOLD:
return ()
return (result["winner"],)
def suggestion_block(names: tuple[str, ...]) -> str:
"""What gets appended after the roster, in the suggestion.
This string is a measured input rather than prose: it goes to the agent, so it is part
of every graded turn's cache key. Editing a word here silently invalidates the shipped
results and costs a live re-run to restore them.
"""
body = (
f"Relevant to the current request: {', '.join(names)}. Ignore this if it does not "
"fit what the user actually asked for."
if names
else "No skill in the roster appears relevant to this request."
)
return f"\n\n<skill_relevance>\n{body}\n</skill_relevance>"
print(suggestion_block(suggest(DEMO[1])))
print(suggestion_block(suggest(DEMO[2])))
<skill_relevance>
Relevant to the current request: pptx-author. Ignore this if it does not fit what the user actually asked for.
</skill_relevance>
<skill_relevance>
Relevant to the current request: xurl. Ignore this if it does not fit what the user actually asked for.
</skill_relevance>步骤 5：测量建议
每个请求都会向智能体发送三次，每次测量一个回合。这些运行的区别仅在于告知智能体的内容不同：
| 系统提示中的内容 | |
|---|---|
| 仅智能体 | 无 |
| 带有建议的智能体 | suggest() 返回的任何内容 |
| 提供答案的智能体 | 覆盖技能的名称，若无则返回“无适用项” |
第三种情况是无法实现的；它是其他两种情况所参照的基准上限。
该建议的措辞承担着两项任务。它表明建议可以被忽略，因为即使建议错误，强行推进也能获得合规性，而错误的建议比没有建议更糟。此外，即使没有可建议的内容，一个回合仍会发送一条说明此情况的句子；如果完全不发送任何内容，将会使角色列表自身的“倾向于加载”指令失去制衡。
texts = [request["text"] for request in REQUESTS]
with ThreadPoolExecutor(max_workers=WORKERS) as pool: # up to 488 x 2 TypeSafe requests
suggested = dict(zip(texts, pool.map(suggest, texts)))
WIDE = {text: rank_wide(text) for text in texts} # all cache hits now; reused below
arms = {
"baseline": {},
"TypeSafe": {
request["text"]: suggestion_block(suggested[request["text"]])
for request in REQUESTS
},
"oracle": {
request["text"]: suggestion_block((request["gold"],) if request["gold"] else ())
for request in REQUESTS
},
}
scores = {
arm: summarise(run_arm(arm, suggestions)) for arm, suggestions in arms.items()
}
print(f"{'run':<10}{'wrong loads':>13}{'needless loads':>16}")
for arm, row in scores.items():
print(f"{arm:<10}{row['wrong_load']:>13.1%}{row['needless_load']:>16.1%}")
def fewer(metric: str) -> str:
"""The plain ratio between the two arms' error rates."""
return f"{scores['baseline'][metric] / scores['TypeSafe'][metric]:.1f}x fewer"
print(
f"\nbaseline -> TypeSafe: {fewer('wrong_load')} wrong loads, "
f"{fewer('needless_load')} needless ones"
)run wrong loads needless loads
baseline 16.8% 9.8%
TypeSafe 7.3% 4.0%
oracle 2.5% 1.2%
baseline -> TypeSafe: 2.3x fewer wrong loads, 2.4x fewer needless onesmoved = [
(
baseline[p["text"]]["loaded"][:1] == [p["gold"]],
run_turn(AGENT_MODEL, "TypeSafe", p["text"], arms["TypeSafe"][p["text"]])[
"loaded"
][:1]
== [p["gold"]],
)
for p in POSITIVES
]
print(
f"of {len(POSITIVES)} covered requests: {sum(not b and a for b, a in moved)} the suggestion "
f"fixed, {sum(b and not a for b, a in moved)} it broke"
)of 315 covered requests: 37 the suggestion fixed, 7 it broke该建议修复了更多请求，但同时也破坏了一些代理原本就能正确处理的情况。一个置信度高的错误建议比完全没有建议更具说服力，这就是在转向前引入建议所付出的代价。
SURFACE, INK, INK2, MUTED = "#fcfcfb", "#0b0b0b", "#52514e", "#898781"
GRID, AXIS, BLUE, ORANGE = "#e1e0d9", "#c3c2b7", "#2a78d6", "#eb6834"
ARM_COLOR = {"baseline": BLUE, "TypeSafe": ORANGE, "oracle": MUTED}
def style(ax):
ax.set_facecolor(SURFACE)
for side in ("top", "right"):
ax.spines[side].set_visible(False)
for side in ("left", "bottom"):
ax.spines[side].set_color(AXIS)
ax.tick_params(colors=MUTED, labelcolor=INK2, labelsize=9)
ax.set_axisbelow(True)
panels = [
("wrong_load", f"wrong loads\n{len(POSITIVES)} covered requests"),
("needless_load", f"needless loads\n{len(NEGATIVES)} uncovered requests"),
]
names = list(scores)
fig, axes = plt.subplots(1, 2, figsize=(8.4, 3.6), facecolor=SURFACE)
for ax, (metric, title) in zip(axes, panels):
style(ax)
ax.grid(axis="y", color=GRID, linewidth=0.8)
values = [scores[arm][metric] for arm in names]
bars = ax.bar(
names,
values,
0.58,
color=[ARM_COLOR[arm] for arm in names],
# the oracle is a ceiling, not a competitor: gray, and hatched so it never depends
# on colour alone
hatch=["", "", "///"],
edgecolor=SURFACE,
linewidth=1.2,
)
ax.bar_label(
bars,
labels=[f"{v:.1%}" for v in values],
padding=3,
color=INK2,
fontsize=9,
)
ax.set_title(title, loc="left", color=INK2, fontsize=9.5)
ax.set_ylim(0, max(values) * 1.28)
ax.yaxis.set_major_formatter(PercentFormatter(xmax=1, decimals=0))
ax.set_ylabel("% of those requests - lower is better", color=INK2, fontsize=9)
fig.suptitle(
f"Hermes' {len(ROSTER)}-skill roster, {len(REQUESTS)} requests, {AGENT_MODEL}",
x=0.02,
ha="left",
color=INK,
fontsize=11,
)
fig.tight_layout()
display(fig)
plt.close(fig)
结果分析
- 错误负载从 16.8% 降至 7.3%，非必要负载从 9.8% 降至 4.0%，这解释了从截断索引中猜测答案与直接提供答案之间的大部分差距。
- 某些原本由代理独立正确处理的请求，在附加建议后反而变为错误。具体数量见上文。
当你的代理需要处理大量候选项时，可参考此模式：先对所有候选项进行低成本排序，再对其中两到三个进行详细审查。任一步骤都可能返回空结果。
在 Playground 中打开
使用每个候选项的完整描述和正文片段作为标准，为第 4 步中的 deck 请求构建一个 Playground 链接。
demo_shortlist = tuple(name for name, _ in rank_wide(DEMO[1])["ranked"][:SHORTLIST])
playground_link = make_playground_link(
build_state(DEMO[1]),
rerank_questions(demo_shortlist, EXCERPT_CHARS),
models=[TYPESAFE_MODEL],
)
display(
Markdown(
f"🔗 [Open the shortlist + questions in the TypeSafe playground]({playground_link})"
)
)下一步
相同的结构也出现在其他地方： 意图路由 用于将请求路由到处理器而非技能，置信度 用于选择两个阈值，以及 推测性扇出 用于在单个请求中处理所有问题。