JevCode结构恢复
从丢失了格式信息的纯文本中重建 Markdown，涉及两个请求：一个用于将硬换行的行重新拼接，另一个用于对每个代码块（标题、列表、代码、提示框）进行分类，并仅在相关时读取配套问题。
本文由机器翻译自 en，未经人工校对，仅供快速参考。
本食谱处理的是已去除标记的纯文本（句子中间被硬换行，无标题标记，无列表符号），并将其重构为 Markdown 格式：包括标题、段落、列表、引用、代码和提示框。输入内容是一份处于该状态的团队备忘录。
文本生成模型可以将文本重写为 Markdown，但重写过程可能会改变原文用词。在此方案中，模型从不生成文本：它仅针对文档回答狭义问题（这行是否承接了上一句的中间部分？此块内容属于何种类型？），而渲染工作由代码完成。因此，输出的每个字符都源自输入，且每个判断都附带概率。
整个处理流程对每份文档执行两次顺序进行的 API 请求：
- 第一遍，拼接（stitch）： 针对相邻的行对，发起一个
Noul问题（是/否问题，其答案表示“是”为正确的概率），询问换行符是否将句子截断在两行之间。所有行对包含在单个请求中，并合并那些承接截断句子的行以形成块。 - 第二遍，分类（classify）： 针对每个合并后的块，发起一个
Choice问题（从列表中选择一个选项，并为每个选项提供概率），在标题、段落、列表项、引用、代码或提示框（note、tip 或 warning，即与正文分离的注释、提示或警告）之间进行选择。由于这些块仅在第一遍回答完成后才存在，因此这是第二次请求；它还包含每个块的伴随问题（标题级别、步骤顺序、提示框类型），这些问题的答案仅在块类型使其相关时才会被读取。 - 直接证据保留在代码中。 空行和显式标记（
-、1.、#）由代码读取，从不发送给模型重新考量；本备忘录保留了空行，但丢失了所有标记。模型仅接收代码无法从文本中直接回答的问题。
所有行为均由第二遍问题的标准指定：三个包含单行描述的字典，以及 classify_questions 内步骤问题的真/假标准。其余代码是围绕这些标准的辅助逻辑。成本和延迟数据见附录：两次往返，10,211 个 token，0.8 秒，本备忘录费用为 $0.0015。
环境配置
pip install ipython "typesafe-sdk>=0.5.7" cooksafe --extra-index-url https://pypi.typesafe.ai/然后设置 TYPESAFE_API_KEY。每次 API 调用都会缓存在 json_cache.json 中，该文件随 cookbook 一起提供，因此重新渲染时会重放已发布的数值，而不会调用 API。删除该文件即可重新实时运行所有内容。
import os
import re
import urllib.request
from pathlib import Path
from time import perf_counter
from cooksafe import JsonCache, make_playground_link
from IPython.display import Markdown, display
from typesafe_sdk import Choice, Noul, NoulCriteria, TypeSafeClient
TYPESAFE_MODEL = "jev-1.12"
PRICE = (0.042, 0.00) # $ per 1M tokens (input, output); TypeSafe jev-1.12 as of 2026-09
client = TypeSafeClient(api_key=os.environ["TYPESAFE_API_KEY"], timeout=120.0)
json_cache = JsonCache(Path("json_cache.json"))文档：一份丢失了格式的团队备忘录
测试文档是一份关于构建系统迁移的备忘录，以它到达纯文本收件箱时的状态呈现：段落被硬换行截断，一行裸 Shell 命令，两个没有项目符号或编号的列表，以及一条没有任何标记的警告。文本从固定的 Gist 中获取，以确保 cookbook 中的数字保持可复现。
GIST = (
"https://gist.githubusercontent.com/eugene-shvarts/6df7daf97233bf92bcdd6b386a0fa561"
"/raw/5da03690611fb6ddcbaabdb91fb9f91d9751b113/build-memo.txt"
)
@json_cache
def fetch_document(url: str) -> str:
request = urllib.request.Request(url, headers={"User-Agent": "typesafe-cookbook/1.0"})
with urllib.request.urlopen(request) as response:
return response.read().decode()
RAW = fetch_document(GIST)
print(RAW[:560])Migration to the new build system
Hi everyone, quick heads up about the build system migration that is
happening next week. We have been running the new pipeline in shadow
mode for three weeks and the results look solid, so it is time to
make the switch for real.
What changes for you
The old make targets keep working until the end of the month. The new
entrypoint is a single command that wraps everything, including the
docs build that used to be separate.
bun run build
Generated artifacts no longer need to be committed. The new pipeline
uploads them行拆分、空行跟踪和 ID 标记均在代码中完成，不涉及模型。
每一行都会获得一个简短的 ID（L014| ）；这些 ID 是普通文本，模型将其作为 state 的一部分进行读取，问题和答案通过这些 ID 引用行（与 semantic search cookbook 相同的方案）。
def to_lines(text: str) -> list[dict]:
lines, gap = [], False
for raw in text.split("\n"):
stripped = re.sub(r"[\t ]+", " ", raw).strip()
if not stripped:
gap = bool(lines) # a leading blank is not a break
continue
lines.append({"text": stripped, "gap": gap})
gap = False
return lines
def tag(items: list[dict], prefix: str) -> str:
return "\n".join(
f"{chr(10) if item['gap'] else ''}{prefix}{i:03d}| {item['text']}"
for i, item in enumerate(items)
)
def line_id(i: int) -> str:
return f"L{i:03d}"
def block_id(i: int) -> str:
return f"B{i:03d}"
LINES = to_lines(RAW)
print(f"{len(LINES)} non-blank lines. The model sees, e.g.:")
print("\n".join(tag(LINES, "L").splitlines()[19:24]))28 non-blank lines. The model sees, e.g.:
L013| The cutover touches three teams, so check whether you are on this
L014| list before you plan anything for Monday:
L015| The platform team
L016| The web client team
L017| Whoever still owns the release tooling第一遍：拼接拆分后的句子
每个 Noul 问题针对相邻的行对，所有行对包含在同一个请求中；由空行分隔的行对将被跳过。该问题经过精心设计，范围非常狭窄（“这一行是否承接了上一句的半句话？”），这接近于对文本的客观事实判断。附录部分涵盖了措辞选择以及合并阈值的推导过程。
def join_question(i: int) -> Noul:
return Noul(
instructions=f"Does line {line_id(i)} pick up mid-sentence, continuing a sentence left unfinished at the end of line {line_id(i - 1)}?",
criteria=NoulCriteria(
true="The line starts in the middle of a sentence that began on the previous line - the line break tore the sentence apart",
false="The line begins a new sentence, item, heading, or thought of its own",
),
)
@json_cache
def stitch(wording: str = "mid-sentence") -> dict:
make = join_question if wording == "mid-sentence" else naive_join_question
questions = {line_id(i): make(i) for i in range(1, len(LINES)) if not LINES[i]["gap"]}
started = perf_counter()
response = client.system_one(
state=tag(LINES, "L"), questions=questions, model=TYPESAFE_MODEL
)
return {
"joins": [
response.answers[line_id(i)].noul if line_id(i) in response.answers else 0.0
for i in range(len(LINES))
],
"seconds": round(perf_counter() - started, 2),
"usage": [response.usage.input_tokens, response.usage.output_tokens],
}
result = stitch()
print(f"{sum(1 for l in LINES if not l['gap']) - 1} pair questions, one request, "
f"{result['seconds']}s")16 pair questions, one request, 0.32s合并的阈值取决于上一行的结尾方式。在悬空行（即没有句末标点符号的行）之后，合并概率达到 0.2 或以上即可合并该对；在句末标点符号（. ! ? : ;）之后，阈值则提高至 0.5。附录详细阐述了这两个数值背后的概率依据。
JOIN_AFTER_DANGLING, JOIN_AFTER_TERMINAL = 0.2, 0.5
def ends_terminal(text: str) -> bool:
return re.search(r'[.!?:;…]["\')\]]*$', text) is not None
def merge(joins: list[float]) -> list[dict]:
blocks = []
for i, line in enumerate(LINES):
bar = (
JOIN_AFTER_TERMINAL
if i and ends_terminal(LINES[i - 1]["text"])
else JOIN_AFTER_DANGLING
)
if blocks and not line["gap"] and joins[i] >= bar:
blocks[-1]["text"] += " " + line["text"]
blocks[-1]["lines"].append(i)
else:
blocks.append({"text": line["text"], "lines": [i], "gap": line["gap"]})
return blocks
blocks = merge(result["joins"])
healed = len(LINES) - len(blocks)
print(f"{len(LINES)} lines -> {len(blocks)} blocks ({healed} line breaks healed)")
for i, block in enumerate(blocks):
n = len(block["lines"])
print(f"{block_id(i)} {n} line{'s' if n > 1 else ' '} {block['text'][:62]}")28 lines -> 17 blocks (11 line breaks healed)
B000 1 line Migration to the new build system
B001 4 lines Hi everyone, quick heads up about the build system migration t
B002 1 line What changes for you
B003 3 lines The old make targets keep working until the end of the month.
B004 1 line bun run build
B005 3 lines Generated artifacts no longer need to be committed. The new pi
B006 2 lines The cutover touches three teams, so check whether you are on t
B007 1 line The platform team
B008 1 line The web client team
B009 1 line Whoever still owns the release tooling
B010 1 line Things to do before Monday
B011 1 line Update your local toolchain to version 2.4 or later
B012 1 line Delete the old build cache directory
B013 1 line Run the doctor script and fix anything it flags
B014 3 lines If the doctor script reports a red result on the toolchain che
B015 2 lines As Dana put it in the kickoff, "a migration nobody notices is
B016 1 line Thanks, and shout if anything looks off.第二遍：分类代码块
每个拼接后的代码块都会获得一个 Choice 问题：这是什么类型的内容？ 以下 classify_questions 中的这三个字典，以及步骤问题中的真/假条件，构成了分类器的完整规范。除此之外没有其他逻辑。要将此管道适配到您自己的文档，请编辑这些描述。
TYPE_CRITERIA = {
"heading": "A short label or title that names the document or the section that follows it - not a full sentence of content",
"paragraph": "Running prose: one or more complete sentences of explanatory or narrative text",
"list_item": "One entry in a list of parallel items - an ingredient, a feature, a task, an attendee; reads as one of several sibling entries",
"quote": "Words attributed to a person or source - quoted speech, a citation, an excerpt someone else wrote",
"code": "Computer code, a shell command, terminal output, or a config snippet meant to be read verbatim",
"callout": "A warning, tip, or important note that interrupts the flow to flag something the reader must not miss",
}
HLEVEL_CRITERIA = {
"title": "The title of the whole document",
"section": "A major section heading within the document",
"subsection": "A minor heading nested under a section",
}
CALLOUT_CRITERIA = {
"note": "Neutral extra information the reader should be aware of",
"tip": "A helpful suggestion or shortcut that makes things easier",
"warning": "A caution about something that can go wrong or cause harm",
}以下内容均为底层实现细节：构建问题、发送一次请求、读取返回的答案。如果类型返回为 heading，渲染器需要知道标题层级；如果为 list_item，则需要知道顺序是否重要；如果为 callout，则需要知道是哪一种类型。由于这些类型尚未确定，等待它们意味着需要第三轮往返通信，因此这些配套问题会在同一请求中提前提出。大多数这些答案实际上不会被读取：段落的步骤概率毫无意义，会被直接忽略。额外增加一个问题带来的开销很小，因为状态包含了大部分 token，并且无论如何都只需发送一次，而额外增加一轮往返通信则会带来完整的请求延迟。
HEADING_MAX_CHARS = 90 # longer blocks can't render as headings, so don't ask
def classify_questions(texts: list[str]) -> dict:
questions = {}
for i, text in enumerate(texts):
bid = block_id(i)
questions[f"type_{bid}"] = Choice(
instructions=f"What kind of content is block {bid}?", criteria=TYPE_CRITERIA
)
if len(text) <= HEADING_MAX_CHARS:
questions[f"hlevel_{bid}"] = Choice(
instructions=f"As a heading, what level would block {bid} occupy in this document's structure?",
criteria=HLEVEL_CRITERIA,
)
questions[f"step_{bid}"] = Noul(
instructions=f"Is block {bid} an instruction in a sequence where the order of the items matters?",
criteria=NoulCriteria(
true="It is one step of a procedure - the items around it must happen in order",
false="Order is irrelevant - it is a loose collection, or not a list item at all",
),
)
questions[f"callout_{bid}"] = Choice(
instructions=f"What kind of aside is block {bid}?", criteria=CALLOUT_CRITERIA
)
return questions
@json_cache
def classify(texts: list[str], gaps: list[bool]) -> dict:
tagged = tag([{"text": t, "gap": g} for t, g in zip(texts, gaps)], "B")
questions = classify_questions(texts)
started = perf_counter()
response = client.system_one(state=tagged, questions=questions, model=TYPESAFE_MODEL)
judgments = []
for i in range(len(texts)):
bid = block_id(i)
type_answer = response.answers[f"type_{bid}"]
hlevel = response.answers.get(f"hlevel_{bid}")
judgments.append(
{
"type": type_answer.choice,
"confidence": type_answer.confidence,
"probabilities": type_answer.probabilities,
"hlevel": hlevel.choice if hlevel else "section",
"step": response.answers[f"step_{bid}"].noul,
"callout": response.answers[f"callout_{bid}"].choice,
}
)
return {
"judgments": judgments,
"n_questions": len(questions),
"seconds": round(perf_counter() - started, 2),
"usage": [response.usage.input_tokens, response.usage.output_tokens],
}
classified = classify([b["text"] for b in blocks], [b["gap"] for b in blocks])
for block, judgment in zip(blocks, classified["judgments"]):
block.update(judgment)
print(f"{classified['n_questions']} questions about {len(blocks)} blocks, one request, "
f"{classified['seconds']}s\n")
print(f"{'block':<6}{'type':<11}{'conf':<6}{'companion used':<18}text")
for i, b in enumerate(blocks):
companion = {
"heading": f"level={b['hlevel']}",
"list_item": f"step={b['step']:.2f}",
"callout": f"kind={b['callout']}",
}.get(b["type"], "-")
print(f"{block_id(i):<6}{b['type']:<11}{b['confidence']:.2f} {companion:<18}"
f"{b['text'][:46]}")62 questions about 17 blocks, one request, 0.51s
block type conf companion used text
B000 heading 0.99 level=title Migration to the new build system
B001 paragraph 0.98 - Hi everyone, quick heads up about the build sy
B002 heading 0.75 level=section What changes for you
B003 paragraph 0.89 - The old make targets keep working until the en
B004 code 1.00 - bun run build
B005 paragraph 0.90 - Generated artifacts no longer need to be commi
B006 paragraph 0.43 - The cutover touches three teams, so check whet
B007 list_item 0.99 step=0.15 The platform team
B008 list_item 1.00 step=0.16 The web client team
B009 list_item 0.99 step=0.12 Whoever still owns the release tooling
B010 heading 0.96 level=section Things to do before Monday
B011 list_item 0.98 step=0.86 Update your local toolchain to version 2.4 or
B012 list_item 0.99 step=0.87 Delete the old build cache directory
B013 list_item 0.92 step=0.90 Run the doctor script and fix anything it flag
B014 callout 0.65 kind=warning If the doctor script reports a red result on t
B015 quote 0.99 - As Dana put it in the kickoff, "a migration no
B016 paragraph 0.92 - Thanks, and shout if anything looks off.每个区块的判断都在该表格中，而伴生列则展示了预先确定的答案是如何被使用的：“周一前要做的事情”这三行带有接近 0.9 的步骤概率（它们将渲染为有序列表），三行团队信息接近 0.1（无序列表），而关于医生脚本的未标记警告被分类为 warning 类型的提示框。附录部分探讨了模型不确定性的那个区块。
渲染
代码根据判断结果组装页面。连续的列表项合并为一个列表，当这些列表项的步骤概率平均值至少为 0.5 时，该列表为有序列表。该阈值是一个分组级别的决策，并非直接询问单个问题的结果。
STEP_THRESHOLD = 0.5
HEADING_MARK = {"title": "#", "section": "##", "subsection": "###"}
CALLOUT_MARK = {"note": "NOTE", "tip": "TIP", "warning": "WARNING"}
def to_markdown(blocks: list[dict]) -> str:
groups = []
for b in blocks:
if b["type"] in ("list_item", "code") and groups and groups[-1][0] == b["type"]:
groups[-1][1].append(b)
else:
groups.append((b["type"], [b]))
parts = []
for kind, items in groups:
if kind == "list_item":
ordered = sum(b["step"] for b in items) / len(items) >= STEP_THRESHOLD
parts.append("\n".join(
f"{n + 1}. {b['text']}" if ordered else f"- {b['text']}"
for n, b in enumerate(items)
))
elif kind == "code":
parts.append("```\n" + "\n".join(b["text"] for b in items) + "\n```")
elif kind == "heading":
parts.append(f"{HEADING_MARK[items[0]['hlevel']]} {items[0]['text']}")
elif kind == "quote":
parts.append(f"> {items[0]['text']}")
elif kind == "callout":
parts.append(f"> [!{CALLOUT_MARK[items[0]['callout']]}]\n> {items[0]['text']}")
else:
parts.append(items[0]["text"])
return "\n\n".join(parts) + "\n"
markdown = to_markdown(blocks)
print(markdown)# Migration to the new build system
Hi everyone, quick heads up about the build system migration that is happening next week. We have been running the new pipeline in shadow mode for three weeks and the results look solid, so it is time to make the switch for real.
## What changes for you
The old make targets keep working until the end of the month. The new entrypoint is a single command that wraps everything, including the docs build that used to be separate.
```
bun run build
```
Generated artifacts no longer need to be committed. The new pipeline uploads them to the registry automatically, and checking them in just creates merge conflicts.
The cutover touches three teams, so check whether you are on this list before you plan anything for Monday:
- The platform team
- The web client team
- Whoever still owns the release tooling
## Things to do before Monday
1. Update your local toolchain to version 2.4 or later
2. Delete the old build cache directory
3. Run the doctor script and fix anything it flags
> [!WARNING]
> If the doctor script reports a red result on the toolchain check, do not proceed with the migration. Ping the infra channel first and we will sort it out together.
> As Dana put it in the kickoff, "a migration nobody notices is the only kind worth shipping."
Thanks, and shout if anything looks off.输入中的每个词都来自原始内容。该流水线仅选择了边界、类型和标记。
在游乐场中打开
此分享链接包含拼接后的代码块和完整的第二遍问题集。打开它以实时重新运行分类。
playground_link = make_playground_link(
tag(blocks, "B"),
classify_questions([b["text"] for b in blocks]),
models=[TYPESAFE_MODEL],
)
display(Markdown(f"🔗 [Open the stitched memo + questions in the TypeSafe playground]({playground_link})"))附录
成本与延迟
tokens = [result["usage"], classified["usage"]]
total_in, total_out = sum(t[0] for t in tokens), sum(t[1] for t in tokens)
cost = total_in / 1e6 * PRICE[0] + total_out / 1e6 * PRICE[1]
n_joins = sum(1 for l in LINES if not l["gap"]) - 1
print(f"pass 1 {n_joins} questions {result['seconds']}s")
print(f"pass 2 {classified['n_questions']} questions {classified['seconds']}s")
print(f"total {total_in + total_out:,} tokens "
f"{result['seconds'] + classified['seconds']:.1f}s ${cost:.4f}")pass 1 16 questions 0.32s
pass 2 62 questions 0.51s
total 10,211 tokens 0.8s $0.0003两次往返，10,211 个 token，0.8 秒，$0.0015。
连接阈值的来源
来自第 1 轮的单行连接概率：
print("join line")
for i, line in enumerate(LINES[:18]):
join = " " if i == 0 or line["gap"] else f"{result['joins'][i]:.2f}"
print(f"{join} {line_id(i)}| {line['text'][:66]}")join line
L000| Migration to the new build system
L001| Hi everyone, quick heads up about the build system migration that
0.77 L002| happening next week. We have been running the new pipeline in shad
0.62 L003| mode for three weeks and the results look solid, so it is time to
0.39 L004| make the switch for real.
L005| What changes for you
L006| The old make targets keep working until the end of the month. The
0.42 L007| entrypoint is a single command that wraps everything, including th
0.59 L008| docs build that used to be separate.
L009| bun run build
L010| Generated artifacts no longer need to be committed. The new pipeli
0.48 L011| uploads them to the registry automatically, and checking them in
0.40 L012| just creates merge conflicts.
L013| The cutover touches three teams, so check whether you are on this
0.50 L014| list before you plan anything for Monday:
0.22 L015| The platform team
0.11 L016| The web client team
0.12 L017| Whoever still owns the release tooling概率分布在两个不同的区间内：将句子拆分的换行得分在 0.39 及以上，而作者意图的换行得分接近零。但是，在两个区间之间设置截止点取决于前一行以何种方式结束，这一事实代码可以直接读取：
- 在悬挂行（即没有句末标点的行）之后，任何 0.2 或以上的得分都视为续行。真正的续行在此处的得分低至 0.39（
L004| make the switch for real.），因此如果统一使用 0.5 作为谨慎的截止点，会破坏健康的段落结构。 - 在终止标点（即结束句子或子句的字符：
.!?:;）之后，截止点提高至 0.5。备忘录中的团队列表说明了原因：L015| The platform team紧跟在冒号之后，得分为 0.22。这是一个低但非零的“句子继续”信号，它会通过 0.2 的截止点，并将该列表合并到引入它的句子中。单一阈值无法同时适用于这两种情况；一旦代码先检查标点符号，这两个区间就会分离开来。
为什么问题是“句中”而不是“同一段落”
此管道的第一个版本提出了一个显而易见的问题：“这两行是否属于同一段落？”它在特定方式下失败了。标题下的一连串短行（无项目符号的列表）在松散意义上确实是一个段落：这些行聚集在一起并共享主题。当被问及段落时，模型会对每一对都回答“是”，拼接步骤会将整个列表合并为一个长块。
同一文档，相同的请求形状，仅措辞发生了变化：
def naive_join_question(i: int) -> Noul:
return Noul(
instructions=f"Are lines {line_id(i - 1)} and {line_id(i)} part of the same paragraph?",
criteria=NoulCriteria(
true="The two lines belong to the same paragraph of running text",
false="The two lines belong to different paragraphs or different pieces of content",
),
)
naive = stitch("same-paragraph")
print(f"{'':14}{'mid-sentence':>13}{'same paragraph':>16}")
for i in (15, 16, 17, 20, 21):
print(f"{line_id(i)}{'':2}{LINES[i]['text'][:36]:<38}"
f"{result['joins'][i]:>7.2f}{naive['joins'][i]:>13.2f}")
print(f"\nblocks after merge: {len(blocks)} (mid-sentence) vs "
f"{len(merge(naive['joins']))} (same paragraph)") mid-sentence same paragraph
L015 The platform team 0.22 0.77
L016 The web client team 0.11 0.81
L017 Whoever still owns the release tooli 0.12 0.78
L020 Delete the old build cache directory 0.08 0.88
L021 Run the doctor script and fix anythi 0.05 0.91
blocks after merge: 17 (mid-sentence) vs 12 (same paragraph)The paragraph wording causes every unmarked list item to score above 0.75, and both lists collapse. The memo merges into a few run-on blocks. “Same paragraph” asks the model to judge whether the topic carries over, and between list items it does. “Picks up mid-sentence” asks about the text itself. When a judgment call feeds a threshold, the question should name the narrowest fact that decides it. Here the wording is the difference between 17 blocks and 12.
置信度最低的块
uncertain = min(blocks, key=lambda b: b["confidence"])
print(f'"{uncertain["text"]}"')
print(f"confidence {uncertain['confidence']:.2f}: ", end="")
print(", ".join(f"{k} {v:.2f}" for k, v in
sorted(uncertain["probabilities"].items(), key=lambda kv: -kv[1])[:3]))"The cutover touches three teams, so check whether you are on this list before you plan anything for Monday:"
confidence 0.43: paragraph 0.53, list_item 0.24, callout 0.19介绍团队列表的句子存在真正的歧义——它既像标题一样命名了后续内容，又是一个完整的句子（段落式），并且出现在本应是提示块的位置。因此，各类别概率分布较为分散（段落 0.53，列表项 0.24，提示块 0.19）。UI 可以据此呈现相关信息——例如，对任何类型置信度（即获胜选项背后的概率）低于 0.55 的区块进行下划线标记，以提示用户审查。