Merge pull request 'feat(review): 审查阶段对齐绘图区分辨率 + 放大取证闭环' (#15) from home-lenovo-legion-r9000/home-ai-dev into main

Reviewed-on: #15
This commit was merged in pull request #15.
This commit is contained in:
gjm committed 2026-10-16 18:57:04 +08:00
commit 3bc64b6c3e
11 files changed
+320 -40

No files matched your search

+1
View File
@@ -51,6 +51,7 @@ python verify_mcp_fault_injection.py # 工具异常保护(类型错误不再
python verify_stdio_bridge.py # stdio ↔ TCP 桥(协议协商 / 转发 / 通知)
python verify_annotate.py # 批注回写链路
python verify_drawing_area.py # 绘图区最大化:基线 → max → restore 全回位(实测 3.68x)
python verify_review_maximized.py # 审查全链路:取证+审查两阶段均在最大化态、结束精确还原
```
这些脚本通过 TCP 连接 `127.0.0.1:8080` 收发 JSON-RPC。
+35 -4
View File
@@ -1,6 +1,6 @@
---
name: Agent 路线与多源架构(2026-10-15)
description: dwg_review agent 的总目标、多源图纸分层架构(视觉为最大公约数)、工具层与 agent 层的正交分工、多源证据层、截图去 UI、grounding 实测、热力图 + 自适应瓦片规划 + VLM 分辨率标定、绘图区最大化(set_drawing_area)与空瓦片过滤、后续路线
description: dwg_review agent 的总目标、多源图纸分层架构(视觉为最大公约数)、工具层与 agent 层的正交分工、多源证据层、截图去 UI、grounding 实测、热力图 + 自适应瓦片规划 + VLM 分辨率标定、绘图区最大化(set_drawing_area)与空瓦片过滤、审查阶段分辨率对齐 + 放大取证闭环、后续路线
type: project
---
@@ -73,9 +73,10 @@ Agent 推理层(审查逻辑 / 清单 / 规则)
## 后续路线(未做)
1. **VLM 主动放大(自适应取证)**:复用受控只读取证子循环,让 VLM 请求放大看不清处。
- 两条路径:① 按实体 `zoom_to_entity(handle)`(可靠);② 按区域(把 VLM 归一化坐标映射成
WCS)——**grounding 已实测可用**(见上节)。
- 必须设**迭代上限 + 兜底**;**不把"看清没"完全交给 VLM**(会过度自信/幻觉)。
- **2026-10-16 部分落地**:审查阶段已对齐分辨率 + 提示词已明确放大协议 + 视口复位,
实测 LLM 真的会连发 `zoom_to_window`+`get_viewport_screenshot`(见下节)。
仍缺:① 按区域放大时的 **grounding→WCS 映射**(目前靠 LLM 自己用 bbox 算窗口,够用);
② 把「看清没」的判定**不交给 VLM**(防过度自信)——现靠回合上限 + 截图预算兜底。
- 现有 `zoom_to_window`/`zoom_to_entity`/`get_viewport_screenshot` 已在只读白名单,
**无需新增工具**。
2. **PDF / 图片源**:在 `phases/evidence.collect()` 加分支,产出**同形** evidence
@@ -171,3 +172,33 @@ Agent 推理层(审查逻辑 / 清单 / 规则)
**新增配置**:`DRAWING_AREA_MAXIMIZE`(True) / `DRAWING_AREA_ENTER_WAIT`(1.5) /
`DRAWING_AREA_RESTORE_WAIT`(1.5,**必须 > 插件内 700ms 延迟还原**)。
## 审查阶段分辨率对齐 + 放大取证闭环(2026-10-16,本轮)
**发现的缺口**:`maximized_drawing_area` 原先**只包住取证循环**,审查阶段没被包住 →
LLM 在受控子循环里 zoom + 截图拿到的是**未最大化的 2110×462**(正是上一轮刚花力气解决的
低分辨率问题,在**最需要放大看细节的审查阶段**又出现了);且审查中的 zoom **不复位视口**,
跑完把用户的 CAD 留在最后一次放大处。
**三项改动**:
1. **`__main__.run_review`**:把 `with evidence_phase.maximized_drawing_area(ctx)` 从
「只包取证」扩展到「**包取证 + 审查**」(两者都要截图);`report`/`annotate` 留在 with 之外。
2. **`phases/review.py`**:新增 `_VIEW_TOOLS = {zoom_to_window, zoom_to_entity}`;
`review_frame` 拆成外壳 + `_review_frame`,外壳 `try/finally` 里**仅当确实动过视口**才
`_reset_viewport`(复位到该图框整幅视图 `zoom_to_window(win, margin=0.02)`)。
—— 只在动过时复位,既省一次调用,也不影响"纯查结构化数据"的路径。
3. **`checklist.build_system_prompt`**:把「可调用只读工具」改写成**明确的放大取证协议**:
截图看不清时用 `zoom_to_window`(坐标由图框/实体 bbox 算)或 `zoom_to_entity`,
再 `get_viewport_screenshot` 复核;并约束**不要为看图而反复放大**。
**实测(真实图纸,1 图框,`verify_review_maximized.py`)**:
- 取证阶段采样 `[-4,-2,3444,1442]`、**审查阶段采样同样是 `[-4,-2,3444,1442]`**(最大化态),
结束**精确还原** `[68,0,2316,960]` → 三项全 PASS。
- 子循环里 LLM **真的连发 8 组 `zoom_to_window`+`get_viewport_screenshot`**
(提示词生效,"看不清→放大重看"闭环跑通)。
**同一轮发现的次生问题**:`_MAX_OBS_IMAGES` 原为硬编码 **3**,而模型一轮内会发 8 次截图
→ **拍了 8 张只回灌 3 张**(其余被"已省略"占位)。改为配置项 `REVIEW_OBS_IMAGES`
(默认 **6**,`config.py`),并把预算耗尽时的占位文案改成**明确告知模型"不要再请求截图,
请给出结论"**(原先只说"已省略")。实测模型截图次数不变(8),但**能看到 6 张而非 3 张**。
+1
View File
@@ -47,6 +47,7 @@ type: reference
| `verify_vlm_resolution_offline.py` | **离线**(纯 PIL 合成)标定 VLM 读图下限 `R_text`/`R_line`,`--dry-run` 只出图 |
| `verify_text_resolution.py` | **CAD 实测**标定(临时加字 + 平行线,测完 `delete_entities` 清理);`--keep` 保留 |
| `verify_drawing_area.py` | 绘图区最大化验收:基线 → `set_drawing_area(max)` → `restore`,断言窗口矩形/最大化态/绘图区尺寸全回位(实测 3.68×) |
| `verify_review_maximized.py` | 审查全链路验收:在 `collect`/`review_frame` 入口各采一次窗口矩形,断言**取证与审查两阶段都在最大化态**、结束精确还原;并统计子循环里的 zoom/截图调用次数 |
> 历史:根目录 `test.py` / `test_autoagent.py` / `countOfCircle.py` / `agent_loop.py` /
> `dwg_review_agent.py` 五个遗留调试脚本已于 2026-10-14 删除(前三个含硬编码 API Key,
+31 -30
View File
@@ -62,11 +62,13 @@ def run_review(ctx, args):
system_prompt = build_system_prompt(ctx.checklist, allow_tools=True)
print("[2/4] 取证:逐图框采集(串行) ...")
evidences = []
# 截屏前把绘图区最大化(窗口铺满 + 干净屏幕),采集完在 finally 里还原;
# 旧版插件没有该工具时静默降级。
frame_findings = []
# 截屏前把绘图区最大化(窗口铺满 + 干净屏幕)。**取证与审查都要截图**——审查阶段的
# 受控子循环会按需放大复核,故一并包住,结束在 finally 里还原;旧版插件无此工具时静默降级。
with evidence_phase.maximized_drawing_area(ctx):
print("[2/4] 取证:逐图框采集(串行) ...")
for idx, frame in enumerate(frames, 1):
print(
f" - 图框 {idx}/{len(frames)} kind={frame.get('kind')} "
@@ -82,35 +84,34 @@ def run_review(ctx, args):
f"截图 {len(evidence_images(ev))} 张"
)
if args.plan_only:
path = _dump_evidence(evidences)
print(f"[evidence] 已写入 {path}")
return 0
if args.plan_only:
path = _dump_evidence(evidences)
print(f"[evidence] 已写入 {path}")
return 0
print("[3/4] 审查:确定性检查前置 + 逐图框 LLM 审查(含受控只读取证子循环) ...")
frame_findings = []
for ev in evidences:
print(f" - 审查图框 {ev['index']} ...")
checks = (
checks_phase.run_deterministic_checks(ctx, ev)
if config.DETERMINISTIC_CHECKS
else []
)
conclusion = review.review_frame(ctx, ev, system_prompt)
finding = findings_mod.build_finding(
ev, conclusion, checks, validate_handles=config.VALIDATE_HANDLES
)
frame_findings.append(finding)
print("[3/4] 审查:确定性检查前置 + 逐图框 LLM 审查(含受控只读取证子循环) ...")
for ev in evidences:
print(f" - 审查图框 {ev['index']} ...")
checks = (
checks_phase.run_deterministic_checks(ctx, ev)
if config.DETERMINISTIC_CHECKS
else []
)
conclusion = review.review_frame(ctx, ev, system_prompt)
finding = findings_mod.build_finding(
ev, conclusion, checks, validate_handles=config.VALIDATE_HANDLES
)
frame_findings.append(finding)
counts = finding["counts"]
preview = finding["summary"]
if len(preview) > 200:
preview = preview[:200] + "..."
print(
f" 结论:{preview}"
f"(确定性 {counts['deterministic']} / LLM {counts['llm']}"
f" / 未核实 {counts['unverified']})"
)
counts = finding["counts"]
preview = finding["summary"]
if len(preview) > 200:
preview = preview[:200] + "..."
print(
f" 结论:{preview}"
f"(确定性 {counts['deterministic']} / LLM {counts['llm']}"
f" / 未核实 {counts['unverified']})"
)
print("[4/4] 汇总报告 ...")
markdown = report.summarize(ctx, frame_findings)
+7 -2
View File
@@ -53,8 +53,13 @@ def build_system_prompt(checklist, *, allow_tools=True):
if allow_tools:
lines += [
"你可以调用**只读**取证工具进一步确认细节(例如放大局部、查实体详情、查重复实体)。",
"当结构化数据不足以判断时,先调用工具取证,再下结论。",
"你可以调用**只读**取证工具进一步确认细节;**不要凭猜测下结论**:",
" - **截图里的文字/标注/细节看不清时**:用 `zoom_to_window` 放大该区域"
"(窗口坐标用图框 bbox 或实体 bbox 计算,可适当留边距),"
"再 `get_viewport_screenshot` 复核;已知 handle 时也可直接 `zoom_to_entity` 放大该实体。",
" - 结构化数据不足以判断时:用 `get_entity_detail` / `find_duplicate_entities` /"
" `find_zero_length` / `get_layers` 等查证后再下结论。",
" - 放大截图是**复核手段**:确认清楚后即写入结论,不要为看图而反复放大。",
"",
]
+4
View File
@@ -90,6 +90,10 @@ FRAME_ENTITY_LIMIT = int(os.environ.get("FRAME_ENTITY_LIMIT", "200"))
RENDER_WAIT = float(os.environ.get("RENDER_WAIT", "0.6"))
# 审查阶段受控工具子循环的最大回合数
REVIEW_MAX_ITERS = int(os.environ.get("REVIEW_MAX_ITERS", "4"))
# 审查子循环里回灌给 LLM 的截图数上限(控制视觉 token 成本)。
# 实测:提示词鼓励「看不清→放大重看」后,模型一轮内会发多次 zoom+截图,
# 上限过小会让它白拍(拍了却看不到)。6 可覆盖多数复核需求。
REVIEW_OBS_IMAGES = int(os.environ.get("REVIEW_OBS_IMAGES", "6"))
# 报告输出目录
REPORT_DIR = _resolve_path(os.environ.get("REPORT_DIR", "reports"))
# 批注回写使用的图层
+41 -4
View File
@@ -7,18 +7,24 @@
结论是**结构化对象** `{"summary": str, "issues": [dict], "raw": str}`,
不再依赖第二次 LLM 调用来抽取结构(见 `findings.parse_conclusion`)。
审查在**绘图区已最大化**的前提下进行(见 `evidence.maximized_drawing_area`),
子循环里的放大截图因此是高清的;若 LLM 动过视口,结束时复位到该图框整幅视图。
"""
import json
from .. import evidence_model
from .. import config, evidence_model
from ..cad_tools import READ_ONLY_TOOLS, is_write_tool
from ..findings import parse_conclusion
from ..llm_client import LlmClient, downsample_image_b64
from ..mcp_client import McpError, McpToolError, result_images, result_texts
# 单次审查回灌给 LLM 的图片数上限(控制视觉 token 成本)
_MAX_OBS_IMAGES = 3
_MAX_OBS_IMAGES = config.REVIEW_OBS_IMAGES
# 会改动视口的只读取证工具:用过之后须把视口复位,别把用户的 CAD 留在放大处
_VIEW_TOOLS = frozenset({"zoom_to_window", "zoom_to_entity"})
def evidence_payload(evidence):
@@ -54,7 +60,12 @@ def _run_readonly_tool(ctx, name, args, image_budget):
content.append({"type": "text", "text": text})
for b64, mime in result_images(resp):
if image_budget[0] <= 0:
content.append({"type": "text", "text": "(后续截图已省略以控制成本)"})
content.append(
{
"type": "text",
"text": "(截图预算已用完,后续截图已省略;请基于已有信息给出结论,不要再请求截图。)",
}
)
break
image_budget[0] -= 1
nb, nm = downsample_image_b64(b64, mime)
@@ -64,8 +75,32 @@ def _run_readonly_tool(ctx, name, args, image_budget):
return content, image_budget
def _reset_viewport(ctx, evidence):
"""把视口复位到该图框的整幅视图(审查中的放大不应残留给用户)。"""
win = evidence.get("window")
if not win:
return
try:
ctx.tools.call("zoom_to_window", dict(win, margin=0.02))
except (McpError, McpToolError):
pass
def review_frame(ctx, evidence, system_prompt, log=print):
"""审查单个图框,返回结构化结论 `{"summary", "issues", "raw"}`。"""
"""审查单个图框,返回结构化结论 `{"summary", "issues", "raw"}`。
审查中的放大取证会改动视口,故在 finally 里复位——**仅当确实动过视口**
(LLM 只查结构化数据时不动用户视图)。
"""
view_changed = [False]
try:
return _review_frame(ctx, evidence, system_prompt, log, view_changed)
finally:
if view_changed[0]:
_reset_viewport(ctx, evidence)
def _review_frame(ctx, evidence, system_prompt, log, view_changed):
payload = evidence_payload(evidence)
user_content = [
{
@@ -105,6 +140,8 @@ def review_frame(ctx, evidence, system_prompt, log=print):
except json.JSONDecodeError:
args = {}
log(f" [取证] {name} {args if args else ''}")
if name in _VIEW_TOOLS:
view_changed[0] = True
content, image_budget = _run_readonly_tool(ctx, name, args, image_budget)
messages.append(
{"role": "tool", "tool_call_id": tool_call.id, "content": content}
+8
View File
@@ -65,6 +65,14 @@ def test_build_system_prompt_tool_hint_toggle():
assert "只读" not in without
def test_build_system_prompt_instructs_zoom_when_unreadable():
prompt = build_system_prompt(DEFAULT_CHECKLIST, allow_tools=True)
assert "看不清" in prompt
assert "zoom_to_window" in prompt
assert "zoom_to_entity" in prompt
assert "get_viewport_screenshot" in prompt
def test_build_system_prompt_requires_structured_json_output():
# allow_tools=False 以隔离「工具提示块」里的「只读」字样
prompt = build_system_prompt(DEFAULT_CHECKLIST, allow_tools=False)
+5
View File
@@ -101,6 +101,11 @@ def test_tiling_defaults_are_sane():
assert config.HEATMAP_CELL_DIV > 0
def test_review_obs_images_default_is_sane():
# 审查子循环回灌截图上限:须为正,且不低于旧硬编码值 3
assert config.REVIEW_OBS_IMAGES >= 3
def test_drawing_area_constants_have_expected_types():
assert isinstance(config.DRAWING_AREA_MAXIMIZE, bool)
assert isinstance(config.DRAWING_AREA_ENTER_WAIT, float)
+79
View File
@@ -260,3 +260,82 @@ def test_review_frame_includes_multiple_image_blocks():
assert len(image_blocks) == 2
assert any(b.get("text") == "(截图:overall)" for b in user_content)
assert any(b.get("text") == "(截图:tile-1)" for b in user_content)
# ---------------------------------------------------------------------------
# 视口复位(放大取证不残留给用户)
# ---------------------------------------------------------------------------
def test_review_frame_resets_viewport_after_zoom():
llm = FakeLlmClient(
script=[
llm_calls([("zoom_to_window", {"min_x": 1, "min_y": 2, "max_x": 3, "max_y": 4})]),
llm_text("done"),
]
)
client = FakeMcpClient()
ctx = make_ctx(client=client, llm=llm)
review_frame(ctx, make_evidence(), "sys", log=lambda *a: None)
zooms = client.called("zoom_to_window")
assert len(zooms) == 2 # LLM 放大一次 + 结束时复位一次
reset = zooms[-1]["arguments"]
assert (reset["min_x"], reset["min_y"]) == (0.0, 0.0) # 复位到图框整幅视图
assert (reset["max_x"], reset["max_y"]) == (100.0, 50.0)
assert reset["margin"] == 0.02
def test_review_frame_resets_viewport_after_zoom_to_entity():
llm = FakeLlmClient(
script=[llm_calls([("zoom_to_entity", {"handle": "A1"})]), llm_text("done")]
)
client = FakeMcpClient()
ctx = make_ctx(client=client, llm=llm)
review_frame(ctx, make_evidence(), "sys", log=lambda *a: None)
assert client.called_names().count("zoom_to_window") == 1
def test_review_frame_resets_viewport_at_iteration_limit():
llm = FakeLlmClient(
script=[llm_calls([("zoom_to_window", {})]), llm_text("收口结论")]
)
client = FakeMcpClient()
ctx = make_ctx(client=client, llm=llm, max_iters=1)
out = review_frame(ctx, make_evidence(), "sys", log=lambda *a: None)
assert out["summary"] == "收口结论"
assert client.called_names().count("zoom_to_window") == 2 # 强制收口路径也复位
def test_review_frame_does_not_touch_viewport_without_zoom():
llm = FakeLlmClient(script=[llm_calls([("get_layers", {})]), llm_text("x")])
client = FakeMcpClient()
ctx = make_ctx(client=client, llm=llm)
review_frame(ctx, make_evidence(), "sys", log=lambda *a: None)
assert "zoom_to_window" not in client.called_names()
def test_review_frame_viewport_reset_failure_is_swallowed():
seen = {"n": 0}
def zoom(_args):
seen["n"] += 1
if seen["n"] > 1:
raise McpToolError("复位失败")
return ok_text({})
llm = FakeLlmClient(
script=[llm_calls([("zoom_to_window", {})]), llm_text("结论")] # noqa: E501
)
client = FakeMcpClient(responses={"zoom_to_window": zoom})
ctx = make_ctx(client=client, llm=llm)
out = review_frame(ctx, make_evidence(), "sys", log=lambda *a: None)
assert out["summary"] == "结论" # 复位失败不影响结论
assert seen["n"] == 2
def test_review_frame_no_viewport_reset_without_window():
llm = FakeLlmClient(script=[llm_calls([("zoom_to_window", {})]), llm_text("x")])
client = FakeMcpClient()
ctx = make_ctx(client=client, llm=llm)
ev = make_evidence()
ev["window"] = None # 非 CAD 源(无窗口)→ 无从复位
review_frame(ctx, ev, "sys", log=lambda *a: None)
assert client.called_names().count("zoom_to_window") == 1
+108
View File
@@ -0,0 +1,108 @@
"""验收:审查阶段是否也在「绘图区最大化」状态下进行,且结束后精确还原。
做法:在取证 `collect` 与审查 `review_frame` 入口各采一次窗口矩形,
并在 `McpClient.call_tool` 上挂计数器,统计 LLM 子循环是否真的用了放大工具。
用法:先启动已加载插件的 CAD 并打开图纸,然后 `python verify_review_maximized.py`。
"""
import ctypes
import sys
from ctypes import wintypes
from dwg_review import __main__ as cli
from dwg_review.mcp_client import McpClient
from dwg_review.phases import evidence as ev
from dwg_review.phases import review as rv
_u = ctypes.windll.user32
_u.GetWindowTextW.argtypes = [wintypes.HWND, wintypes.LPWSTR, ctypes.c_int]
_u.IsWindowVisible.argtypes = [wintypes.HWND]
_u.GetWindowRect.argtypes = [wintypes.HWND, ctypes.POINTER(wintypes.RECT)]
_WNDENUMPROC = ctypes.WINFUNCTYPE(wintypes.BOOL, wintypes.HWND, wintypes.LPARAM)
def _find_main_window():
hit = [None]
def cb(hwnd, _lparam):
if not _u.IsWindowVisible(hwnd):
return True
n = _u.GetWindowTextLengthW(hwnd)
if n == 0:
return True
buf = ctypes.create_unicode_buffer(n + 1)
_u.GetWindowTextW(hwnd, buf, n + 1)
if "AutoCAD" in buf.value and "[" in buf.value:
hit[0] = hwnd
return False
return True
_u.EnumWindows(_WNDENUMPROC(cb), 0)
return hit[0]
HWND = _find_main_window()
if not HWND:
print("[X] 未找到 AutoCAD 主窗口")
sys.exit(1)
def rect():
r = wintypes.RECT()
_u.GetWindowRect(HWND, ctypes.byref(r))
return [r.left, r.top, r.right, r.bottom]
def area(r):
return (r[2] - r[0]) * (r[3] - r[1])
# --- 采样点 ---------------------------------------------------------------
phase_rects = {"evidence": [], "review": []}
zoom_calls = []
_orig_collect = ev.collect
_orig_review = rv.review_frame
_orig_call = McpClient.call_tool
def collect(ctx, index, frame):
phase_rects["evidence"].append(rect())
return _orig_collect(ctx, index, frame)
def review(ctx, evidence, system_prompt, log=print):
phase_rects["review"].append(rect())
return _orig_review(ctx, evidence, system_prompt, log=log)
def call_tool(self, name, arguments=None, **kw):
if name in ("zoom_to_window", "zoom_to_entity", "get_viewport_screenshot"):
zoom_calls.append(name)
return _orig_call(self, name, arguments, **kw)
ev.collect = collect
rv.review_frame = review
McpClient.call_tool = call_tool
before = rect()
print(f"起始窗口矩形 = {before} 面积={area(before)}")
rc = cli.main(["--frames", "1"])
after = rect()
print()
print("=== 结果 ===")
print(f"取证阶段采样 = {phase_rects['evidence']}")
print(f"审查阶段采样 = {phase_rects['review']}")
print(f"结束窗口矩形 = {after} 面积={area(after)}")
print(f"子循环取证调用 = {zoom_calls}")
ok_ev = all(area(r) > area(before) for r in phase_rects["evidence"]) if phase_rects["evidence"] else False
ok_rv = all(area(r) > area(before) for r in phase_rects["review"]) if phase_rects["review"] else False
print()
print(f"[{'PASS' if ok_ev else 'FAIL'}] 取证阶段处于最大化(面积大于基线)")
print(f"[{'PASS' if ok_rv else 'FAIL'}] 审查阶段处于最大化(面积大于基线)")
print(f"[{'PASS' if after == before else 'FAIL'}] 结束精确还原窗口矩形")
sys.exit(0 if (ok_ev and ok_rv and after == before and rc == 0) else 1)