Translation(model translation, not the source's own text)
DroidPath Agentic Benchmarks
本基准测试套件将 DroidPath(Shadow DOM + Layer 3 A Spatial Graph Router)* 与自主 AI 智能体使用的两种主流 Android 自动化范式进行对比:
- Pure Vision(截图 + VLM 循环)—— 每一步都捕获全分辨率 PNG 截图,并发送给视觉语言模型以预测 (x, y) 坐标。
- Appium / uiautomator dump XML 循环 —— 每一步都通过 ADB 运行 uiautomator dump,并将原始 XML 层级结构(约 25–35 KB)输入 LLM。
- DroidPath(Shadow DOM + Layer 3 A) —— 通过持久化的 loopback TCP socket(android-daemon)流式传输增量 NDJSON 无障碍差异,并在本地 Rust 中通过 A 执行学习到的多跳路径,无需中间 LLM 轮次。
Measured Results(Live Unlocked Pixel 10a Hardware + Replay Suite)
透明性说明:以下硬件原语延迟和端到端(E2E)Settings 导航计时是在一台已解锁的 Google Pixel 10a(61061JEA307110,Android com.android.settings 根屏幕,54 个实时 UI 节点)上通过 USB ADB 实时测得的,使用 node scripts/benchmark_agentic.mjs --mode live --json 对编译后的 target/release/droidpath-mcp 运行。回放表中的多步 Pure Vision 与 Appium/XML 对比运行真实的 droidpath-mcp stdio JSON-RPC 调用,针对 60 节点 loopback Shadow DOM 守护进程,并配合 Methodology & Token Model 下记录的显式 token 与成本公式。
1. Live Hardware Probe(已解锁 Google Pixel 10a · com.android.settings,54 个节点)
| Primitive / Method | Status | Measured Latency | Payload Size | Notes |
|---|
| DroidPath android_get_ui_tree(稳态) | ok | 0.32 ms | 10,742 bytes(约 2,686 tok) | 来自本地 Shadow DOM 流的 54 个实时 UI 节点(比 uiautomator 快 7,575×) |
| DroidPath android_get_ui_tree(冷连接) | ok | 41.89 ms | 10,742 bytes(约 2,686 tok) | 首次 TCP 连接 + 实时快照(比 uiautomator 快 58×) |
| DroidPath Hybrid Vision( include_vision: true ) | ok | 452.33 ms | 38,581 bytes JPEG(540×1212) | 54 个实时 UI 节点 + 压缩 JPEG(q45,比原始 PNG 小 6.1×) |
| adb exec-out screencap -p(原始 PNG) | ok | 658.1 ms | 235,258 bytes PNG | 仅视觉捕获——无结构化 UI 树 |
| adb exec-out uiautomator dump /dev/tty(XML) | ok | 2,423.9 ms | 29,357 bytes(约 7,340 tok) | 需要停止 UiAutomation 守护进程;文本 token 数比 Shadow DOM 多 2.7× |
2. Live End-to-End Hardware Navigation(Google Pixel 10a · 真实种子水合 → 预热 A*)
| Task(Settings → Target) | Cold-Start(Seed Hydration) | Cold Status | Warm A* Graph | Warm Status | Live Success Rate |
|---|
| Settings → Display | 378.6 ms | arrived | 296.2 ms(1 step) | arrived | 100% |
| Settings → Battery | 655.9 ms | arrived | 642.8 ms(1 step) | arrived | 100% |
| Settings → About phone(需要滚动) | 775.7 ms | arrived | 744.1 ms(1 step) | arrived | 100% |
| 实时套件平均值 | 603 ms | 100%(3/3) | 561 ms | 100%(3/3) | 100% |
3. Local MCP Runtime Micro-Benchmarks(实测)
| Metric | Measured Value |
|---|
| tools/list JSON-RPC 延迟(9 个工具) | 0.25 ms |
| android_graph_stats JSON-RPC 延迟 | 0.42 ms |
| 启动时加载的预水合 Seed Graph | 20 vertices / 19 edges |
| 紧凑 Shadow DOM 树(60 节点模拟 Settings 屏幕) | 4,537 bytes(约 1,135 tokens) |
4. Architecture & Cost Comparison(5 个 WebArena 风格多步任务 · Replay + API Cost Model)
| Architecture | Avg LLM Calls / Task | Avg Tokens / Task | Avg Est. Task Time | Est. Cost / 1,000 Tasks | Advantage of DroidPath |
|---|
| DroidPath (Shadow DOM + A Router) * | 1.0 | 39 (warm A )* / ~1,135 (cold tree) | 464 ms (561 ms live warm) | $0.12 | 100% live E2E success · 192× cheaper than Vision |
| Pure Vision (Screenshots + VLM) | 4.2 | 7,673 | 9,954 ms | $23.02 | Baseline |
| Appium / uiautomator XML | 4.2 | 29,400 | 8,946 ms | $88.20 | 735× more expensive than Warm DroidPath |
Per-Task Breakdown
| Task | Hops | DroidPath (Calls / Tokens / Time) | Pure Vision (Calls / Tokens / Time) | Appium XML (Calls / Tokens / Time) |
|---|
| Enable Night Light (Display) | 3 | 1 call · 39 tok · 435 ms | 4 calls · 7,308 tok · 9,480 ms | 4 calls · 28,000 tok · 8,520 ms |
| Open Wi-Fi Settings (Network) | 4 | 1 call · 39 tok · 580 ms | 5 calls · 9,135 tok · 11,850 ms | 5 calls · 35,000 tok · 10,650 ms |
| Navigate to Battery Saver | 3 | 1 call · 39 tok · 435 ms | 4 calls · 7,308 tok · 9,480 ms | 4 calls · 28,000 tok · 8,520 ms |
| Open App Notifications | 3 | 1 call · 39 tok · 435 ms | 4 calls · 7,308 tok · 9,480 ms | 4 calls · 28,000 tok · 8,520 ms |
| Scroll to About Phone | 3 | 1 call · 39 tok · 435 ms | 4 calls · 7,308 tok · 9,480 ms | 4 calls · 28,000 tok · 8,520 ms |
Methodology & Token Model
- Text tokens: ceil(utf8_bytes / 4)(JSON/XML 的标准 BPE 近似值)。
- Cost / 1,000 tasks: 按 $3.00 / 1M 输入 tokens 透明计算(标准 Claude Sonnet 级 API 定价):(avg_tokens_per_task 1,000 / 1,000,000) $3.00。
- Vision image tokens: Anthropic 公布的 (W × H) / 750 公式,针对缩放到长边 1568px 的 1080×2400 Android 截图(706 × 1568 / 750 = 1,477 个图像 tokens + ~350 个提示/工具 tokens = 1,827 tokens/步)。
- Opt-in Hybrid Vision (include_vision: true): 降采样至 540px 宽的 JPEG(质量:45,在解锁的 Pixel 10a Settings 上实测 38.6 KB,而原始 PNG 为 235.3 KB)严格按需启用,仅在 agent 明确需要视觉消歧时使用(例如 canvas/icons),在 FLAG_SECURE 屏幕上仍严格禁止。
Reproduce on Your Device
# Build release binary first (optional — falls back to target/debug)
cargo build --release -p droidpath-mcp
# Replay benchmark (zero device required)
node scripts/benchmark_agentic.mjs
# Live physical device / emulator probe (measures real screencap -p & uiautomator dump)
node scripts/benchmark_agentic.mjs --mode live
The translation is machine-generated and individual terms may be off. Nothing is rewritten or cut; the original is authoritative.
Published by HN AI
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗