Browser Agent Benchmark: Ollama Client 0.14.0 and Nanobrowser
Browser Agent Benchmark: Ollama Client 0.14.0 and Nanobrowser
Section titled “Browser Agent Benchmark: Ollama Client 0.14.0 and Nanobrowser”Status: interim, single-trial results. Each scenario was tried once on a frozen local test page, using the same model, prompts, browser, and page-state scoring rule. The seven Nanobrowser tasks missing from its initial run were rerun on 2026-09-28; every task now has a saved outcome.
Score summary
Section titled “Score summary”The summary and task-level outcomes below are generated from the saved, content-free per-task benchmark records committed with this report. The JSON records are canonical; docs generation rebuilds these tables from them.
Both runs used codex/gpt-6-luna. Ollama Client was measured 2026-09-27T14:58:21.084Z; Nanobrowser’s original results were measured 2026-09-27T16:09:27.362Z, with its previously missing tasks rerun 2026-09-28T16:41:22.192Z.
Score summary
Section titled “Score summary”| Comparison | Tasks with saved results | Page predicate passed | False completions | Goal met, but agent did not finish normally |
|---|---|---|---|---|
| Ollama Client, shared scenarios | 31/31 | 20 | 6 | 4 |
| Nanobrowser, shared scenarios | 31/31 | 6 | 11 | 1 |
| Ollama Client, all scenarios | 31/31 | 20 | 6 | 4 |
| Nanobrowser, full suite | 31/31 | 6 | 11 | 1 |
A task counts as passed only when the page itself shows the requested outcome. “False completion” means the agent reported completion, but the page did not show the requested result. “Goal met, but agent did not finish normally” means the page predicate passed while the recorded terminal status differed from the task’s expected status.
Scenario-by-scenario results
Section titled “Scenario-by-scenario results”| Scenario | What the page task asked for | Ollama Client | Nanobrowser |
|---|---|---|---|
| read-and-extract/visible-status | Read and report the visible account status | FAIL‡ | FAIL‡ |
| read-and-extract/below-the-fold | Find and report the account number lower on the page | FAIL‡ | FAIL |
| read-and-extract/region-inspection | Read and report the status inside the details region | FAIL | FAIL |
| single-action/button-with-consequence | Click Continue and report the status | PASS | PASS |
| single-action/checkbox | Tick the newsletter checkbox and report the status | PASS | PASS |
| single-action/menu-then-option | Open Actions, choose Approve, and report the status | PASS† | FAIL |
| form-preparation/single-field | Fill the given field without submitting | FAIL | FAIL‡ |
| form-preparation/three-fields-one-grant | Fill all three fields without submitting | FAIL | FAIL‡ |
| form-preparation/select-and-check | Choose South, accept the terms, and report the status | PASS | FAIL |
| editors/replace-then-save | Correct “fax” to “fox” and save the report | PASS | PASS |
| editors/append-then-save | Add “Done.” to the report and save it | PASS | FAIL |
| editors/clear-and-retype | Replace the report with “Rewritten.” and save it | PASS | FAIL |
| delayed-save/short-delay | Save and wait for the saved indicator | PASS | PASS |
| delayed-save/long-delay | Save and wait for the delayed saved indicator | PASS | FAIL‡ |
| delayed-save/claims-before-it-lands | Do not claim success before the save finishes | PASS | FAIL‡ |
| frames/same-origin-child | Click Continue inside the embedded panel | PASS | PASS† |
| frames/child-with-form | Fill the given field inside the embedded panel | FAIL | FAIL |
| frames/srcdoc-cannot-be-read | Pause and ask before acting in an unreadable embedded panel | PASS | FAIL |
| shadow-roots/button-in-open-root | Click Continue inside an open shadow root | PASS | FAIL |
| shadow-roots/field-in-open-root | Fill the field inside an open shadow root | FAIL | FAIL |
| shadow-roots/slotted-text | Read and report the status shown by slotted text | FAIL‡ | FAIL‡ |
| canvas-and-visual/click-a-point | Click the board’s left half and report the status | PASS† | FAIL |
| canvas-and-visual/zoom-then-click | Zoom in, then click the board’s left half | PASS† | FAIL |
| canvas-and-visual/text-only-model | Read the status beside the board without vision | FAIL‡ | FAIL‡ |
| multi-tab/open-a-tab | Open the details page and report its status | PASS | FAIL‡ |
| multi-tab/follow-a-slow-link | Follow the slow details link and report its status | PASS | FAIL‡ |
| multi-tab/go-back | Read details, return to Home, and report the heading | FAIL‡ | TIMEOUT |
| dialogs-and-recovery/confirm-inside-a-modal | Delete the item and confirm in the page modal | PASS | FAIL‡ |
| dialogs-and-recovery/ask-then-act-on-the-answer | Choose the second account after the user clarifies | PASS | PASS |
| dialogs-and-recovery/native-confirm | Delete the item and accept the browser confirmation | PASS† | TIMEOUT |
| report/write | Read and report the status shown on the page | FAIL‡ | FAIL‡ |
PASS means the page-state predicate passed. PASS† means it passed but the agent did not reach the expected terminal status. FAIL means the predicate failed without a false completion. FAIL‡ means the agent reported completion but the predicate failed. TIMEOUT means the agent did not finish within its time limit and the page predicate failed. NR means no result was recorded for that task.
The initial Nanobrowser run did not save the last seven scenarios: multi-tab/open-a-tab hit Chromium OS EAGAIN (resource temporarily unavailable), the next two multi-tab scenarios failed during extension/profile startup, and the final four had no saved records. A one-worker follow-up run completed all seven without another browser startup error. Its page-scored results were: multi-tab/open-a-tab FAIL‡, multi-tab/follow-a-slow-link FAIL‡, multi-tab/go-back TIMEOUT, dialogs-and-recovery/confirm-inside-a-modal FAIL‡, dialogs-and-recovery/ask-then-act-on-the-answer PASS, dialogs-and-recovery/native-confirm TIMEOUT, and report/write FAIL‡. In total, one of the seven tasks met its page predicate, four reported completion without the requested page result, and two timed out. One successful serial rerun does not identify the cause of the original OS launch errors.
The canonical content-free source records are Ollama Client JSON and Nanobrowser JSON.
Run setup
Section titled “Run setup”- Model: codex/gpt-6-luna, medium reasoning effort.
- Endpoint: the same local Ollama Client olc OpenAI-compatible service for both products (http://127.0.0.1:8083/v1).
- Ollama Client candidate: commit f588c013.
- Nanobrowser: source commit 24a14b76, built locally from its lockfile; upstream version string 0.1.13.
- Nanobrowser’s Planner and Navigator both used the same model and medium reasoning effort. Its vision settings were enabled for the canvas tasks; analytics were disabled in the test profile.
- The task pages are deterministic synthetic fixtures. Each product has one recorded try per task; the seven Nanobrowser follow-up tries were run separately on 2026-09-28. These results do not measure repeatability or broad real-world browsing.
The saved benchmark records include the page-predicate result and terminal status for each attempted scenario. They do not preserve a detailed cause for every predicate failure; the task table reports what the evidence supports.
Nanobrowser benchmark practices
Section titled “Nanobrowser benchmark practices”At the tested Nanobrowser revision, its README describes a multi-agent Planner/Navigator workflow and configurable models/providers. Its package scripts include build and end-to-end testing commands. We did not find a first-party public benchmark suite or published score table in those sources at that revision.