Project · AI Tooling
A multi-bot pipeline that finds the UX issues automated testing misses. It runs on every test pass, deduplicates against known bugs, and verifies what survives in simulator, so reviewers only see net-new issues that have already been reproduced.
When the offshore manual testing team was replaced by automated test passes, the coverage gap wasn't obvious at first. Automated tests are written to verify behavior — they pass as long as the app does what it's programmed to do. They don't notice when something looks off: a misaligned UI element, a flow that's technically correct but confusing, a rendering issue that only appears in certain states.
Real users notice these things. There was no systematic way to catch them without reintroducing manual review, which was exactly what automation was supposed to eliminate. The question became: how do you get the coverage of a manual tester without the manual tester?
The biggest risk was noise. A pipeline that files inaccurate or duplicate bugs erodes engineering trust fast, and a system engineers learn to ignore is worse than no system at all. Getting it right required iteration: bugs had to be reproduced in a simulator before filing, not just theorized; correct details (description, build info, reproduction steps) had to be consistently attached; and the pipeline had to coexist with existing automated workflows without disruption. That calibration happened through review cycles with engineering and cross-functional partners, adjusting the detection logic until accuracy was high enough to trust.
Decision 01
Bot-per-concern, not one monolithEach stage of the pipeline runs as a separate bot — ingestion, analysis, dedup, filing. This means any stage can be updated, retrained, or replaced independently without rebuilding the whole pipeline. When the analysis bot needed refinement, the filing and dedup logic stayed untouched.
Decision 02
Dedup before testing, not afterThe dedup check runs before simulator testing, not after. Running a simulator test only to find out the issue is already filed wastes compute and time. Checking the bug database first means only net-new issues ever reach the testing stage.
Decision 03
Propose, don't auto-actThe original design filed confirmed issues automatically. Engineering pushed back: a pipeline that writes into their queue without a human in the loop is a trust problem before it's a tooling problem, and they were right. I pulled auto-filing and repositioned the pipeline around what it was actually best at, which was deduplication and simulator verification. It now hands reviewers a short list of net-new issues that have already been reproduced, instead of filing on their behalf. Less autonomous, considerably more used.
Started with one app and a few features. Now covers two apps with a full feature suite. In a representative month the pipeline surfaced 66 candidate issues, 10 of which were confirmed in simulator — and none of them were duplicates of something already in the bug database. The dedup stage is the reason that last number holds, and it's what makes the output worth a reviewer's time.