We built a repo with 4 tests that fail on purpose

After we shared AI Test Triage, the CEO of an engineering consultancy gave us short, fair feedback: he could not try it, because he had no failing tests on hand. Most people don't, when they are just curious.

So we built an example project with 4 tests that fail on purpose, each for a different reason: a real product bug, a broken test, a service missing in CI, and a timing problem. The AI got the product bug and the broken test right every time. On the missing service it went back and forth between two answers, which is fair, since you could argue either. On the timing test it said "unclear", because the real cause sits in code the report never shows. We published those misses next to the hits.

The demo page now also shows how accurate the AI is. On 40 test failures we labelled by hand, it classified 39 correctly, the same in all three runs. On the 10 trickiest, where the error name points the wrong way, it found the real cause every time, while a simple keyword rule found none. The one miss is explained on the page. The cases are hand-written, so this is an early test, not proof.

After you get a result, the demo shows the few lines you need to run the same triage in your own GitHub Actions pipeline, with a copy button. We also shipped two small releases of the Action: cleaner punctuation in its comments, and code names wrapped properly so GitHub never mistakes them for a username.

Written by Rushi Prajapati