Use AI Test Triage on your own repo in 5 lines

AI Test Triage is a free GitHub Action. It reads the test report after a failed build and posts a short comment on the pull request: each failure, its likely cause, and a suggested next step. It works with any test runner that writes JUnit XML, which covers most Java, Python and JavaScript setups. Setting it up is one step in a workflow you already have.

Add this after the step that runs your tests:

- name: AI test triage
  if: failure()
  uses: Rushi7795/ai-test-triage@v1
  with:
    anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}

Then store an Anthropic API key in your repository as the secret ANTHROPIC_API_KEY (Settings, Secrets and variables, Actions, New repository secret). That is the whole setup. The "if: failure()" line means the step only runs when the tests fail, so a green build costs nothing.

What happens on the next failed build: the Action finds the JUnit XML your runner wrote, groups the related failures, sends the error text to the model, and posts one comment on the pull request. For each failure you get the test name, one of four labels (PRODUCT BUG, TEST BUG, ENVIRONMENT/FLAKY or UNCLEAR), a short explanation and a suggested first thing to check. Code names in the comment are wrapped so GitHub never mistakes them for user mentions.

What it never does: it never changes the build result. A red build stays red; the comment is extra information, not a verdict. It does not retry tests, edit code or open issues. Without an API key it still lists the failures, just without the explanation. If the Action itself hits an error, it says so and exits cleanly rather than failing your workflow.

How much to trust it: on a hand-written set of 40 labelled failures it classified 39 correctly, the same in three runs, and on the 10 tricky cases where the exception name points the wrong way it found the real cause every time. The cases were written by us, so treat that as an early test, not proof. The one it got wrong in all three runs was a test that waited the full 30 seconds behind a spinner, which the AI read as the page having changed. It is a strong first guess that saves you opening a long log, not a replacement for reading the failure.

If you want to see it before touching your own pipeline, there is an example repo with four tests that fail on purpose and an open pull request showing the comment it posts, and the same model runs in the browser on our demo page, where you can paste a failure of your own.

Written by Rushi Prajapati