Build -
The smoke test is built, and there is a new self-healing demo
After shipping a demo with a retired model, we added a daily check on the live demos. We also added a second live demo for broken test locators.
In our last entry we said the live AI Test Triage demo went out calling a retired model, and that nothing alerted us. We have now built the fix. A small GitHub Action calls both live demos once a day with a sample input. If either one returns an error, or the AI reply has no classification, the job fails and GitHub emails us.
It needs no secrets, because the demo endpoints are public. It retries twice before it fails, so one network blip does not wake us up. We also set a recurring reminder to read Anthropic's model deprecations page, so we hear about a retirement date before it arrives.
The new demo is self-healing test locators. A UI change breaks a locator, so the test cannot find its button. Paste the old locator and a snippet of the page's current HTML, and the AI proposes a replacement with a confidence score and a backup. It runs on a real model, and you can try it on the demos page.
Two honest notes. The demo shows the AI step on its own. Applying the new locator and re-running the test happens in our open-source framework, and the page links to it. And the AI gives a strong first guess, not a guarantee, which is why the framework re-runs the test to confirm. Both demos now also say that pasted inputs are logged, so please do not paste secrets or confidential code.
Written by Rushi Prajapati