Where Automated Flaky-Test Fixes Still Need Human Judgment

Flaky tests are a special case in backlog automation, because "fixed" is harder to verify than it looks. A test that fails intermittently and then passes ten times in a row hasn't necessarily been fixed; it might just be failing less often, or failing under conditions the last ten runs didn't happen to hit.

Why flakiness resists mechanical verification

Most backlog work has a clean success signal: the build passes, the test suite is green, the linter stops complaining. Flaky tests break that pattern. The test suite passing after a change doesn't tell you the underlying race condition, timing assumption, or shared-state bug is gone; it tells you it didn't reproduce this time.

That's true whether a human or an automated agent makes the fix. It's a property of flaky tests, not a limitation specific to any tool.

What an automated workflow can reasonably do

Given a ticket that identifies a specific failing test and describes what's been observed (frequency, error message, any suspected cause), an agent can:

• Read the test and the code it exercises for common flakiness patterns: unhandled async timing, shared mutable state between tests, order-dependent assertions, environment-sensitive assumptions.
• Propose a fix addressing the pattern it finds.
• Run the test suite repeatedly to build some confidence the fix isn't obviously wrong.

What it can't reasonably do is guarantee the fix addresses root cause rather than symptom; no process can, from a bounded number of test runs.

Where human judgment stays load-bearing

This is the part worth stating directly rather than glossing over: reviewing a flaky-test fix means asking whether the explanation makes sense, not just whether the test passed more times. A fix that adds a sleep() before an assertion might make the test pass more reliably while doing nothing about the actual race condition, and a reviewer who understands the system is who catches that difference.

Practically, that means flaky-test fixes deserve review that reads the diff and the reasoning, not just the test-run outcome. It's a good default for any automated fix, and specifically the shape of review flaky-test PRs need.

How RepoFixer approaches this

RepoFixer treats flaky-test tickets the same as other backlog work structurally (isolated execution, a scoped pull request, no autonomous merge), but doesn't claim to independently discover flakiness across your suite or guarantee root-cause diagnosis on every ticket. It works from the ticket you give it, and the PR it opens is exactly as reviewable as any other change. See the flaky tests use case for what a typical ticket looks like, and how it works for the surrounding workflow.