Make it possible to automatically rerun selected flaky tests

[bsideup]

There are a few tests that fail due to the underlying Docker issues. Some examples:

• shared memory test - fails from time to time on Azure Pipelines
• stdout/stderr ordering
• Couchbase k/v

Since we agreed in #1594 that we should only rerun the tests that are known to be flaky, we should have a mechanism (e.g. with a help form JUnit) to rerun them automatically, N times (where N is ~1..3)

Such test cases (important! not classes) should be explicitly marked (e.g. with an annotation) as flaky. This will also make it possible to find usages of that annotation and discover all flaky tests in the project.

---

"Flaky test" here means a test that fails randomly, due to some timing issues, or CIs problem with resource allocation or the underlying Docker issues

[bsideup]

If somebody wants to work on this, please comment here, to avoid multiple contributors working on the same task.

[chungngoops]

I would like to work on this issue.

[bsideup]

@chungngoops thank you! Feel free to ask any questions you have before implementing it 👍

[bsideup]

FYI the current state can be found here:
#1680

[rnorth]

@chungngoops I wanted to have a play with one possible way of doing it in #1680, as it's causing us a lot of pain at the moment (our build should not be failing as it is right now, and we need to fix this urgently).

Consider #1680 an experimental approach though - if you have a better way, it'd be good to hear about it.

[chungngoops]

@rnorth: I have a similar approach, mine just directly implement TestRule interface.
I think you can have a variable to define how many time should we re-run a flaky test.

[rnorth]

@chungngoops thanks - I think being able to specify the number of retries is a nice thing to have, so I will add as an annotation property.

Shall we just use #1680? I think I'd like to try and complete this before next week.