Handling of broker failure anomalies ignored due to temporary self heal disable

[bsandeep23]

Issue

When self healing is disabled and a broker failure anomaly reaches the auto fix threshold, today's behavior is the anomaly gets ignored.
In cases where self healing is permanently disabled this is accepted. However in cases where self healing is disabled temporarily, this behavior is not ideal as the under replicated partitions from this broker failure gets fixed only with a human intervention or subsequent self heal for another anomaly.

Proposed Approach

When the broker failure anomaly reaches the auto fix threshold and self healing is disabled, instead of ignoring the anomaly recheck the anomaly after certain time period in the hope that self healing gets re enabled back and a fix can be triggered.

• Introduce a configurable max retry count so that the anomaly recheck is not done forever in cases where self healing is permanently disabled. After the tries exceed this max, the anomaly can be ignored.
• Introduce a a recheck interval config.