Do not resync blocks in running store gateways during rollout deployment and container restart

[justinjung04]

What this PR does:

Problem

Currently the logic for existing store gateway instances to re-sync block is too aggressive, leading to unnecessary re-sync when pods are restarted due to OOM or when doing rollout deployment.
Unnecessary re-sync increases chance of OOM and query unavailability.

Current Behavior

• ring.HasReplicationSetChanged is used to determine whether re-sync should be performed (true if number of instances, or state/address/token of any instance changes)
• Store gateway is scaled up => Number of instances in the ring is increased, then the instance state changes from JOINING -> ACTIVE => existing instances re-sync blocks at most twice, once for number of instance change and once for state change
• Store gateway is scaled down => Instance state changes from ACTIVE -> LEAVING, then number of instances in the ring is decreased => existing instances re-sync blocks at most twice, once for state change and once for number of instance change
• Store gateway instance is restarted (OOM) => Instance state changes from ACTIVE -> JOINING -> ACTIVE => existing instances re-sync blocks at most twice, both for state change
• Store gateway instance is replaced by a new instance (rollout deployment) => Instance state changes from ACTIVE -> LEAVING, instance address changes, then new instance state changes from JOINING -> ACTIVE => existing instances re-sync blocks at most three times, twice for state change and once for address change

Proposed Behavior

• ring.HasReplicationSetChangedWithoutStateAndAddress is used to determine whether re-sync should be performed (true if number of instances, or token of any instance changes)
• Store gateway is scaled up => Number of instances in the ring is increased, then the instance state changes from JOINING -> ACTIVE => existing instances re-sync blocks at most once for number of instance change
• Store gateway is scaled down => Instance state changes from ACTIVE -> LEAVING, then number of instances in the ring is decreased => existing instances re-sync blocks at most once for number of instance change
• Store gateway instance is restarted (OOM) => Instance state changes from ACTIVE -> JOINING -> ACTIVE => existing instances do not re-sync blocks
• Store gateway instance is replaced by a new instance (rollout deployment) => Instance state changes from ACTIVE -> LEAVING, instance address changes, then new instance state changes from JOINING -> ACTIVE => existing instances do not re-sync blocks if tokens_file_path is set. Otherwise, existing instances re-sync blocks once

Which issue(s) this PR fixes:
n/a

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[yeya24]

The change looks good to me. Thanks for the detailed writeup and contribution!

[yeya24]

cc @alvinlin123 or @alanprot for another review.

[alanprot]

Nice!! This LGTM.