TSO behavior

VDCs in a geo-replicated environment have a heartbeat mechanism. Sustained loss of heartbeats for a configurable duration (by default, 15 minutes) indicates a network or site outage and the system transitions to identify the TSO.

NOTE: TSO behavior applies to IAM configuration as well, and when the IAM configuration is changed, the effect of those changes do not take effect immediately.

ECS marks the unreachable site as TSO and the site status displays as Temporarily unavailable on the Replication Group Management page in the ECS Portal.

There are two important concepts that determine how the ECS system behaves during a TSO.

  • Owner site: If a bucket or object is created within a namespace in Site A, and then Site A is the owner site of that bucket or object. When a TSO occurs, the behavior for read/write requests differs depending on whether the request is made from the site that owns the bucket or object, or from a nonowner site that does not own the primary copy of the object.
  • Access During Outage (ADO) bucket setting: Access to buckets and the objects within them during a TSO differs depending on whether the ADO setting is turned on for buckets. The ADO setting can be set at the bucket level; meaning you can turn this setting on for some buckets and off for others.
    • If the ADO setting is turned off for a bucket, strong consistency is maintained during a TSO by continuing to enable access to data owned by accessible sites and preventing access to data owned by a failed site.
    • If the ADO setting is turned on for a bucket, read and optionally write access to all geo-replicated data is enabled, including the data that is owned by the failed site. During a TSO, the data in the ADO-enabled bucket temporarily switches to eventual consistency. Once all sites are back online, it reverts to strong consistency.