TSO behavior

VDCs in a geo-replicated environment have a heartbeat mechanism. Sustained loss of heartbeats for a configurable duration (by default, 15 minutes) indicates a network or site outage and the system transitions to identify the TSO.

ECS marks the unreachable site as TSO and the site status displays as Temporarily unavailable on the Replication Group Management page in the ECS Portal.

There are two important concepts that determine how the ECS system behaves during a TSO.

  • Owner site: If a bucket or object is created within a namespace in Site A, then Site A is the owner site of that bucket or object. When a TSO occurs, the behavior for read/write requests differs depending on whether the request is made from the site that owns the bucket or object, or from a non-owner site that does not own the primary copy of the object.
  • Access During Outage (ADO) bucket setting: Access to buckets and the objects within them during a TSO differs depending on whether the ADO setting is turned on for buckets. The ADO setting can be set at the bucket level; meaning you can turn this setting on for some buckets and off for others.
    • If the ADO setting is turned off for a bucket, strong consistency is maintained during a TSO by continuing to allow access to data owned by accessible sites and preventing access to data owned by a failed site.
    • If the ADO setting is turned on for a bucket, read and optionally write access to all geo-replicated data is allowed, including the data that is owned by the failed site. During a TSO, the data in the ADO-enabled bucket temporarily switches to eventual consistency; once all sites are back online it will revert back to strong consistency.