Widespread outage in Atlanta

(0609 MT) We are currently investigating several critical alerts for many services in the Atlanta facility. This may be impacting access to EOS.Web, Symphony, Horizon, Enterprise and other ancillary products. Updates will be posted here.

Update 1 (0635 MT) We have multiple engineers investigating. From our initial assessment it appears to be SAN / fabric / storage related. We will continue to post updates here.

Update 2 (0722 MT) We are continuing to focus on storage components, specifically the SAN switches. Internal escalation, to include members of management, has taken place.

Update 3 (0752 MT) At this time we are working with our colocation partner to review lights on various devices within our space. We are seeing 2 switch ports offline which is impacting the hosts ability to access their respective storage devices.

Update 4 (0912 MT) We are currently rebooting one of the suspect SAN switches. We believe this will reset the SAN links allowing host to storage communication.

Update 5 (1015 MT) At this time we are testing areas of the infrastructure prior to bringing hosts and services online.

Update 6 (1129 MT) Testing is near completion and preparations for service restoration are underway.

Update 7 (1220 MT) Individual service restorations are underway. Anticipate all services restored by 2PM (MT).

Update 8 (1321 MT) Roughly 50% of sytems are restored to service, including all EOS.web sites.

Update 9 (1425 MT) Restorations continue - at about 80% completed

Update 10 (1530 MT) A handful of restorations remain.

Update 11 (1635 MT) Addressing individual outage-related issues as reported by customers. We continue to have staff working this issue until all services are restored and alerts are cleared.

Update 12 (1755 MT) Final hourly update - Most monitoring is now green. SaaS teams are working through remaining details and some test instances. Anticipate "all clear" by 1900 (MT). Please contact SirsiDynix Customer Support with any issues - use the Critical Care number for down systems. Staff are online through the holidays.

Sat, 23 Dec 2017 - 1:09pm UTC to Sun, 24 Dec 2017 - 12:55am UTC