Admin - Messages

Title Service Type Datesort ascending

BLUEcloud Service Disruption

We are currently investigating a problem with the BLUEcloud environment in the APAC region.  We are working to restore service as quickly as possible. Updates to be posted here.

 

UPDATE:

Issues were found while patching the environment that led the unexpected extended down time of the environment. The issues were resolved and the patching completed successfully. All services have been restored.

SaaS-Melbourne Update 16 Jan 2018
9:31pm UTC

North America - BLUEcloud Analytics

Update 12:10pm (MST) - The issue has been resolved.

We are currently investigating an issue with BLUEcloud Analytics in North America were it may not be possible to log in or access reports.

SaaS-Atlanta Info 03 Jan 2018
6:09pm UTC

EOS.Web Login, Reporting, and Email Issues

Update - the issue preventing some customers from being able to log into EOS.Web, and other customers from using email or reporting capabilities has been fully resolved.  All customers should now have normal functionality.

We are experiencing issues at the data center preventing some customers from being able to log into their EOS.Web systems.  Others are able to login and use EOS.Web but are not able to use the email capabilities, or run certain reports.  Our SaaS team is working to resolve all issues as quickly as possible, the resolution is expected later today.  We will keep you updated here.

SaaS-Atlanta Alert 03 Jan 2018
4:04pm UTC

Widespread outage in Atlanta

(0609 MT) We are currently investigating several critical alerts for many services in the Atlanta facility. This may be impacting access to EOS.Web, Symphony, Horizon, Enterprise and other ancillary products. Updates will be posted here.

Update 1 (0635 MT) We have multiple engineers investigating. From our initial assessment it appears to be SAN / fabric / storage related. We will continue to post updates here.

Update 2 (0722 MT) We are continuing to focus on storage components, specifically the SAN switches. Internal escalation, to include members of management, has taken place.

Update 3 (0752 MT) At this time we are working with our colocation partner to review lights on various devices within our space. We are seeing 2 switch ports offline which is impacting the hosts ability to access their respective storage devices.

Update 4 (0912 MT) We are currently rebooting one of the suspect SAN switches. We believe this will reset the SAN links allowing host to storage communication.

Update 5 (1015 MT) At this time we are testing areas of the infrastructure prior to bringing hosts and services online.

Update 6 (1129 MT) Testing is near completion and preparations for service restoration are underway.

Update 7 (1220 MT) Individual service restorations are underway. Anticipate all services restored by 2PM (MT).

Update 8 (1321 MT) Roughly 50% of sytems are restored to service, including all EOS.web sites.

Update 9 (1425 MT) Restorations continue - at about 80% completed

Update 10 (1530 MT) A handful of restorations remain.

Update 11 (1635 MT) Addressing individual outage-related issues as reported by customers. We continue to have staff working this issue until all services are restored and alerts are cleared.

Update 12 (1755 MT) Final hourly update - Most monitoring is now green. SaaS teams are working through remaining details and some test instances. Anticipate "all clear" by 1900 (MT). Please contact SirsiDynix Customer Support with any issues - use the Critical Care number for down systems. Staff are online through the holidays.

SaaS-Atlanta Alert 23 Dec 2017
1:09pm UTC

North America - BLUEcloud Analytics

Update 11:00am - The issue has been resolved.

We are currently investigating an issue with BLUEcloud Analytics in North America were it may not be possible to log in or access reports.

SaaS-Atlanta Info 19 Dec 2017
3:16pm UTC

COMPLETE - Unscheduled HIP Server Reboot Required

Complete

We need to perform an unscheduled reboot of a single HIP server in Atlanta.  We hope to have service returned to normal very shortly.

SaaS-Atlanta Info 03 Dec 2017
5:46pm UTC

North America - BLUEcloud Analytics

A small portion of customers using BLUEcloud Analytics in North America were experiencing issues with connecting to their projects this morning.  The issue was again caused by a deadlock on the database containing the metadata.  

SaaS-Atlanta Info 15 Nov 2017
3:47pm UTC

North America - BLUEcloud Analytics

A small portion of customers using BLUEcloud Analytics in North America are experiencing issues with connecting to their projects.  We are working on the problem and will provide updates as soon as they become available.


 

UPDATE:  Our investigation discovered a database deadlock which was preventing Microstrategy Intelligence servers from being able to connect to the metadata.  This failure to connect also sent the Intelligence servers into a stop/start loop.  We have cleared out the lock and the Microstrategy Intelligence servers are starting up normally.

SaaS-Atlanta Info 14 Nov 2017
2:41pm UTC

RESOLVED - EOS.Web - Investigating Latency

The following event was resolved

  • We are currently proactively investigating latency alarms for select EOS.Web VM's.  At this time we do not consider this a widespread problem.  Updates to be posted here.
  • We believe we have identified the offending vm.  We have decided to quckly shutdown the vm and run a backend vmlevel procedure to clear the latency.  This is not expected to take more than 5 minutes.
  • The backend procedure has completed and the vm in question has been booted.  Service for this specific set of customers should be restored shortly.
  • Monitoring service checks for this specific set of customers are clearing
  • We have identified another vm and are taking the same action as described above. The next update will be posted when services for this single vm have been restored.
  • Monitoring service checks for 2nd specific set of customers are green.
SaaS-Atlanta Info 09 Nov 2017
12:36pm UTC

RESOLVED - Odd Enterprise latency - hit and miss

We received the following update from our colocation provider on November 7, 2017 at 17:23 CDT.  This will service as a final update


On Monday, November 6th, QTS Networking Team engaged our Upstream Provider regarding an internet connectivity issue. QTS has been advised that this issue is resolved. At this time, we are awaiting a Reason of Outage (RFO) from our Upstream Provider.


QTS upstream provider has a configuration issue that impacted IP services. They have reverted a policy change to restore services to a stable state. QTS will monitor for stability and route traffic back to normal during maintenance hours.


We received the following from our colocation partner just a few moments ago.

  • QTS is aware of a connectivity issue affecting internet traffic. At this time, QTS routed traffic away from the affected internet provider so that QTS customers would not be impacted. QTS is closely monitoring this situation and will provide more information as they become available.

Our colocation partner alerted us to a known problem with connections into Atlanta.  They are looking into the situation and contacting upstream providers.  Once we receive word from them we will update this page. 


Our colocation partner has confirmed they are receiving many calls about a potential upstream provider problem.  They are currently looking into the situation


We are opening a ticket with our colocation partner to see if there are any issues upstream causing all the problems with Enterprise across several clusters as well as some Symphony customers.


We are currently investigating oddities in Atlanta where some Enterprise and Symphony connections up and down. There are no obvious infrastructure issues that we have been able to identify.  We have many people working on this and hope to have more details soon.  

SaaS-Atlanta Info 07 Nov 2017
3:14am UTC

RESOLVED - EOS.Web Cluster has extreme performance degradation

The following service event is now resolved and all customer systems should be operating normally.  We will be performing an analysis of the service disruption in the coming days.  Once complete, a root cause summary will be provided to EOS Client Services.  We hope to have the summary finalized by the middle part of next week.  


Our monitoring has alerted us to performance degradation on one of our EOS.Web clusters.  The SaaS team is working to resolve this as quickly as possible.  Service may temporarily be out for multiple clusters during the resolution process.  Updates to be posted here.

SaaS-Atlanta Info 02 Nov 2017
3:29pm UTC

RESOLVED - Investigating Enterprise Performance Issues

The following event has been resolved.


Our monitoring has alerted us to performance degredation on one of our Enterprise / Portfolio clusters.  Updates to be posted here.

  • 09:10 CDT - decision made to shutdown the database.  Connections to database are much higher than normal which is causing massive system load.  The hope is the database shutdown and restart will kill off whatever processes is causing site instability.   
  • 10:15 CDT - after 1 hour the database finally started.  A decision was madeto shutdown the database and unmount the entire cluster.
  • 11:00 CDT - continuing to shutdown cluster related application servers.  We believe that we have identified the source of latency but want to make sure we take the Enterprise services down cleanly.  
  • 11:05 CDT - In short, this event was related to controller performance.  Though not critically high, the load was high enough to cause I/O performance problems on the SAN.  After we moved applicable LUN to another controller port the average request service times at the disk level improved dramatically. 
  • 11:08 CDT - We are in the process of final testing and hope to have service restored within 30 minutes.   
  • 11:13 CDT - Restarting app servers one at a time.  Some customers should see service restoration now.
  • 11:34 CDT - All processes have been started for the Enterprise cluster.  It does take some time for each to become fully accessible.  The disk service times are much improved over those that my staff were seeing when this event initially began this morning. 
  • 11:50 CDT - Our monitoring has cleared all customer service checks.  Environment stabalized.  Email forthcoming to affected customers.  
SaaS-Atlanta Info 31 Oct 2017
1:56pm UTC

BLUEcloud Analytics Maintenance Downtime Schedule for Oct. 26

SirsiDynix will be conducting planned maintenance on BLUEcloud Analytics in all regions. During this maintenance window, Microstrategy will be unavailable as we perform work on the backend components that manage data extractions. To avoid interrupting scheduled extractions, the maintenance window will occur during regional workday hours.

The planned schedule is below:

    For customers using APAC servers (the URL you use begins with apac1), downtime is scheduled for Thursday, October 26, 2017 starting at 8:00 a.m. AEDT and ending at 10:00 a.m. AEDT.

There is no action required by customers. We look forward to continuing to improve the BLUEcloud Analytics experience.

SaaS-Melbourne Maintenance 26 Oct 2017
12:00am UTC

BLUEcloud Analytics Maintenance Downtime Schedule for Oct. 25

SirsiDynix will be conducting planned maintenance on BLUEcloud Analytics in all regions. During this maintenance window, Microstrategy will be unavailable as we perform work on the backend components that manage data extractions. To avoid interrupting scheduled extractions, the maintenance window will occur during regional workday hours.

The planned schedule is below:

    For customers using EMEA servers (the URL you use begins with emea1), downtime is scheduled for Wednesday, October 25, 2017 starting at 6:00 p.m. BST and ending at 8:00 p.m. BST.

There is no action required by customers. We look forward to continuing to improve the BLUEcloud Analytics experience.

SaaS-EU Maintenance 25 Oct 2017
6:00am UTC

BLUEcloud Analytics Maintenance Downtime Schedule for Oct. 24

SirsiDynix will be conducting planned maintenance on BLUEcloud Analytics in all regions. During this maintenance window, Microstrategy will be unavailable as we perform work on the backend components that manage data extractions. To avoid interrupting scheduled extractions, the maintenance window will occur during regional workday hours.

The planned schedule is below:

    For customers using North American servers (the URL you use begins with na1), downtime is scheduled for Tuesday, October 24, 2017 starting at 9:00 a.m. MDT. The process is projected to be completed by 11:00 a.m. MDT.

There is no action required by customers. We look forward to continuing to improve the BLUEcloud Analytics experience.

SaaS-Atlanta Maintenance 24 Oct 2017
3:56pm UTC

Investigating Symphony connectivity issues in EMEA

We are working with our hosting partner to identify the reason for connectivity issues in EMEA. Currently this seems to only be impacts one or two customers.  Updates will be posted here. 

Update - 2:20pm BST - We believe that things have now stabilised, although we are continuing to investigate with our hosting partner.

Update - 4:10pm BST - Out hosting partner has confirmed that the situation has stabilised. Please contact our customer support team if you experience any further issues.

SaaS-EU Info 16 Oct 2017
12:43pm UTC

RESOLVED - Enterprise - intermittent access

The reboot is complete and the new configuration is in place.  All services should now return to normal operating conditions.


We are in the process of rebooting the Montreal firewall to implement changes which will stabilize connections to Enterprise.


We are continuing to investigate reports of intermittent access to hosted Enterprise environments in Canada.  Updates will be posted here. 

SaaS-Montreal Info 10 Oct 2017
7:59pm UTC

RESOLVED - Horizon Connectivity Issues - Single Group

The following event has been resolved.

We are in the process of rebooting one of the Horizon SQL servers that is causing login problems foe some customers.

SaaS-Atlanta Info 07 Oct 2017
2:24pm UTC

RESOLVED - Network Issue - Montreal

The following event has been resolved.


We are continuing to investigate a network related issue in Montreal.  Some sites in Montreal may experience periods of time that websites are available and then are suddenly unreachable.  We are in the process of contacting our colocation partner for additional troubleshooting.   

SaaS-Montreal Info 07 Oct 2017
3:20am UTC

COMPLETE - Scheduled Maintenance - Single Symphony Server

The following scheduled maintenance was completed without issue. 


There will be a brief maintenance window for a single Symphony server beginning at 11:30pm AEDT on Tuesday, October 2, 2017.  Customers have been notified individually via email.

SaaS-Melbourne Maintenance 03 Oct 2017
12:30pm UTC

Pages