Admin - Messages

Title Service Type Datesort ascending

North America - BLUEcloud Analytics

A small portion of customers using BLUEcloud Analytics in North America were experiencing issues with connecting to their projects this morning.  The issue was again caused by a deadlock on the database containing the metadata.  

SaaS-Atlanta Info 15 Nov 2017
3:47pm UTC

North America - BLUEcloud Analytics

A small portion of customers using BLUEcloud Analytics in North America are experiencing issues with connecting to their projects.  We are working on the problem and will provide updates as soon as they become available.


 

UPDATE:  Our investigation discovered a database deadlock which was preventing Microstrategy Intelligence servers from being able to connect to the metadata.  This failure to connect also sent the Intelligence servers into a stop/start loop.  We have cleared out the lock and the Microstrategy Intelligence servers are starting up normally.

SaaS-Atlanta Info 14 Nov 2017
2:41pm UTC

RESOLVED - EOS.Web - Investigating Latency

The following event was resolved

  • We are currently proactively investigating latency alarms for select EOS.Web VM's.  At this time we do not consider this a widespread problem.  Updates to be posted here.
  • We believe we have identified the offending vm.  We have decided to quckly shutdown the vm and run a backend vmlevel procedure to clear the latency.  This is not expected to take more than 5 minutes.
  • The backend procedure has completed and the vm in question has been booted.  Service for this specific set of customers should be restored shortly.
  • Monitoring service checks for this specific set of customers are clearing
  • We have identified another vm and are taking the same action as described above. The next update will be posted when services for this single vm have been restored.
  • Monitoring service checks for 2nd specific set of customers are green.
SaaS-Atlanta Info 09 Nov 2017
12:36pm UTC

RESOLVED - Odd Enterprise latency - hit and miss

We received the following update from our colocation provider on November 7, 2017 at 17:23 CDT.  This will service as a final update


On Monday, November 6th, QTS Networking Team engaged our Upstream Provider regarding an internet connectivity issue. QTS has been advised that this issue is resolved. At this time, we are awaiting a Reason of Outage (RFO) from our Upstream Provider.


QTS upstream provider has a configuration issue that impacted IP services. They have reverted a policy change to restore services to a stable state. QTS will monitor for stability and route traffic back to normal during maintenance hours.


We received the following from our colocation partner just a few moments ago.

  • QTS is aware of a connectivity issue affecting internet traffic. At this time, QTS routed traffic away from the affected internet provider so that QTS customers would not be impacted. QTS is closely monitoring this situation and will provide more information as they become available.

Our colocation partner alerted us to a known problem with connections into Atlanta.  They are looking into the situation and contacting upstream providers.  Once we receive word from them we will update this page. 


Our colocation partner has confirmed they are receiving many calls about a potential upstream provider problem.  They are currently looking into the situation


We are opening a ticket with our colocation partner to see if there are any issues upstream causing all the problems with Enterprise across several clusters as well as some Symphony customers.


We are currently investigating oddities in Atlanta where some Enterprise and Symphony connections up and down. There are no obvious infrastructure issues that we have been able to identify.  We have many people working on this and hope to have more details soon.  

SaaS-Atlanta Info 07 Nov 2017
3:14am UTC

RESOLVED - EOS.Web Cluster has extreme performance degradation

The following service event is now resolved and all customer systems should be operating normally.  We will be performing an analysis of the service disruption in the coming days.  Once complete, a root cause summary will be provided to EOS Client Services.  We hope to have the summary finalized by the middle part of next week.  


Our monitoring has alerted us to performance degradation on one of our EOS.Web clusters.  The SaaS team is working to resolve this as quickly as possible.  Service may temporarily be out for multiple clusters during the resolution process.  Updates to be posted here.

SaaS-Atlanta Info 02 Nov 2017
3:29pm UTC

RESOLVED - Investigating Enterprise Performance Issues

The following event has been resolved.


Our monitoring has alerted us to performance degredation on one of our Enterprise / Portfolio clusters.  Updates to be posted here.

  • 09:10 CDT - decision made to shutdown the database.  Connections to database are much higher than normal which is causing massive system load.  The hope is the database shutdown and restart will kill off whatever processes is causing site instability.   
  • 10:15 CDT - after 1 hour the database finally started.  A decision was madeto shutdown the database and unmount the entire cluster.
  • 11:00 CDT - continuing to shutdown cluster related application servers.  We believe that we have identified the source of latency but want to make sure we take the Enterprise services down cleanly.  
  • 11:05 CDT - In short, this event was related to controller performance.  Though not critically high, the load was high enough to cause I/O performance problems on the SAN.  After we moved applicable LUN to another controller port the average request service times at the disk level improved dramatically. 
  • 11:08 CDT - We are in the process of final testing and hope to have service restored within 30 minutes.   
  • 11:13 CDT - Restarting app servers one at a time.  Some customers should see service restoration now.
  • 11:34 CDT - All processes have been started for the Enterprise cluster.  It does take some time for each to become fully accessible.  The disk service times are much improved over those that my staff were seeing when this event initially began this morning. 
  • 11:50 CDT - Our monitoring has cleared all customer service checks.  Environment stabalized.  Email forthcoming to affected customers.  
SaaS-Atlanta Info 31 Oct 2017
1:56pm UTC

BLUEcloud Analytics Maintenance Downtime Schedule for Oct. 26

SirsiDynix will be conducting planned maintenance on BLUEcloud Analytics in all regions. During this maintenance window, Microstrategy will be unavailable as we perform work on the backend components that manage data extractions. To avoid interrupting scheduled extractions, the maintenance window will occur during regional workday hours.

The planned schedule is below:

    For customers using APAC servers (the URL you use begins with apac1), downtime is scheduled for Thursday, October 26, 2017 starting at 8:00 a.m. AEDT and ending at 10:00 a.m. AEDT.

There is no action required by customers. We look forward to continuing to improve the BLUEcloud Analytics experience.

SaaS-Melbourne Maintenance 26 Oct 2017
12:00am UTC

BLUEcloud Analytics Maintenance Downtime Schedule for Oct. 25

SirsiDynix will be conducting planned maintenance on BLUEcloud Analytics in all regions. During this maintenance window, Microstrategy will be unavailable as we perform work on the backend components that manage data extractions. To avoid interrupting scheduled extractions, the maintenance window will occur during regional workday hours.

The planned schedule is below:

    For customers using EMEA servers (the URL you use begins with emea1), downtime is scheduled for Wednesday, October 25, 2017 starting at 6:00 p.m. BST and ending at 8:00 p.m. BST.

There is no action required by customers. We look forward to continuing to improve the BLUEcloud Analytics experience.

SaaS-EU Maintenance 25 Oct 2017
6:00am UTC

BLUEcloud Analytics Maintenance Downtime Schedule for Oct. 24

SirsiDynix will be conducting planned maintenance on BLUEcloud Analytics in all regions. During this maintenance window, Microstrategy will be unavailable as we perform work on the backend components that manage data extractions. To avoid interrupting scheduled extractions, the maintenance window will occur during regional workday hours.

The planned schedule is below:

    For customers using North American servers (the URL you use begins with na1), downtime is scheduled for Tuesday, October 24, 2017 starting at 9:00 a.m. MDT. The process is projected to be completed by 11:00 a.m. MDT.

There is no action required by customers. We look forward to continuing to improve the BLUEcloud Analytics experience.

SaaS-Atlanta Maintenance 24 Oct 2017
3:56pm UTC

Investigating Symphony connectivity issues in EMEA

We are working with our hosting partner to identify the reason for connectivity issues in EMEA. Currently this seems to only be impacts one or two customers.  Updates will be posted here. 

Update - 2:20pm BST - We believe that things have now stabilised, although we are continuing to investigate with our hosting partner.

Update - 4:10pm BST - Out hosting partner has confirmed that the situation has stabilised. Please contact our customer support team if you experience any further issues.

SaaS-EU Info 16 Oct 2017
12:43pm UTC

RESOLVED - Enterprise - intermittent access

The reboot is complete and the new configuration is in place.  All services should now return to normal operating conditions.


We are in the process of rebooting the Montreal firewall to implement changes which will stabilize connections to Enterprise.


We are continuing to investigate reports of intermittent access to hosted Enterprise environments in Canada.  Updates will be posted here. 

SaaS-Montreal Info 10 Oct 2017
7:59pm UTC

RESOLVED - Horizon Connectivity Issues - Single Group

The following event has been resolved.

We are in the process of rebooting one of the Horizon SQL servers that is causing login problems foe some customers.

SaaS-Atlanta Info 07 Oct 2017
2:24pm UTC

RESOLVED - Network Issue - Montreal

The following event has been resolved.


We are continuing to investigate a network related issue in Montreal.  Some sites in Montreal may experience periods of time that websites are available and then are suddenly unreachable.  We are in the process of contacting our colocation partner for additional troubleshooting.   

SaaS-Montreal Info 07 Oct 2017
3:20am UTC

COMPLETE - Scheduled Maintenance - Single Symphony Server

The following scheduled maintenance was completed without issue. 


There will be a brief maintenance window for a single Symphony server beginning at 11:30pm AEDT on Tuesday, October 2, 2017.  Customers have been notified individually via email.

SaaS-Melbourne Maintenance 03 Oct 2017
12:30pm UTC

RESOLVED - Symphony Service Disruption

RESOLVED - the following event has been resolved


Due to a hardware failure we are currently experiencing an outage on some Symphony systems. We are working to restore service.

 

SaaS-Atlanta Info 30 Sep 2017
10:11am UTC

RESOLVED - BLUECloud Services

We are currently investigating login issues with BLUEcloud applications in the APAC region.  We will post updates as they become available.


UPDATE:  Access to BLUEcloud applications in APAC has been restored.  Please contact us at https://support.sirsidynix.com if you have any questions.

SaaS-Melbourne Info 19 Sep 2017
11:27am UTC

RESOLVED - BlueCloud Analytics Service Disruption

We are currently experiencing an outage with the BlueCloud applications in APAC.


 

UPDATE:  Access to BLUEcloud applications in APAC has been restored.  Please contact us at https://support.sirsidynix.com (link is external) if you have any questions.

SaaS-Melbourne Info 19 Sep 2017
9:12am UTC

RESOLVED - Network Disruption - Symphony, Enterprise / Portfolio, EOS.Web

UPDATE: The following event has been resolved.  Over the coming days we will be working on a root cause and will provide your LRM's with the final copy.  I hope to have this in their hands by this Friday.  Thank you all for your patience. Please contact us at https://support.sirsidynix.com (link is external) (link is external) if you have any questions.


We have multiple network engineers continuing to troubleshoot the disruption for the following services in Atlanta.  At this time we do not have an estimate on service restoration. Updates will continue to be posted here.  


Symphony

Enterprise / Portfolio

EOS.Web

SaaS-Atlanta Info 19 Sep 2017
4:25am UTC

RESOLVED - Network Disruption - EOS.Web

The following event has been resolved.


We have multiple network engineers continuing to troubleshoot the disruption for the EOS.Web services in the EU.  At this time we do not have an estimate on service restoration. Updates will continue to be posted here.

SaaS-EU Info 19 Sep 2017
4:25am UTC

Scheduled Network Maintenance - COMPLETE

The scheduled maintenance was completed successfully.


WHEN: Sunday, September 17th, 2017 beginning 12:01 AM EDT

DURATION:  3 hours

WHAT:  Firewall Upgrade

WHY:  Scheduled Hardware Refresh

IMPACT:  Access to those products listed below will be unavailable during the maintenance window.  JetStream / BLUEcloud products will not be affected by this maintenance.  Horizon customers were notified separately. 

ACTION:  None. All servers should remain up during this period so all nightly processing should run normally.

PRODUCTS:

  • Symphony Production & Test
  • Enterprise / Portfolio
  • EOS.Web
  • Director's Station
  • Web Reporter
  • MobileCirc
  • eRes
SaaS-Atlanta Maintenance 17 Sep 2017
4:00am UTC

Pages


Deprecated: Methods with the same name as their class will not be constructors in a future version of PHP; backup_file has a deprecated constructor in /home/sirsidyn/public_html/sites/all/modules/backup_migrate/includes/files.inc on line 205