tag:status.assurestor.com,2005:/historyAssurestor Status - Incident History2024-03-29T05:23:59+00:00Assurestortag:status.assurestor.com,2005:Incident/203526412024-03-25T17:33:03+00:002024-03-25T17:33:03+00:00EMERGENCY MAINTENANCE - dr2cloud Zerto<p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>17:33</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>17:03</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>17:02</var> GMT</small><br><strong>Scheduled</strong> - We will be performing some emergency planned maintenance on our storage platform, to increase resources due to some identified performance issues. During this time users may experience brief (<1 minute) interruption in replication.<br /><br />Should anyone see any VPG out of sync for an extended period of time post this notification, please raise a ticket to our service desk. <br /><br />Note. Replication will continue even if the Cloud ZVM is showing as disconnected</p>tag:status.assurestor.com,2005:Incident/202902092024-03-18T10:49:41+00:002024-03-18T10:49:41+00:00Zerto Analytics Issue impacting DR2Cloud and Zerto Aggregation reporting<p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>10:49</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>10:35</var> GMT</small><br><strong>Update</strong> - Issue has been reported to Zerto, awaiting feedback and ETA to resolve</p><p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>10:34</var> GMT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.assurestor.com,2005:Incident/199926452024-02-16T15:00:57+00:002024-02-16T15:00:57+00:00PLANNED MAINTENANCE - dr2cloud Zerto<p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>15:00</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>13:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>15:27</var> GMT</small><br><strong>Scheduled</strong> - We will be performing some planned maintenance on the dr2cloud platform as we upgrade to version 10. During this time users may experience intermittent access to the affected platform.<br />Note. Replication will continue even if the Cloud ZVM is showing as disconnected</p>tag:status.assurestor.com,2005:Incident/199735772024-02-14T18:00:58+00:002024-02-14T18:00:58+00:00PLANNED MAINTENANCE - dr2cloud Zerto<p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>18:00</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>14:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>09:44</var> GMT</small><br><strong>Scheduled</strong> - We will be performing some planned maintenance on the dr2cloud platform as we upgrade to version 10. During this time users may experience intermittent access to the affected platform.<br />Note. Replication will continue even if the Cloud ZVM is showing as disconnected</p>tag:status.assurestor.com,2005:Incident/189875792023-11-01T11:23:07+00:002023-11-01T11:23:07+00:00my2cloud Service Down<p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>11:23</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>10:20</var> GMT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the result.</p><p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>09:34</var> GMT</small><br><strong>Investigating</strong> - We are currently investigating the issue. Users may not have access to the service at this time.</p>tag:status.assurestor.com,2005:Incident/185354992023-09-19T00:44:46+01:002023-09-19T00:44:46+01:00dr2cloud UK1 communication issues with VCD<p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>00:44</var> BST</small><br><strong>Resolved</strong> - The issue has been identified as VCD MQTT self signed certificates having expired, this has now been fixed and full VCD connectivity restored meaning all VPGs are no longer being flagged as unhealthy.</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>18:29</var> BST</small><br><strong>Update</strong> - We are continuing to investigate this issue with Zerto engineers to understand why the error is being reported.</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>15:19</var> BST</small><br><strong>Investigating</strong> - We are currently experiencing an communication issues with VCD at the dr2cloud UK1 site. <br /><br />Engineers have been deployed to investigate this and we endeavour to have this operational again soon. <br /><br />Please note, replication and data IS in tact, its simply the connection to VCD that is flagging the mass alerts.</p>tag:status.assurestor.com,2005:Incident/185328152023-09-18T18:23:31+01:002023-09-18T18:23:31+01:00backup2cloud DB Issue<p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>18:23</var> BST</small><br><strong>Resolved</strong> - We have now seen multiple backups complete successfully and are confident this issue is now resolved.</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>17:32</var> BST</small><br><strong>Monitoring</strong> - The new DB server is being monitored but we expect all scheduled backups to now be able to run and connect to the cloud service.</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>17:31</var> BST</small><br><strong>Identified</strong> - The Asigra DB server had to be rebuilt and have its latest DB backup restored.</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>09:30</var> BST</small><br><strong>Investigating</strong> - We are currently experiencing some issues with the backup2cloud DBs that is currently being investigated. Access to the service will be impacted at this time.</p>tag:status.assurestor.com,2005:Incident/181719622023-08-20T16:00:20+01:002023-08-20T16:00:20+01:00PLANNED MAINTENANCE - veeam2cloud Management Console<p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>16:00</var> BST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>14:01</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>11:27</var> BST</small><br><strong>Scheduled</strong> - We will be performing some minor planned maintenance on the veeam2cloud platform, during this time users may experience intermittent access to the veeam2cloud Management Console.</p>tag:status.assurestor.com,2005:Incident/178142102023-07-11T17:00:18+01:002023-07-11T17:00:18+01:00PLANNED MAINTENANCE - dr2cloud Zerto<p><small>Jul <var data-var='date'>11</var>, <var data-var='time'>17:00</var> BST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'>11</var>, <var data-var='time'>15:00</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'>10</var>, <var data-var='time'>14:42</var> BST</small><br><strong>Scheduled</strong> - We will be performing some minor planned maintenance on the dr2cloud platform, during this time users may experience intermittent access to the affected platform.<br />Note. Replication will continue even if the Cloud ZVM is showing as disconnected</p>tag:status.assurestor.com,2005:Incident/175703952023-06-16T12:01:43+01:002023-06-16T12:01:43+01:00PLANNED MAINTENANCE - dr2cloud Zerto<p><small>Jun <var data-var='date'>16</var>, <var data-var='time'>12:01</var> BST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>16</var>, <var data-var='time'>10:00</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>11:12</var> BST</small><br><strong>Scheduled</strong> - We will be performing some planned maintenance on the dr2cloud platform, during this time users may experience intermittent access to the affected platform.<br />Note. Replication will continue even if the Cloud ZVM is showing as disconnected</p>tag:status.assurestor.com,2005:Incident/175586182023-06-13T16:36:16+01:002023-06-13T16:36:16+01:00Veeam DC2 - Gateways are down<p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>16:36</var> BST</small><br><strong>Resolved</strong> - The incident has been resolved with all cloud gateways being back online.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>12:45</var> BST</small><br><strong>Investigating</strong> - Due to unforeseen events, the Veeam Gateway servers are currently down at DC2. <br /><br />We have engineers working on restoring functionality ASAP.</p>tag:status.assurestor.com,2005:Incident/174796342023-06-08T12:02:15+01:002023-06-08T12:02:15+01:00PLANNED MAINTENANCE - dr2cloud Zerto<p><small>Jun <var data-var='date'> 8</var>, <var data-var='time'>12:02</var> BST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'> 8</var>, <var data-var='time'>10:00</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>15:56</var> BST</small><br><strong>Scheduled</strong> - We will be performing some planned maintenance on the dr2cloud platform, during this time users may experience intermittent access to the affected platform.<br />Note. Replication will continue even if the Cloud ZVM is showing as disconnected</p>tag:status.assurestor.com,2005:Incident/171516032023-05-05T16:00:22+01:002023-05-05T16:00:22+01:00PLANNED MAINTENANCE - veeam2cloud<p><small>May <var data-var='date'> 5</var>, <var data-var='time'>16:00</var> BST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'> 5</var>, <var data-var='time'>13:00</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'> 5</var>, <var data-var='time'>09:13</var> BST</small><br><strong>Scheduled</strong> - We will be performing some minor planned maintenance on the veeam2cloud platform, during this time users may experience intermittent access to the affected platform.</p>tag:status.assurestor.com,2005:Incident/158841972023-01-23T16:00:41+00:002023-01-23T16:00:41+00:00PLANNED MAINTENANCE - dr2cloud Zerto<p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>16:00</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>10:01</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>16:34</var> GMT</small><br><strong>Scheduled</strong> - We will be performing some minor planned maintenance on the dr2cloud platform, during this time users may experience intermittent access to the affected platform.
<br />Note. Replication will continue even if the Cloud ZVM is showing as disconnected</p>tag:status.assurestor.com,2005:Incident/158841842023-01-23T16:00:40+00:002023-01-23T16:00:40+00:00PLANNED MAINTENANCE - veeam2cloud<p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>16:00</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>10:01</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>16:33</var> GMT</small><br><strong>Scheduled</strong> - We will be performing some minor planned maintenance on the veeam2cloud platform, during this time users may experience intermittent access to the affected platform.</p>tag:status.assurestor.com,2005:Incident/158369902023-01-18T16:01:34+00:002023-01-18T16:01:34+00:00PLANNED MAINTENANCE - veeam2cloud<p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>16:01</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>10:03</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>13</var>, <var data-var='time'>15:36</var> GMT</small><br><strong>Scheduled</strong> - We will be performing some minor planned maintenance on the veeam2cloud platform, during this time users may experience intermittent access to the affected platform.</p>tag:status.assurestor.com,2005:Incident/158369762023-01-18T16:01:34+00:002023-01-18T16:01:34+00:00PLANNED MAINTENANCE - dr2cloud Zerto<p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>16:01</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>10:03</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>13</var>, <var data-var='time'>15:35</var> GMT</small><br><strong>Scheduled</strong> - We will be performing some minor planned maintenance on the dr2cloud platform, during this time users may experience intermittent access to the affected platform.
<br />Note. Replication will continue even if the Cloud ZVM is showing as disconnected</p>tag:status.assurestor.com,2005:Incident/134137262022-11-20T14:47:56+00:002022-11-20T14:47:56+00:00DC2 offline<p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>14:47</var> GMT</small><br><strong>Resolved</strong> - Platforms have returned back to BAU</p><p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>14:16</var> GMT</small><br><strong>Update</strong> - We have resolved the root cause of the DC2 sit outage and platforms are now coming back online. We will monitor the site for the next few hours before closing this incident</p><p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>14:13</var> GMT</small><br><strong>Monitoring</strong> - We have now resolved the root cause of the DC2 site outage and platforms are now coming back online, we will continue to monitor this over the next few hours before closing this incident</p><p><small>Nov <var data-var='date'>19</var>, <var data-var='time'>22:09</var> GMT</small><br><strong>Identified</strong> - After some planned works at the DC on Saturday morning on the A power feeds, we have unfortunately had an issue with the B feed PDU tripping multiple time due to a faulty PSU. Replacements are being shipped to the DC tomorrow so we can swap out the faulty unit/s and bring all platforms back online. Target for recovery of all services is for Sunday 20th by 3pm</p><p><small>Nov <var data-var='date'>19</var>, <var data-var='time'>11:00</var> GMT</small><br><strong>Investigating</strong> - We have noted DC2 being offline and are currently investigating</p>tag:status.assurestor.com,2005:Incident/125888042022-10-26T16:00:32+01:002022-10-26T16:00:32+01:00PLANNED MAINTENANCE - Firewall Updates<p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>16:00</var> BST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>14:00</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>11:41</var> BST</small><br><strong>Scheduled</strong> - We will be performing some firewall updates during this planned maintenance window. Client's may see a brief interruption in access to the affected platforms.</p>tag:status.assurestor.com,2005:Incident/123079782022-10-21T18:02:43+01:002022-10-21T18:02:43+01:00PLANNED MAINTENANCE - dr2cloud Zerto<p><small>Oct <var data-var='date'>21</var>, <var data-var='time'>18:02</var> BST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'>21</var>, <var data-var='time'>16:01</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>14:37</var> BST</small><br><strong>Scheduled</strong> - We will be performing some minor planned maintenance on the dr2cloud platform storage nodes, during this time users may experience intermittent access to the affected platform.<br />Note. Replication will continue even if the Cloud ZVM is showing as disconnected</p>tag:status.assurestor.com,2005:Incident/111173112022-09-13T12:01:43+01:002022-09-13T12:01:43+01:00Partial backup2cloud outage<p><small>Sep <var data-var='date'>13</var>, <var data-var='time'>12:01</var> BST</small><br><strong>Resolved</strong> - Everything has been updated and the platform is operational again.</p><p><small>Sep <var data-var='date'>13</var>, <var data-var='time'>10:31</var> BST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.assurestor.com,2005:Incident/106413652022-07-22T19:33:49+01:002022-07-22T19:33:49+01:00PLANNED MAINTENANCE - dr2cloud DC1<p><small>Jul <var data-var='date'>22</var>, <var data-var='time'>19:33</var> BST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'>22</var>, <var data-var='time'>19:01</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>19:17</var> BST</small><br><strong>Scheduled</strong> - We will be performing some planned maintenance on the dr2cloud DC1 platform, to upgrade some firmware components that have been identified as a risk to the platforms performance. During this period end-users may see reports of disconnections to the platform and see RPO values increase.</p>tag:status.assurestor.com,2005:Incident/106606082022-07-21T14:12:37+01:002022-07-21T14:12:37+01:00dr2cloud DC1 ZVM offline<p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>14:12</var> BST</small><br><strong>Resolved</strong> - This incident has now been resolved, all VPGs are syncing again as expected.</p><p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>13:25</var> BST</small><br><strong>Monitoring</strong> - ZVM recovered from back up and has booted successfully, we will now monitor that all VPGs come back in sync as expected.</p><p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>13:06</var> BST</small><br><strong>Update</strong> - ZVM OS not booting as expected since blue screen, we will now roll back to a previous backup to confirm if this resolves the boot issue</p><p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>12:52</var> BST</small><br><strong>Update</strong> - ZVM windows OS has blue screened, we are currently looking to hard restart the OS and verify the server boots as normal</p><p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>12:48</var> BST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>12:48</var> BST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.assurestor.com,2005:Incident/106522802022-07-21T12:58:49+01:002022-07-21T12:58:49+01:00my2cloud & Hybrid Partner API missing Zerto data<p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>12:58</var> BST</small><br><strong>Resolved</strong> - Zerto have now resolved their issue with their analytics platform. <br /><br />my2cloud and Hybrid partner API working as expected again.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>14:29</var> BST</small><br><strong>Investigating</strong> - Zerto Analytics is experiencing a technical issue, meaning the my2cloud portal and Hybrid API are unable to collect Zerto data.<br /><br />Zerto are working on the fix for this at present.</p>tag:status.assurestor.com,2005:Incident/106235872022-07-20T07:55:57+01:002022-07-20T07:55:57+01:00VPGs falling out of sync<p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>07:55</var> BST</small><br><strong>Resolved</strong> - VPGs have remained syncing as usual with RPOs back to their normal values with the expected bitmap syncs</p><p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>15:22</var> BST</small><br><strong>Monitoring</strong> - VPGs are now syncing as expected, we will continue to monitor the platform closely for the next 24 hours to ensure stability is good.</p><p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>14:41</var> BST</small><br><strong>Identified</strong> - We believe the fault has been identified as an out of date firmware driver in the storage path that has now been upgraded during todays planned maintenance event. We are seeing VPGs come back in sync.</p><p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>08:41</var> BST</small><br><strong>Update</strong> - We are continuing to investigate this issue, at present only a minority of VPGs are impacted</p><p><small>Jul <var data-var='date'>18</var>, <var data-var='time'>16:52</var> BST</small><br><strong>Update</strong> - We are continuing to investigate this issue, initial log analysis via Zerto point s to potential resource exhaustion at multiple client sites, we will continue to monitor this and may reach out to individual clients to discuss increasing available RAM to source VRAs</p><p><small>Jul <var data-var='date'>18</var>, <var data-var='time'>12:19</var> BST</small><br><strong>Update</strong> - We are continuing to investigate this issue with our vendor (Zerto)</p><p><small>Jul <var data-var='date'>18</var>, <var data-var='time'>12:01</var> BST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>