Emergency Server Reboot: Node01 (Resolved)
  • Priority - Critical
  • Affecting Server - NODE01.MCO.NODESPACE.US
  • Recently, a severe bug was discovered in the email software installed on one of our web servers which is hosting your website (on NODE01). In order to resolve this issue, we have patched the software which will only take effect once we reboot the server. We have scheduled emergency maintenance tonight to reboot the server.

    Please note: if you have other services with us such as dedicated or virtual servers, these are not affected.

    Full details about the bug: https://nvd.nist.gov/vuln/detail/CVE-2019-10149


    Thank you for your understanding and we apologize about the inconvenience.

    UPDATES:

    1. Server has been rebooted.

  • Date - 2019-06-06 23:20 - 2019-06-06 18:13
  • Last Updated - 2019-06-06 18:13
PHP 7.2 Default (Resolved)
  • Priority - Medium
  • Affecting Server - NODE01.MCO.NODESPACE.US
  • As part of our commitment to provide the best possible infrastructure for your services, we're sending you this notice to inform you of some upcoming server maintenance in our Orlando Data Center. Please read the following details carefully:

    Maintenance type:
    Software Upgrade

    Procedure:
    PHP 5.6 is being officially deprecated as it will no longer receive security updates after January 19th. We will be setting the default PHP version on node01.mco.nodespace.us (the server your website is hosted on) to PHP 7.2.

    Latest versions of most web software (such as WordPress, phpBB, MyBB, etc.) already has PHP 7.2 support, however we do recommend testing plugin and addon compatibility. 

    If you are running any custom software, please use this time to test for PHP 7.2 compatibility. We've made it easy to switch your site to PHP 7.2 for testing. Please see the following support article on how to enable PHP 7.2 for your site (and how to switch it back): https://my.gonodespace.com/knowledgebase.php?id=174. If everything works as expected, you can reset your site to "inherited". If you encounter errors, we recommend setting the PHP option to version 5.6.

    This email is regarding your hosting service with us as we are making a major change in January. As this is a major change, we are sending you this notification as a courtesy so you have time to prepare your website.

    NodeSpace does not offer any sort of coding assistance, however our support team is more than happy to assist you with collecting any errors on your site. Please open a ticket through the My NodeSpace portal (https://my.gonodespace.com/submitticket.php) and we'll assist you as much as possible.

    Maintenance Window:
    Start - 01/19/2019 - 10:00 PM EST
    End - 01/19/2019 - 11:59 PM EST

    Estimated downtime:
    There will be no downtime associated with this maintenance.

    Thank you for taking the time to review this maintenance notice. If you do have any questions or concerns about this procedure and how you may be affected, please don't hesitate to contact our support team for more details.

  • Date - 2019-01-19 00:00 - 2019-01-19 00:05
  • Last Updated - 2019-01-21 23:50
Emergency Data Center Maintenance (Resolved)
  • Priority - High
  • Affecting Other - Orlando2 Data Center
  • Initial Report

    As part of our commitment to provide the best infrastructure for your services, we're sending you this notice to inform you of emergency maintenance taking place in our Orlando based data center facility. We recommend reviewing the following details:

    Maintenance type:
    Electrical Infrastructure

    Reason for Maintenance:
    Our monitoring systems alerted our data center technicians to potential issues with electrical systems powering a portion of our Orlando data center facility. In order to proactively address the problems our team of engineers will be running tests to locate any potential flaws and replace any hardware found to be faulty.

    Maintenance Window:
    Current - ONGOING

    Datacenter:
    Orlando.DC2

    Estimated amount of downtime:
    We understand the importance in minimizing downtime while performing any type of maintenance and for that reason we will do everything we can to expedite this process. There are no guarantees, and while we have not encountered any actual power issues affecting your services, it is worth noting that due to the nature of the issue(s) at hand we are considering this an At-Risk maintenance and hence are alerting you to the incident. We do not expect to see any type of downtime but should issues arise power to various sectors of our facility may be affected. We will be sending out a follow up to this notice once the maintenance has been concluded.

    Thank you so much for taking the time to review this notice and we do apologize for any inconvenience these unforeseen issues may pose to you and your clients. If you have any questions or concerns about this reboot procedure, please don't hesitate to contact our staff for more details.

    Update 1 - 9:51 PM

    We thank you very much for your extended patience and simply wanted to provide an update to our prior notice regarding emergency maintenance in our Orlando data center resulting in us placing it into maintenance mode.

    At this time our engineers continue to monitor the status of our electrical systems and will continue to do so until our facility is out of maintenance mode. We were able to limit down the factors that triggered our monitoring systems and steps are already being taken to resolve those issues. We remain cautiously optimistic that we will finalize the maintenance with no impact to your services and are aiming to have everything completed by tomorrow afternoon.

    As promised, a recap of the situation will be made available after the data center is out of maintenance mode. Lastly, in case an emergency should occur, we do urge you to follow our social media accounts, which are listed below, for updates:

    Twitter: https://twitter.com/nodespace

    Thank you once again for your patience and understanding during this time and we hope you have a great evening.

    Resolution - 11/30 7:49 PM

    First and foremost thank you very much for your extended patience and understanding while our team worked on the emergency maintenance over the last day and a half.

    At this time we are happy to announce that our team has successfully concluded the maintenance with no reports of downtime and/or power issues and everything is functioning in an optimal state.

    In order to be transparent, and as promised, we have included details provided by our engineers as to what prompted the at-risk emergency maintenance below.

    At around 11 AM EST on November 29th, our monitors alerted our team to issues with one of our UPS systems as it was failing and eventually did. We were successfully able to transfer the load on the UPS to maintenance bypass and onto full commercial power resulting in the *At-Risk* notice. Upon further investigation the cause of the failure was diagnosed and attributed to a failed capacitor in the system which was successfully replaced. Once that was done we were able to run tests on UPS that proved to be successful and the critical load was transferred back with no issue. After further monitoring everything continues to function as it should and we are concluding our emergency maintenance.

    We thank you, once again, for your patience and understanding and assure you that our goal is to always do what we can to ensure you get the services and support that you not only expect, but deserve. As always, if you have any further questions and/or concerns please do not hesitate to let us know. Have a great day!

  • Date - 2018-11-29 14:00 - 2018-11-30 19:49
  • Last Updated - 2018-11-30 22:11
Knowledgebase not showing all articles (Resolved)
  • Priority - Low
  • Affecting System - Knowledgebase
  • We have identified an issue with our client portal that is causing some knowledgebase articles to not show up. This issue has been reported to our software vendor for further investigation.

    Work Around: In the meantime, please check our Knowledgebase on our support portal site: https://support.gonodespace.com/knowledgebase
    Update#1: We have started updating the links to our support site. You can access our support site with your My NodeSpace login.

    Resolution: We have resolved the issue and knowledgebase articles are now showing in the customer portal.

  • Date - 2017-08-07 04:42 - 2018-05-30 23:10
  • Last Updated - 2018-05-30 23:10
Announcements not showing in Customer Portal (Resolved)
  • Priority - Low
  • Affecting System - Service Desk
  • We have identified an issue with our Service Desk software that isn't properly importing issues into our customer portal properly. Our software vendor is currently investigating this issue and will supply a patch when it is fixed.

    Work Around: To view all important customer announcements, view them within our support portal here: https://support.gonodespace.com/announcements

    Update#1: We have started updating the links to our support site. You can access our support site with your My NodeSpace login.

    Resolution: We have resolved the issue and announcements now show in the customer portal.

  • Date - 2017-05-01 00:00 - 2018-05-30 23:09
  • Last Updated - 2018-05-30 23:10
[RFO] NODE01.MCO down (Resolved)
  • Priority - Critical
  • Affecting Server - NODE01.MCO.NODESPACE.US
  • Reason for Outage (RFO)

    Location:
    NODE01.MCO.DC2

    Date:
    August 26, 2017

    Events:
    11:24 AM (GMT-4) During routine server maintenance, the NODE01.MCO server encountered an issue which caused Apache (HTTP) to stop responding to web requests. Our Linux Admins' attempts to bring Apache back online were unsuccessful so they issued a reboot of the server under emergency maintenance procedures.
    11:34 AM (GMT-4) The server still had not come back up from reboot. Our Linux Administration Team alerted our NOC team to check the physical server out.
    12:34 PM (GMT-4) Our NOC team reported that the server was fine hardware side and provided our Linux Team KVM access.
    1:07 PM (GMT-4) Our Linux Admin Team was able to restore services.

    Root cause of the issue:
    Our Linux Administration team noticed that during server boot up, several of the critical services were not starting. Upon investigation, the root cause was SELinux becoming enabled in the "permissive" state instead of "disabled.

    Resolution:
    Our Linux Administration team booted the affected server into single user mode and disabled SELinux and rebooted the server normally.

    Action plan:
    Our Linux Administration team will be closely monitoring the server and reviewing any updates that were automatically installed and working with the upstream software vendors to ensure that this does not happen again.

    Total downtime:
    Customers on NODE01.MCO experienced 1 hour, 43 minutes of downtime. This downtime is able to be claimed for an SLA credit.

    We sincerely apologize about the inconvenience and will do our best to reduce/eliminate future occurrences of the same problems.

  • Date - 2017-08-26 11:24 - 2017-08-26 13:07
  • Last Updated - 2017-08-26 13:55
Network Maintenance (Resolved)
  • Priority - High
  • Affecting Server - NODE01.MCO.NODESPACE.US
  • Maintenance Type:
    Network Infrastructure Maintenance

    Action to be taken:
    In an effort to maximize the reliability of our services, we will be replacing a line card in one of our core routers at our Orlando.DC2 datacenter.

    Maintenance Window:
    Start - Wednesday, August 2nd @ 23:00 (11:00 PM EDT)
    End - Thursday, August 3rd @ 01:00 (1:00 AM EDT)

    Datacenter Location:
    Orlando.DC2

    Estimated Downtime:
    We understand the importance of minimizing the downtime associated with any maintenance. We will take every reasonable precaution to ensure this process is completed safely and without downtime. Our fiber is redundant in our datacenter, so we do not expect any downtime will result from this maintenance. We are notifying you as a precaution. As traffic is re-routed throughout the network, you may experience up to 45 minutes of increased latency, minor packet loss, or possible service interruption during the two-hour maintenance window.

    We do apologize for any inconvenience and ask that you please plan accordingly.

  • Date - 2017-08-02 23:00 - 2017-08-03 01:00
  • Last Updated - 2017-08-06 19:24
Network Maintenance (Resolved)
  • Priority - High
  • Affecting Server - NODE01.MCO.NODESPACE.US
  • Maintenance Type:
    Network Maintenance

    Action to be taken:
    To ensure the reliability of our services, we will be replacing a line card in one of our core routers at our Orlando.DC2 datacenter.

    Maintenance Window:
    Start - Sunday, July 23rd @ 23:00 Local time (11:00 PM EDT)
    End - Monday, July 24th @ 01:00 Local time (1:00 AM EDT)

    Datacenter Location:
    Orlando.DC2

    Estimated Downtime:
    We understand the importance of minimizing the downtime associated with any maintenance. Our team will take every reasonable precaution to ensure this process is completed safely and without downtime. Our fiber is redundant in our datacenter, so we do not expect any downtime will result from this maintenance; however, we are notifying you as a precaution. As traffic is re-routed throughout the network, you may experience up to 45 minutes of increased latency, minor packet loss, or possible service interruption during the two-hour maintenance window.

    We do apologize for any inconvenience and ask that you please plan accordingly.

    Thank you so much for taking the time to review this notice. If you have questions or concerns about this procedure, please don't hesitate to contact us.

  • Date - 2017-07-23 23:00 - 2017-08-06 19:22
  • Last Updated - 2017-07-23 02:49
Brief Service Desk Outage from Customer Porta (Resolved)
  • Priority - Low
  • Affecting Other - Service Desk
  • Our service desk was temporarily unavailable from the My NodeSpace customer portal. Our technical team found the issue and resolved it promptly. We have manually opened any opened tickets that were misdirected. Any tickets that were opened via Facebook, email, or Twitter were all opned correctly and you should now see these tickets opened within your customer portal. If you have any questions, please don't hesitate to reach out.

  • Date - 2017-07-11 04:41 - 2017-07-12 00:42
  • Last Updated - 2017-07-12 04:46
Brief outage July 3 (Resolved)
  • Priority - Critical
  • Affecting Other - Network
  • We had a brief outage lasting approximately 1 minute during the early morning on July 3, 2017. Our data center identified an issue with one of the core routers providing network connectivity to the data center. The issue has been mitigated and will be resolved during network maintenance.

    If you have any questions, please do not hesitate to ask.

    Thank you for your patience and understanding!

  • Date - 2017-07-03 00:00 - 2017-07-03 00:00
  • Last Updated - 2017-07-05 04:18
Switch maintenance (Resolved)
  • Priority - Medium
  • Affecting Server - NODE01.MCO.NODESPACE.US
  • Scheduled Date and Time: 
    Start - December 7th, 2015 @ 8:00PM EST 
    End - December 8th, 2015 @ 12:00AM EST 

    Maintenance Window: 
    4 Hours 

    Datacenter Location: 
    Orlando.DC2 

    Maintenance Type: 
    In order to ensure the most secure and reliable services possible, we will be updating firmware on the switch your server is connected to. To complete this process, we will need to power cycle the switch. This will cause a brief period of service unavailability for all servers using this switch. 

    Estimated Downtime: 
    We understand the importance of minimizing downtime while performing maintenance and for this reason we will do everything we can in hopes that we can expedite the process. As it stands we expect no more than 20 minutes of downtime during the maintenance window as the switch is updated and power cycled. Should there be any complications, we will have personel on-site to address these issues immediately. 

    We do apologize for any inconvenience and ask that you please plan accordingly. 

    If you have any questions or concerns, please feel free to submit a support ticket. 

  • Date - 2015-12-07 20:00 - 2015-12-10 01:55
  • Last Updated - 2015-12-01 01:39

Powered by WHMCompleteSolution