Our connection to Savvis in New York City took some errors at 9 a.m. this morning, and was down for less than a minute. Then at around 9:30 a.m., Savvis experienced a routing event in New York City that reset our BGP session with them. The session was not restored fully until 8 minutes, 30 seconds later. Our redundant connectivity (4 separate networks) kept traffic flowing, with few hiccups.
Update: The problem was traced to human error. A routing engineer working on the NYC node on another customer's problem accidentally cleared all BGP sessions on that router, not just the session he was working on. This affected a lot of customers. Savvis apologizes for the error.
Our USENET news provider experienced a system failure in one of their servers. Customers using news-1.dialmaine.com for newsfeeds may experience delays in receiving articles or connection failures. None of the other news server were affected by this outage.
DialMaine will be performing system maintenance on it's electrical systems this evening resulting in brief outages between midnight and 6:00 a.m. We will working on systems affecting e-mail, Web hosting and core router. Outages should be brief and should only be for a few minutes at a time.
Core-w-1 experienced some sort of software crash and reload at around 15:40:11, and was running again and routing traffic at 15:42:38. Core-w-2 was not affected. Further investigation is needed.
Our connection to Boston with Savvis went down for a period of 10 minutes. Carrier has been contacted, and Sprint is going to do an intrusive test on the circuit until 4 a.m. to try to determine the cause. This outage did not have a major impact on DialMaine customers, due to our redundant backbond connections.
DialMaine is currently experiencing some slow network performance, due to two separate circuit outages. Our Savvis connections to New York City and Boston went down approximately 9:23 a.m. The carrier has been notified, tickets created, and we are awaiting an update. Both of these circuits ride Sprint's network, and we believe Sprint has experienced a circuit outage into Maine. More information as we have it.
Update: Sprint couldn't loop back to our premises, and referred back out to Verizon. They believe they isolated it to a CO and are dispatching a tech, as of about 11:40.
Update: Savvis called us to report Verizon found something they fixed: "Verizon Replaced the SONET software on the Virtual DAC's inside the CO." The T1's are currently up and working, and came back online about 1:13 and 1:20.
There will be brief (90 second) service outage at 5 a.m. on 7/31/03 due to a software upgrade on our core router in Waterville. Additional, customers using our new 500-699-1010 access number will also experience a 5 minute outage at the same time for a memory upgrade in another router.
We will be performing routing change updates during the July 24th maintenance window. This may result in occasional periods of slow connectivity or loss of connectivity. The periods of low connectivity should be very small, in the range of 1–5 minutes. We'll be changing a few pieces of cabling as well as massaging the routing tables a bit more. We're adding significant bandwidth to the network during this outage, so overall speed to the Internet should increase. We'll post more as we complete the tasks. Maintenance window for this update is midnight to 6 a.m.
We will be performing system wide network maintenance tonight in response to a recently announced vulnerability in some of our router software. It does sound like we're vulnerable right now, but since no exploits are currently known, we're being proactive in patching the hole now before the exploits are available. If you know a Network Administrator in your life, chances are, he or she should be doing the same thing with their Cisco hardware tonight too! We will announce here when we have completed the upgrades. This will result in loss of connectivity to the Internet for all customers for a period of 5–10 minutes. Since we have multiple routers, this may mean multiple periods of disconnection. The upgrades will take place between 2 and 6 a.m. this coming morning.
Update (7/22/03): Those updates went through without a problem. Downtime was minimal.
Last nights (or last mornings) scheduled upgrade, noted below, did not occur until this morning. Intermittent outages occurred in authentication between about 4:45 and 5:15 a.m. We apologize for any inconvenience this may have caused. You'll never really see what these changes acommplished for you at the other end, but several people in the engineering group are now breathing a sigh of relief now that the changes are made. Please feel free to contact technical support if you are having any trouble connecting.
We will be using the maintenance window of Midnight to 6:00 a.m., June 25th, to work on our Authentication servers. We'll be updating some software which will cause intermittent outages to authentication. This means that you won't be affected if you're already logged in, but might be unable to log in at times during this maintenance window. There may also be a router OS upgrade, which would mean a quick reboot of the route which handles all of the dial-up access. If this happens the outage should be less than 10 minutes. We will update as we work and will update to let people know that we have finished.
Verizon Frame Relay backhaul network experiencing intermittent problems between 4:04 a.m. and 6 a.m., totaling around 25 minutes of downtime during this 2 hour period. VADI contacted (after waiting 25 minutes on hold!) at around 5:20 a.m. VADI said they would "put it in for testing". Circuit has been 100% down since 6:08 a.m., due to testing. VADI contact again at 6:22 a.m., no updates on their end, still waiting for the test results.
Update (9:30 a.m. rto): Since the network has been down for more than two hours, all customers will receive a one-day credit as per standard policy.
Update (9:45 a.m. rto): The network is up and functioning.
Update (10:21 a.m. jak): Issue turned over to John, jlt to other issues. The network connection came back up at May 26 9:10:20.413. The problem has not been 'returned' back to Advanced Data yet, so there's no official cause. Officially, the circuit is not back according to Verizon Advanced Data. It's not back until the call returns from the CO. However, we are passing traffic and so far things look good. We're awaiting the 'official' word from them and are requesting a "Root Cause Analysis" as part of the outage.
Update, 11:30 a.m. (jak): Received a call from Advanced Data that essentially said "It looks like it came back on it's own". They will continue to monitor the circuit for the next 24 hours. The ticket remains open.
The Waterville area experienced a power outage between 6:15 p.m. and 7:30 p.m., but our systems were not affected. Our generator and UPS systems kept us online.
Just a quick, belated note to indicate that the problems experienced on January 23, 2003 were repaired, and that we have had no significant network problems to report since then.
The router that serves as our primary dial-up gateway crashed and was unable to reboot properly. We are investigating the cause of the crash. Dial-up access, DNS and authentication were affected. No incoming e-mail from the outside world was lost, as those systems are on a different segment of our network. We apologize for the outage. A more detailed explanation will follow tomorrow after we have investigated the cause of the router crash. If you still experience problems, please contact us at support@dialmaine.com or 1-888-734-3410.
The primary news server, news.dialmaine.com, is unreachable. We have changed where 'news.dialmaine.com' points to make it point to one of the other news servers we use. This change should take effect over the next 4 hours (to wait for DNS propagation). DialMaine users can also use news1.dialmaine.com, news2.dialmaine.com (longer retention) and news3.dialmaine.com (binaries only) for Usenet news service. To find out more about Usenet, please see the FAQ for it, entitled What is Usenet? We are in contact with our Usenet provider to resolve the issue.
Customers who use DialMaine from any of the Northland/Utilities Inc. telephone company properties are experiencing problems connecting. This appears to be an issue with Northland/UINC and not DialMaine. We have been trying to contact some of our customers, but cannot get through on the telephone. If you are dialing in from this area and cannot connect, please attempt to dial our number from your telephone keypad and make sure you can get through that way before calling technical support. Feel free to contact us if you have other problems with our service though!
Update (02:20 p.m.): We have been in contact with Northland telephone and they confirm a problem exists in their network. They are working to try and make calls to DialMaine go through more often.
