We decided to use another method for making mail work better. Instead of swapping the failing hard drive on the old server we put the mail services on our cluster of servers. We have been planning to do this for some time now, but we didn't know the programming group was done with their work on it. We have changed the mail services to go to a new group of servers, so that in case of other outages, we'll have other servers online to pick up the slack. We now have two incoming and two outgoing mail servers in place of the one single mail server we had for both tasks previously. We've also make some updates to our IMAP server that runs our webmail interface - so you should see some vast improvements there.
The update took a little bit longer than expected, lapsing into the 6 a.m. hour and for some aspects even the 7 a.m. hour. We're finalizing the location of our logs for the new mail cluster, but that's only an behind the scenes issue.
Thank you for your patience as we update the mail service to help serve you better.
We experienced an outage of our mail server which resulted in customers not being able to send or receive e-mail. The outage was for two hours and is the result of a hardware failure. While the problem is fixed for now, we consider it a band-aid (tm) fix which we will put a better fix on later tonight during a maintenance window.
Did we ever explain maintenance windows? At DialMaine, we have to do maintenance on the servers, routers, hardware and software that make the internet work. Often these are security updates, software updates, hardware fixes and upgrades. It means that there will be downtime where certain services may not be available. Instead of doing this during normal working hours or during normal internet usage hours, we wait until "0-dark-thirty" when MOST of you are sleeping. We do this to interrupt as few people as possible. We call this time a "maintenance window." For most fixes, the maintenance window is 2:00 am to 5:00 am. For some other simple fixes that would be less disruptive, we might start as early as midnight. So, that's a maintenance window, and that's when they happen. So if you see us some morning and we look a little glassy-eyed when we're getting breakfast, perhaps we were up all night for a maintenance window. Have a little pity on us ;)
Our upstream provider (Arcus Digital) experienced another core router problem that caused a 100% loss of Internet connectivity for DialMaine customers for approximately 2 minutes. We apologize for the outage.
Verizon experienced an outage on the circuit that backhauls your calls to our data center. While we were waiting for Verizon to get back to us, the circuit came back up and they called saying it had fixed itself. We've requested further investigation of the problem so we can find the cause. PMFI isn't good enough. Apologies for this early morning outage.
Our upstream provider (Arcus Digital) experienced a core router problem that caused a 100% loss of Internet connectivity for DialMaine customers for approximately 2 minutes. The core router problem has been fixed. We apologize for the outage.
DialMaine will be doing routine maintenance this evening. This will result in momentary outages of our circuits to the Internet, but due to the redundancy built into the network, you should not notice the interruption. Also, we will be rebooting the shell server, so shell access will be unavailable for a short period of time. Verizon is also planning to move a circuit for us at 5:00 a.m.—again, our redundant connections should make that change go unnoticed.
We intend to have all our work done by 5:00 am in time for the Verizon change, and then have Verizon work complete by 6:00 am. All work should be complete by 6:00 am. Thank you for your patience while we improve our systems to serve you better.
Update! (jak 10/31/02, 06:01 a.m.): We did several pieces of maintenance but some were not complete. We planned for more than we could accomplish in the maintenance window. We still have some re-cabling to do on the Waterville dialup facility and the Frame Relay that handles all the calls for the rest of the state. We will plan another late night (erm, early morning rather) maintenance window for this.
Completed items include the shell.dialmaine.com server, rewiring of two of our T1's to the Internet, facilities improvements and several things that run in the background in our Network Operations Center.
It's 6:00 am, so all work on cabling is concluded, we won't start anything else after 6:00 am. We apologize for any inconvenience this work may have caused.
Verizon reported an outage of a piece of hardware in the Augusta CO that made our dialup number unavailable from 11:40 to 2:07 this evening/morning. The hardware has been reset and all tests indicate that service has been restored. This would have affected users in the Greater Augusta Area (those whose calls are routed through Augusta's facility) only. If you find that you still have troubles, please feel free to contact technical support at 888-734-3410 or send an e-mail to support@dialmaine.com. We apologize for any inconvenience this may cause.
Verizon is experiencing another difficulty with, what we think is, the same high speed OC-3 link, though they have not yet confirmed it. DialMaine customers calling in from the areas north of Portland area are experiencing massive packet loss. We have opened a ticket with Verizon and are on the phone with them presently. We hope to resolve this, once and for all, as soon as possible.
Customers calling from the greater Portland area are NOT affected by this problem, nor are customers calling our local Waterville access number.
Update (jak 20:10): Verizon has isolated the problem to a point between Portland and Waterville. They're checking a few more things and will call me back shortly.
Update (jak 23:20): We've received response again from Verizon and we've also seen the problem go away. We know the problem only occurs when we are at peak usage (high dialup capacity) and goes away when it's less busy. We're scheduling some work with Verizon tomorrow. More as we have it.
Verizon is currently experiencing a problem with a high speed OC-3 link in the Portland central office. This OC3 problem is affecting the Lewiston, Ellsworth, and Augusta PRI hubs. Customers may experience slow connections, and high (20%) packet loss intermittently. This OC3 problem affects all Verizon IPRS customers in Maine. Verizon is awaiting replacement parts to effect a repair in the next 48 hours.
Customers calling from the greater Portland area are NOT affected by this problem, nor are customers calling our local Waterville access number.
Update (August 29th): This issue has abated somewhat and now only occurs during the evening between approximately 7pm and midnight. Verizon continues to work on the issue.
Update (August 30th): Verizon has scheduled replacement of equipment to happen at 4 a.m. on September 2nd.
Update (September 4th): Replacement of equipment at Verizon CO in Portland did not happen as scheduled. We are awaiting an answer as to why. Verizon re-confirms that they see the same problem that we are reporting. Intrusive circuit testing scheduled for 4am on 9/5/02.
Update (September 5th, 5:30 a.m.): Verizon/VADI performed extensive testing between 4:00am and 5:02am. IROC techs have not yet received a testing report back from VADI. IROC will get from VADI and call back in 15 minutes.
Update (September 5th, 6:15 a.m.): Verizon IROC tech calls back. Tests were run for 30 minutes to the "smart jack" and 30 minutes to our equipment, both were 100% clean and no errors were found. Nobody is surprised by this. Tech will investigate more, and hand off to day shift at 8 a.m. for some real attention.
One of our circuits for the dial-up pool has been going up and up down since 7:15 a.m. today. We are with Verizon to try to resolve this issue. This is a Verizon problem and they are currently testing the circuit. More details as we get them.
Verizon has picked up the ticket in the local CO (Waterville) and is working on it from there. Ticket was picked up at 10:51.
We escalated this issue with Verizon several times, finally getting the test center to look at the issue again around 1:45 p.m. Circuit back up and operational at 2:15:53 p.m.
The server responsible for ftp.dialmaine.com is back online again. A hard drive failure caused the outage and it took a while to get the replacement drive online. We have put the system back online and things look normal. We apologize for any inconvenience this may have caused.
The server responsible for ftp.dialmaine.com had to be taken offline today due to hardware problems. We are in the process of getting this back up and running and should have access restored shortly. We apologize to all of our customers who are experiencing FTP problems during this downtime.
The www.dialmaine.com website was unavailable due to a database server crash. This outage did NOT affect web browsing, email, or any other DialMaine related service.
Verizon was experiencing difficulties forwarding calls to our 500 number. This occurred between 9am and 3pm today, but has been resolved.
A Verizon circuit cut-over for one of our backhaul circuits for the dialup pool didn't go as smoothly as we had hoped. This caused session disconnections and problems connecting during this 14 minute period. All fixed now. The Verizon tech has been given more coffee.
We will be performing emergency maintenance on one of our disk arrays, due to a multiple drive failure. This will cause a short outage for webmail, and the DialMaine Web site itself. This will NOT affect web browsing, dial-up service, or normal e-mail services.
Update (6/7/2002): This issue now resolved.
Minor operator glitch on primary DNS server for about a 70 second period. During this 70 seconds, browsing of new web pages would not have worked.
Service during April 28th maintenance window: Tomorrow morning between 2 and 4 a.m. we will be replacing a disk array power supply. This will cause an outage for webmail and shell services for DialMaine customers. This will also cause outages for several commercial web pages. The outage should be limited to about 1/2 hour–1 hour, but we maintain the entire window for fixing this.
Due to a vehicle collision with a telephone pole last night, much of the Waterville area was without power from approximately 6pm to 9pm. This included the DialMaine NOC in downtown Waterville.
During this power outage our technical crew worked very hard to keep services up and running, including bringing a portable generator on-site, and shutting down non-essential equipment to conserve backup capacity. Our facility has an emergency backup generator, but it failed to operate when we needed it.
Unfortunately, DialMaine subscribers did experience intermittent loss of Internet service during this time.
All DialMaine subscribers will be issued a full day's outage credit on your next billing cycle.
We apologize for the outage, and we are taking the necessary steps to make sure that this sort of outage does not occur again.
Service announcement: This weekend during the maintenance window beginning Sunday April 14th, 2:00 a.m., we will be performing system maintenance on the mail server. During this period, the sending and receiving of e-mail may be affected. This will not result in lost e-mail should someone be trying to e-mail you, but you might not be able to check mail during this time. We expect the outage to be less than one hour, but may need additional time for testing all the functionality of the new server. All work will be completed by 6:00 am Sunday morning.
Our Verizon Frame Relay circuit is currently down. We are in contact with Verizon to get this resolved.
Our circuit came back up at 9:41 a.m. We were down for a total of 20 minutes.
Verizon has a circuit down which results in our dialup being down. At this time, they've isolated the outage to their Portland facility and have dispatched a technician. We are waiting for them to finish solving the problem and bring the circuit back up.
Update (11:51 a.m.): Verizon is continuing to work on the problem and thinks it should be done soon, but is unwilling to commit to an ETA. We're staying on them.
Update (12:20 p.m.): The circuit finally came back up. Verizon is attributing the outage to a bad coaxial connector to a DSX3 in a Verizon facility. This was an 11 and a half hour outage. We're completely unhappy with Verizon on this and will be taking further action. We have deliberately left the ticket open with Verizon until we can find out more information on their response.
We apologize to you for this outage and while it's directly attributable to Verizon, we still don't like the fact that we had such an outage.
Service from Verizon has been restored. They told us there was an inter-office problem between the Waterville and Augusta COs with an OC-12 connection, and this is what took us out of service. Total outage was 5 hours, 14 minutes. Our first outage.... sigh.
We're experiencing an outage in the circuit that brings all the dialup calls into our facility. This means that dialup users cannot get online, although users will hear modem sounds and negotiation. You'll likely get an error of authentication of '691' in Windows. We've been in contact with Verizon to remedy the issue and are working on it as efficiently as we can. I have no current ETA, but I'll update as I can. -jak
We accomplished a lot tonight and put several new UPS' online. A small configuration problem moved our planned changes for one of our T1 circuits to later this week. We have completed our work for the morning. If you feel that something is broken, please feel free to call into technical support.
One of our upstream providers has announced system maintenance for the following. Due to our redundant connections, we do not anticipate any appreciable downtime.
This Sunday (3/17/02) morning in the maintenance window (2 a.m. to 6 a.m.) we will be performing system upgrades to the following:
