E-mail system breaks on undergraduates

Undergraduates have barely had access to their email since Saturday after OIT ran into trouble after upgrading the University's messaging software.

Normal operations resumed this afternoon, but technology officials warned that the University's IMAP and Webmail servers might continue to experience problems.

"We believe there is still some instability occurring on the undergraduate email server," said Donna Tatro, manager of OIT's collaboration services group.

OIT had announced a total shutdown of the IMAP system for most of Saturday to upgrade the Sun Microsystems Corp.-provided server software to the newest version, Sun ONE Messaging Server, on faster computers. During that period, University email users were not able to read or send email through programs such as Outlook and Eudora or through the web-based alternative, Webmail.

But unexpected problems occurred after the upgrade to the server that handles undergraduate email accounts.

Since Sunday night, students have had difficult sending messages, receiving attachments and writing emails.

In particular, a corrupt system configuration file led to the unexpected outage on Sunday evening, Tatro said. The file was fixed, but Sun and the University are still searching for the underlying cause, she said.

"I don't think we are out of the woods yet because we've asked Sun why this is occurring and have yet to receive an explanation from them," she said. "It is not uncommon for databases to be tuned and tweaked and certain elements in them to be reconstructed."

This morning, OIT discovered that the system was holding mail sent to undergraduates.

"Once again, we found that the corrupt configuration file we initially rebuilt still needed repair," Tatro explained. "We got back on the phone with the Sun engineers to resolve the new problem."

OIT got the first hint the upgraded system was not working as planned around 8:30 p.m. Sunday. The Help Desk reported students having difficulty logging into and using Webmail.

"As time progressed, the intermittent problems became worse and worse," Tatro said.

Get the best of the ‘Prince’ delivered straight to your inbox. Subscribe now »

OIT and Sun Microsystems officials began discussing the problem, but by 10:30 p.m., the messaging server collapsed and email service was unavailable.

Through Monday afternoon, OIT technicians and Tatro's group, which manages the infrastructure of the University's UNIX and Windows servers, worked with Sun to solve the problem.

OIT exchanged various configuration and log files with Sun to target the source of the problems.

"OIT technicians thought they had identified the problem and were reconstructing everyone's mailboxes," said Rita Saltz, senior policy adviser to OIT. "But the reconstruction halted about midway on the account of one particular student."

"They finally got a reconstruction that went beyond that point, but the problems were still not resolved," she said.

Remaining questions

The University is still waiting for answers.

Tatro said Sun has not provided OIT with an explanation of how the specific configuration file became corrupted.

"It is frustrating as IT [Information Technology] workers to not know the source of the problem. Computers don't just do unexpected things like this," Tatro said.

OIT's support director Steven Sather informed the undergraduate body of the continuing "sluggishness" of the email servers in an email Tuesday afternoon.

While Tatro could not ascertain if the email problems were a result of Saturday's upgrade, Sather suggested they were in his statement.

"We have encountered several problems with the new version of the server software. The most serious problem relates to email message delivery to inboxes on the undergraduate student mail server," Sather wrote.

On Saturday, the University's almost four-year-old hardware was replaced by hardware with more than double the capabilities.

OIT chose Saturday for the server upgrade to minimize the impact of the scheduled outage on student's work, since classes ended Friday for Fall Break.

Tatro could not rule out future outages as software has a tendency to have bugs and careless coding even within the most stable programs, she said.

"Looking at our inbox and sent mail, everything seems like such a simple and wonderful system, but under the covers there is quite a lot going on managing many mailboxes," Tatro said. "Vendors strive to provide extremely reliable and robust software systems and we strive to, in turn, administer those systems and provide good service."

Status of the current problems can be found at helpdesk.princeton.edu.

OIT has said that any additional problems should be reported to the Help Desk at 609-258-HELP.