Last night, we upgraded the journal websites to Topaz 0.9 rc1 [2] (rc1 because this is a “beta” 0.9 release). The development for this release focused on performance and stability - specifically to alleviate the sluggish speed of the websites and the pain of ingests.

We're going to take down the journal websites for a few hours tonight starting at 7pm PST. Russ [5] is going to upgrade to Topaz 0.9 (rc1) on the production servers while Josh [6] replaces/rebuilds the dead drive on the Mulgara server.

We experienced a hardware malfunction yesterday that caused the TOPAZ [9] hosted journals to be offline from 4pm - 10pm PST.

James [10] sent an email late afternoon yesterday indicating site errors on the PLoS journal websites. Soon after his email, the IT team started receiving SMS alerts. I assumed that something had occurred with the Topaz framework and started looking at the appropriate log files but couldn't find anything. I spent the requisite amount of time banging my head against the "site error" wall without success and I called up Russ [11] for assistance. After a bit of digging through the server logs, he found the culprit - a drive had failed on the Mulgara [12] server. This drive is part of a RAID 0 configuration, so we didn't lose any data but we also mysteriously lost the connection from the Mulgara server to the DAS array [13] (disk storage for the Mulgara data).

Performance of the websites hosted on Topaz [16] has increased over the last two weeks with a variety of patches ported to the production servers. We still have an outstanding memory problem that requires a restart of the Topaz applications three times a day (these restarts usually occur around midnight, 8am and 4pm with a duration of less than 10 minutes). I feel that we're close to diagnosing the memory problem which is the last performance hurdle.

We’re continuing to experience slowness and intermittent downtime for the websites hosted on Topaz [19]. At this time, the priorities for the IT and Topaz teams are to improve stability of the Topaz applications and increase the performance of the web sites.