Since this morning's events, the sites have been performing great. Russ applied patches to resolve this morning's XSL issue and to disable an article cache flush. We realized that all cache keys for images were being removed if a user added a note or rating to an article. These actions shouldn't affect the static article images, so the flush of this cache was disabled. There are now over 10k images in this cache (affectionately know as SPCF), which will improve performance.

After enabling detailed logging of queries to Mulgara, we haven't had a query hang retrieving data (go figure). I've found a few queries that are taking a long time to retrieve data and will continue digging through the logs. We ingested articles throughout the afternoon without any disturbance to the Topaz force. We'll (hopefully) cut a new release next week with a more updates to cache and we will re-enable search.

This has been a painful week for PLoS, authors, the communities of PLoS Computational Biology, PLoS Genetics and PLoS Pathogens, and users of the sites. We've slept little but have learned a LOT in the past week. We're closing in on the last bit of issues and should have a much more stable platform soon.