Richard Cave's blog

Journal Websites - Topaz 0.9 rc1 Upgrade

Richard Cave — Thu, 17 Jul 2008 17:03:14 -0700

Last night, we upgraded the journal websites to Topaz 0.9 rc1 (rc1 because this is a “beta” 0.9 release). The development for this release focused on performance and stability - specifically to alleviate the sluggish speed of the websites and the pain of ingests.

The development included a major re-architecture of the publishing application and weeks of performance testing. There are a few minor issues but the sites are quite zippy and the new ingest times have, to quote James "exceeded expectations." We’ll probably have a few more quick restarts over the next few days as we shift through the logs/bugs but we won’t have to rebuild the article cache ever again (another pain point).

Thanks to Russ for the heroic migration day, Josh for the speedy drive rebuilds of the Mulgara server last night, and to the Topaz/PLoS developers that got this "beta" release out the door. We still have some cleanup after the dust settles, but I feel confident that the site performance/stability has greatly improved.

Trackback URL for this post:

http://www.plos.org/cms/trackback/381

Topaz 0.9 (rc1) - Site Maintenance Tonight on the PLoS Journals

Richard Cave — Wed, 16 Jul 2008 18:08:17 -0700

We're going to take down the journal websites for a few hours tonight starting at 7pm PST. Russ is going to upgrade to Topaz 0.9 (rc1) on the production servers while Josh replaces/rebuilds the dead drive on the Mulgara server.

Topaz 0.9 is primarily a performance release which will significantly enhance the speed of the journal websites. This release candidate was motivated by the performance problems uncovered with the CJ migration. There has been a re-architecture of the cache code and many OTM/Mulgara improvements by the Topaz developers. And they've upgraded many of the underlying applications. We've been performance testing this release candidate for the last two weeks and have seen significant improvements across the board from 0.8.

This release candidate isn't available to the public yet but will occur later this month after a bit more cleanup to the configuration files. The 0.9 release will also include the Ambra publishing application and two default "journals" - I'll have more information on Ambra closer to the release date.

Performance enhancements in Topaz 0.9

Pre-ingest script (process_sip) for images transforms.
Re-architected the cache code.
Moved transactions to read-only queries where possible.
Optimized slideshow retrieval/display.
Optimized many actions (e.g. FetchArticleAction, GetAverageRatingsAction, etc.).
Lots of Ambra code improvements.
Upgraded to Dojo 1.1.
Upgraded to Mulgara 1.2 (multi-core!).
Upgraded to Lucene 2.3.
Upgraded to ehCache 1.4.1.
Many Ambra configuration changes for ease of installation.

Features fixed in RC 0.9

Safara bugs fixed; ratings, drop-down menus.
Ingest fixes; ingest doesn’t change .mov to .qt; ingest handles .swf files.
Advanced search by past month & past three months working.
Search results with correct journal context.
XSL fixes.

Trackback URL for this post:

http://www.plos.org/cms/trackback/379

Sunday Outage Sunday

Richard Cave — Mon, 14 Jul 2008 15:38:01 -0700

We experienced a hardware malfunction yesterday that caused the TOPAZ hosted journals to be offline from 4pm - 10pm PST.

James sent an email late afternoon yesterday indicating site errors on the PLoS journal websites. Soon after his email, the IT team started receiving SMS alerts. I assumed that something had occurred with the Topaz framework and started looking at the appropriate log files but couldn't find anything. I spent the requisite amount of time banging my head against the "site error" wall without success and I called up Russ for assistance. After a bit of digging through the server logs, he found the culprit - a drive had failed on the Mulgara server. This drive is part of a RAID 5 configuration, so we didn't lose any data but we also mysteriously lost the connection from the Mulgara server to the DAS array (disk storage for the Mulgara data).

We restarted the server but couldn't confirm that it was rebuilding the RAID correctly. I drove down to the colo, confirmed the drive failure and babysat the server until the platform was healthy. We'll swap out the defective drive on Wednesday during the migration to a pre-release of Topaz 0.9.

In case you missed the reference to U2's Sunday Bloody Sunday....

Trackback URL for this post:

http://www.plos.org/cms/trackback/377

Update on Performance Issues of PLoS Websites

Richard Cave — Tue, 18 Mar 2008 17:56:58 -0700

Performance of the websites hosted on Topaz has increased over the last two weeks with a variety of patches ported to the production servers. We still have an outstanding memory problem that requires a restart of the Topaz applications three times a day (these restarts usually occur around midnight, 8am and 4pm with a duration of less than 10 minutes). I feel that we're close to diagnosing the memory problem which is the last performance hurdle.

We're waiting for the Topaz developers to finish two outstanding tickets before we create a new release candidate (RC 0.8.2.2 for those that are counting). This release candidate will contain all of the patches, even better caching features and XSL/CSS bug fixes. We'll test the release candidate later this week and the production servers will run on this RC no later than Friday, March 28.

We also have a patch for search and will test the patch tomorrow. If testing goes well, then we will be able to place basic and advanced search back on the journal websites by the end of the week.

And now for something completely different.... We had a problem with one of our production servers Friday morning which caused an extended downtime. The Fedora server mysteriously stopped all network traffic and then rebooted. Russ and Josh are bringing the servers down for diagnostic tests and maintenance from 9-9:30pm PST. During this time, the journal websites will display the "undergoing site maintenance" page.

Trackback URL for this post:

http://www.plos.org/cms/trackback/341

TOPAZ update

from A Blog Around The Clock on Tue, 2008-03-18 19:13

The IT/Web has been hard at work to make TOPAZ, the platform for 5 out of 7 PLoS journals work smoothly again....

Performance Issues of PLoS Websites

Richard Cave — Mon, 3 Mar 2008 12:19:25 -0800

We’re continuing to experience slowness and intermittent downtime for the websites hosted on Topaz. At this time, the priorities for the IT and Topaz teams are to improve stability of the Topaz applications and increase the performance of the web sites.

The IT and Topaz teams are digging through many system files, configuration settings and code to improve the situation but it may take one or two weeks before we overcome the performance issues. Please bear with us as during this time as we make additional tweaks/upgrades to improve performance.

Hopefully we’ll address these issues quickly so that our users can enjoy the journal websites.

Trackback URL for this post:

http://www.plos.org/cms/trackback/334

Topaz Release Candidate 0.8.2.1

Richard Cave — Mon, 25 Feb 2008 15:20:10 -0800

Looks like I’ve been a bit remiss about posting release candidates on this blog as we’re on the third release candidate since RC 0.8. We’ve obviously been busy as the release candidate now has an extra numerical identifier. New features include:

minor and formal corrections on articles
advanced search
dynamic table of contents
dynamic journal archives

As with all release candidates, you may need to refresh your browser once or twice to get the new CSS changes.

And while all this development was going on, the I.T. and web teams were busy cleaning and ingesting +1400 articles to make way for PLoS Computational Biology, PLoS Genetics and PLoS Pathogens migration to the Topaz framework. Busy time at PLoS? A bit....

The most exciting features for this release are minor and formal corrections. Users can add a note and select "Correction" from the drop-down list. The PLoS staff will review the note and it will either stay as a note or be changed to a minor or formal correction. A minor correction is represented by a small red triangle "cloud" within the article text. A formal correction is signified the same way but a block appears above the title of the article displaying all formal corrections for the article. Clicking into a formal correction will also show the citation information.

These online corrections are another forward-looking feature for online STM publishing. In the past, PLoS would publish a new "formal correction article" with a link to the original article days/months after the original article published. But this "formal correction article" usually appeared in a different table of contents and could easily be overlooked by the user. Now the formal correction appears directly on the article text and all corrections for a specific article can be retrieved from a single URL. Next steps are to automatically add these corrections to the article PDF as a cover sheet when a user downloads the PDF and create a REST interface to easily retrieve the corrections for an article.

We’ve also re-indexed every published article for advanced search. Now you can search for text within the entire article, the abstract, article title or article references and search by subject category. The search fields also support a variety of search mechanisms include range searches, grouping and Boolean operators. We’ve a bit more work to do on search – look for a "quick article locator" block in the near future.

What else has been going on?!? A small crew at PLoS have been cleaning over 1400 articles from the PLoS Computational Biology, PLoS Genetics and PLoS Pathogens journals for ingest into the Topaz framework. This cleanup, spearheaded by Lynn Murdock, was necessary as we’ve been through a few rounds of XML changes in the last 2.5 years. The ingest of such a large volume of articles has been slowing PLoS ONE, PLoS Neglected Tropical Diseases and PLoS Hub – Clinical Trials down the last few weeks, but we’ve finally finished and the sites should run a bit more smoothly.

We’re still not down yet though – we’re migrating the PLoS community journals to the Topaz platform in short order. And then we’ll get to enjoy the final days of Strong Beer Month!

Here’s a rundown of all the features since RC 0.8 (saving the best for last).

Features implemented in Topaz RC 0.8.1

Display links to related articles on Article and Browse pages.
Administrative interface for collecting Articles into Issue and Issues into Volumes.
Administrative interface for attaching Issue Image article to Issues, and setting the current Issue.
Browse by Issue (Table of Contents) page.

Features implemented in Topaz RC 0.8.2:

Fixes to Browse by Issue (Table of Contents) to add more article types and next/previous links.
A dynamic Browse by Journal (Journal Archive) page.
A single rating field for all front matter articles (e.g. Editorial, Opinion).
Affiliation footnote links are displayed for group authors.
Minor FreeMarker and CSS fixes.
Updates to the volume and issue object models.
Migration from OSCache to ehcache for faster performance.
Administrative interface for viewing and releasing caches.

Features implemented in Topaz RC 0.8.2.1:

Minor corrections on articles.
Formal corrections on articles.
”View All” corrections page.
Admin interface for converting notes to corrections.
Citations for formal corrections.
Advanced search:

Search by author name
Search article text
Search by date published
Search by subject category

Support for note DOIs in the DOI resolver.
Dynamic "Most Viewed" tab for journal home pages.

Features fixed in RC 0.8.2.1

Nomenclature of annotations/discussions changed to notes/comments.
Dynamic Table of Contents with article caching.
Dynamic Journal Archive with article caching.
Re-index of past published articles for advanced search.
Slideshow accommodates small thumbnail issue images.
Overall rating correctly calculates front matter articles.
Order articles in the Table of Contents and RSS feeds ordered by date published and then by article DOI.
Correct ordering of issues in Table of Contents.
Ingest correctly identifies article XML file when multiple XML files are found in the article zip file.
Correctly display front matter ratings when a rating is deleted.
Table of Contents displays "Author Summary" links for research articles.

Trackback URL for this post:

http://www.plos.org/cms/trackback/331

TOPAZ Upgrade and other Big News from PLoS

from A Blog Around The Clock on Wed, 2008-02-27 07:25

It was a heroic (and sometimes nerve-wrecking) couple of months for the IT/Web team at PLoS, but the fruits of their labor are now visible to all. PLoS Computational Biology, PLoS Genetics and PLoS Pathogens have migrated, last night, onto...

Undergoing (Too Much) Site Maintenance

Richard Cave — Fri, 25 Jan 2008 17:11:20 -0800

Many users have seen the "undergoing routine site maintenance" page splashed across their browsers a bit too often recently when browsing to the PLoS ONE, PLoS Neglected Tropical Diseases or PLoS Hub Clinical Trials journals. We've taken some steps to resolve the problems and the TOPAZ developers are digging deep into code. We won't have a full fix in place until TOPAZ RC 0.8.2.1 is released at the end of February, but bandaids are in place that should make the site outages much less frequent. If you're interested in the gorey details, read on....

Russ and I were horrified witnesses to Java processes (that use ehcache) eating away at the 4Gb of memory on the publishing application (pubApp) servers. Eventually, all the memory would fill up and the TOPAZ application would turn into a zombie. To try and remedy the problem, we setup a cron job to automatically restart the pubApp every eight hours. But this caused a secondary problem - if Mulgara was in the middle of a transaction while the pubApp was restarting, the application would hang.

We have alerts setup any time that the pubApp hangs but now Russ and I were waking up in the middle of the night from the unpleasant sounds of SMS alerts inundating our cell phones. And after a few nights of that, we were ready to:

1. Throw the cell phones out the window at 4AM

2. Purchase more memory for the servers.

Yesterday, Josh and I installed 4Gb more RAM (total of 8gb) in the servers. We're still seeing the Java process eat up a bunch of memory, but it's not able to topple the 8Gb of RAM. We see ehcache having some problems when it overflows from memory to disk. And we still have automatic restarts a few times a day. But the sites should be more stable and we'll hopefully have this beast tamed by the next release.

Trackback URL for this post:

http://www.plos.org/cms/trackback/318

Zotero Translator for PLoS Articles

Richard Cave — Tue, 4 Dec 2007 09:27:33 -0800

The Zotero developers announced that Zotero is now compatible with all PLoS journals. The new Zotero translator allows users to capture metadata and full-text PDFs of PLoS articles. I've tried Zotero with PLoS ONE and PLoS NTD articles and it automatically gathers metadata information including the authors of an article, abstract, volume/issue and DOI. You'll need to download the latest Zotero 1.0.1 release to get the new PLoS translator.

There's probably a fair bit of people scratching their head and asking "What is Zotero?" Zotero is a great Firefox extension that allows you to collect, manage and cite your research sources. It works seamlessly in your browser and automatically collects metadata from an article, allows you to create tags about the information and take notes. Best of all - it's free!

The developers have created a huge number of translators for libraries and universities around the world as well as translators for Amazon and Flickr. Zotero also has integration with Microsoft Word and OpenOffice so you can easily cite something from your Zotero collection. On of the Zotero team members is also working on a PLoS bibliographic style export which should be released in the next version.

Zotero is a great application and I highly recommended giving it a try.

Trackback URL for this post:

http://www.plos.org/cms/trackback/295

Topaz Release Candidate 0.8

Richard Cave — Thu, 20 Sep 2007 09:15:51 -0700

In the past, I used the title "PLoS ONE Release Candidate." But with this release candidate, the Topaz framework allows multiple journal websites to use a single repository. So the release candidate isn't specific to just PLoS ONE. Why is this important? Enabling multiple journals on one repository allows articles to be viewed across all sites accessing the repository. The interactive tools of PLoS ONE will be available for all journals hosted on Topaz. This will include articles originally published in PLoS Clinical Trials and articles soon-to-be published in PLoS Neglected Tropical Diseases.

Features implemented in Topaz RC 0.8:

Enable multiple journals using a single repository.
Skins for multiple journals.
Filter search results by journal using OTM.
TrackBack linkbacks for articles. TrackBacks are used primarily to facilitate communication between blogs - a TrackBack allows a blog author to see who is linking back to a blog post. We're using this same feature, based on the Six Apart specification, to see who is linking back to a published article. To create a trackback to a PLoS ONE article, you can use a trackback URL similar to http://www.plosone.org/article/info:doi/10.1371/journal.pone.0000000/trackback - you just need to change the "0000000" to the actual article number.
Citation download of the article. Many users have requested this feature and now you can download the citation for an article in the RIS file format (compatible with EndNote, Reference Manager, ProCite, RefWorks) or BibTex (compatible with BibDesk, LaTeX). Test the feature out by downloading the citation for the article on Protein-Tyrosine Kinase Activity Profiling in Knock Down Zebrafish Embryos.
A fix to a rating display issue in IE7.
Allow multiple email addresses for "E-mail this Article."
Administrative modifications to annotations.
Administrative interface for multiple journals.
Migration to the Struts 2 web application framework. Struts 2 is a merger of Struts and another popular J2EE framework, WebWork. This release offers many enhancements and refinements while retaining the same general architecture of the original Struts framework.

Props to Russ, Susanne and the entire Topaz development team!

Trackback URL for this post:

http://www.plos.org/cms/trackback/260

SciVee Launch = Scientific Publication and Video Presentation

Richard Cave — Mon, 20 Aug 2007 12:40:29 -0700

SciVee is now open to the scientific community. SciVee allows scientists to upload a video or audio file and synchronize the video/audio to the content of their scientific paper. This is great for scientists that want to quickly summarize their research, engage the scientific community in discussions around an article and broaden the exposure of their publications.

SciVee allows users to tag, rate and comment on videos. Links to the original articles are included with the video. An example is Eric Scheeff explaining his paper on Structural Evolution of the Protein Kinase–Like Superfamily. Eric talks the users through a nice slideshow that introduces the Protein Kinase family before getting into the detail his paper.

There is a catch – SciVee only allows scientists to upload publications that were published in Open Access journals - another great reason to publish in an Open Access journal.

There’s a nice discussion about SciVee in the SlashDot article YouTube for Science?.

But SciVee is not alone - there are a few other sites that present scientific videos. ScienceHack provides a search engine for scientific videos. The videos that they host have been screened and approved on accuracy and quality by their scientists – a peer-review system for science videos. The Journal of Visualized Experiments is an online research journal that allows researches to publish videos of their experiments and methods. And VideoLectures contains over 2500 science videos and presentations related to Computer Science, Data Mining, Semantic Web, etc.

It’s great to see this new medium (video over the internet) applied to science and accessible to the public. A couple of years ago, the bandwidth and storage requirements brought many a start-up company to their knees. But now that these costs are minimal, sites live SciVee are launching and making science more accessible to scientists and the general public.

Trackback URL for this post:

http://www.plos.org/cms/trackback/251