We're making more progress on a couple of fronts. Russ has been able to demonstrate that memory is being gobbled up on the servers. We see memory filling up on the pubApp servers over the course of 6-8 hours. Once the memory fills up, we see a high CPU usage problaby due to ehcache having to invalidate items from the cache on every operation. Russ is setting up a profiler to examine memory use on our test servers to see if he can reproduce the problem under load. We're still not exactly sure where the memory leak occurs but have a couple of ideas:

• ehcache
• Apache Axis memory mapper / SOAP server
• Dead mod_jk connections

We've also tracked down a number of actions that are querying Mulgara directly rather than using cache including some queries than can return a huge amount of data. The developers are re-writing the code to use the cache or to be smarter about data they're pulling.
We're not going to make any more patches to the production server until we diagnose and fix the memory problem. Search will be reinstated back on the site after we can build and test a new release candidate in the next 1-2 weeks.