Today I was passed a whitepaper titled "Optimizing Drupal Performance - Benchmark Results" from a joint Zend and Acquia effort. The quick summary is pretty clear and straightforward:

This paper presents the results of performance benchmarks for comparing several common PHP runtime environments and configurations. The application used for benchmarking is Drupal, the open source social publishing platform.

While I have not attempted to duplicate or validate any of their individual numbers or conclusion as a whole, I have been a Drupal user for well over five years and have launched 30+ sites on it. Even more importantly, in the coming month, we are launching a network of Drupal-based sites for a US professional sports league, so performance and optimization are near and dear to my heart and at the front of my mind.

Regardless, a few things that were of particular interest in this report:

• First, they provided a great deal of information on their Apache, Linux, and PHP configuration. While this is a good start, there are dozens - if not hundreds - of configuration options not documented here. If Zend or Acquia release their configuration files, I'd like to test their configuration and duplicate or further tune it.
• Next, they clearly lay out the configuration of the hardware itself. When you're performing a benchmark, you have to minimize variables. Personally, it's nice to see this well done and described for once.
• Next, page 4 lists the urls requested (included below). While those are valid pages, it's rare that they would be requested in this form. To duplicate a more common usage pattern, they should turn on and use both the Path module and Clean Urls. After all, everyone loves Google juice.

/drupal/ (home page)
/drupal/?q=content/hello-blog-entry (blog entry with 2 comments)
/drupal/?q=node/1 (top node)
/drupal/?q=node/2 (child node)

• Next, the Windows performance ranged from sluggish to abysmal. On the low-end of with Drupal's caching at normal, it was 126 to 105 requests per second with  Linux and Windows respectively. On the high end, using Zend Server+page cache the numbers move to 987 and 628 respectively. If Zend and Acquia's goal is to discourage users from using Windows, this will be quite effective.
• Next, what are we really optimizing here? Admittedly, I didn't notice this one initially, but when I shared this article with fellow DCPHP'er and Drupal expert Oscar Merida, he pointed out the oddity of this configuration compared to the title of the paper "Optimizing Drupal Performance". If you look closely, this is only attempting to improve PHP's performance, not any other component of the system. How do we know that PHP is the bottleneck at this point?
• Finally, what portions of Drupal are we really testing? On page 10, they describe the modules used but most of these modules can be used in a variety of ways. Some - like Views - can be lightweight and provide great functionality or used in ways that bring the system to a crawl and make kittens cry. Without knowing how they used these modules - and it looks like only a few were used - we can't evaluate if this is a real world configuration or just a test.

Overall, like most whitepapers, this has to be taken with a grain of salt. If your configuration is substantially similar to the one described, you may see these numbers and the corresponding gains, but most likely you won't. Regardless, the tools and configurations described would be enough to investigate optimization options.

If Zend and Acquia wanted to improve this study, it would be valuable to find a reasonably-sized but unoptimized Drupal site, clone it, and optimize that. Even better, by looking at the system logs, it would be feasible to create requests - or a series of requests - that mimic visitors' use of the site.

Disclosure: I have worked for Zend at the previous two ZendCons where I served as their unconference chair. Their most recent two community managers - Cal Evans and Eli White - are close personal friends. I have no relationship with Acquia in any way shape or form but have been a Drupal user/builder/customizer for 5+ years. Microsoft is a customer of Blue Parabola and some of their Evangelists are friends and well-respected in the PHP community.