We at CBC.ca have made major improvements in our web platform over the last two years. When I first returned to CBC in September 2006, we were still running Apache 1.3.29 on SuSE Linux Enterprise Server. Since then, we’ve upgraded first to Apache 2.0.59 (still on SuSE) and, with the migration to Red Hat Enterprise Linux in July of this year, to Apache 2.2.8. (You can see the evolution of our web platform over at Netcraft.)
Two days after the Canadian Federal Election, we implemented the next major upgrade of that platform and that was to convert from the prefork MPM to the worker MPM. Since we monitor the performance of all our Apache servers using Cacti, I can share some detailed information about the performance improvement that has resulted from this change.
It should be no surprise that changing to a more modern MPM results in such performance improvements. The worker MPM spawns a small number of child processes and each process in turn spawns threads to handle incoming HTTP connections, in contrast to the prefork MPM which relies solely on processes. The reason we have been so conservative about changing MPMs is that we have had trouble with older libpthread implementations on Linux that resulted in really bad interactions with the kernel (e.g. the system load would be in the hundreds but there were no processes on the CPU). Most of these issues occurred under SuSE Enterprise Linux but now that we are on Red Hat Enterprise Linux, we are much more confident that such problems have been solved.
So, onto the stats. The following graph shows Apache server performance (of one webserver) during a four week period ending today.
(Since we use the Akamai CDN, these graphs bear no resemblance whatsoever to our actual site stats so I can share them publicly.)
I’ve marked the day of the federal election specially so that you can see that the site traffic on that day was exceptional. However, the week before shows typical origin traffic patterns and volume, although slightly elevated due to the election buildup.
You’ll notice that after the MPM switch (two days after the election), there are more active threads, although the number of threads in KeepAlive is much lower. I believe this represents a performance improvement; the pool of threads servicing incoming requests is smaller because they can respond faster. On the other hand, there are more available threads (in "Waiting" state) to respond to a surge in requests.
Note that when we switched from prefork to worker, we did not change the MPM tuning parameters at all, except where dictated by the MPM. By this I mean that we kept MaxClients the same and only varied MPM-specific parameters like ThreadsPerChild. So I believe this is a valid comparison between MPMs but I”m willing to entertain any comments about my methodology.
Overall we’re very satisfied with the improved performance by switching MPMs. Our origin webservers are rarely CPU-constrained (running at about 20-30% utilization on a good day) but are memory-constrained, as one might expect of a large website with a large MaxClients setting.
Future work in Apache performance tuning will include increasing MaxKeepAliveRequests, which we believe is set quite low (we have it at 150) and potentially also increasing ThreadsPerChild (while revising the number of processes downward). Since we made the switch to worker, we have also implemented mod_deflate to gzip-encode content between the CDN and the origin, in order to reduce bandwidth (and since we have CPU to spare on the webservers).