The drupal site has a page on tuning server performance. While a few of the tips are specific to Drupal, the majority of them apply to anyone who's using a LAMP server.

Understanding LAMP performance. This LAMP performance study{rel="nofollow"} revealed that Apache is bandwidth limited, PHP is CPU limited, and MySQL is memory limited and disk I/O bound. Be sure to compile your stack natively for maximum performance. You can also read Optimize high traffic servers{rel="nofollow"}.\