The drupal site has a page on tuning server performance. While a few of the tips are specific to Drupal, the majority of them apply to anyone who’s using a LAMP server.
Understanding LAMP performance. This LAMP performance study revealed that Apache is bandwidth limited, PHP is CPU limited, and MySQL is memory limited and disk I/O bound. Be sure to compile your stack natively for maximum performance. You can also read Optimize high traffic servers.