I stumbled on this post over at the flickr developers blog while chatting with Jo about web applications designed and built for “the cloud”. An article earlier this week piqued my curiosity about message queues, and got me thinking about how they can be used to move processing tasks out of the way. I’d known already that minimizing those tasks within, let’s say, a PHP script that builds a page can help your application scale or seem more responsive.
To do so, I’ve relied on cron jobs, but the main problem with that approach is either that your cron job runs when it has nothing to do, if you run it too frequently, or it has too much to do and you run the risk of one job starting while another is in progress and working with potentially the same data.
Code: Flickr Developer Blog » Flickr Engineers Do It Offline
For scale, Flickr separates these three lookups into three different places. When you upload that photo, we immediately tell you about it, and get out of your way so you can go off and marvel at it. We don’t make you wait while we tell your contacts about it, insert it into the search system, notify third-party partners, etc. Instead, we insert several jobs into our queueing system to do these steps “later”. In practice, every single one of these actions subsequently takes place within 15 seconds of your upload while you’re free to do something else.