Fix SSL timeouts with the Facebook PHP-SDK

I ran into SSL timeouts on in local development setup when I was re-factoring some integration code with facebook and using their SDK. It was tricky to diagnose, I was sure that my changes couldn’t be the cause, and I finally confirmed it by running our production codebase. Since it was having the same timeout error, I knew the bug had to be in an underlying layer.

For the record, I’m running this version of curl on my Archlinux box:

curl 7.25.0 (x86_64-unknown-linux-gnu) libcurl/7.25.0
OpenSSL/1.0.1 zlib/1.2.6 libssh2/1.4.0

I also got the error from the command line with

curl ""

But it is fixed with

curl --sslv3 ""

Debian Server

On a debian squeeze server, with the latest (4/3/2011) version of curl:

curl 7.21.0 (x86_64-pc-linux-gnu) libcurl/7.21.0
OpenSSL/0.9.8o zlib/ libidn/1.15 libssh2/1.2.6

The timeout does not happen with either of the following commands:

curl ""
curl --sslv3 ""


The time out does not happen on OS X which runs curl 7.21.4


So, this timeout only seems to affect users with very new version of curl. Fixing it requires adding a line to the Facebook PHP SDK, which while minor, you have to remember if you ever upgrade it. At the same time, this bug could come back and bite you down the road if your operating system sneaks in a newer version of curl. You can see a fork of the PHP SDK with this fix on github.

Other references:

  1. Facebook bug ticket
  2. Maybe related PHP bug

Linux Package management overview

I have to confess that delegating software installation to Debian and Ubuntu’s apt command is what finally converted me to Linux.  I stillhave a bias against .rpms and building from source based on disastrous experiences hunting down obscure .rpms or figuring out why make would not work.  If you’re trying out Ubuntu or another Linux distribution, you should stop and read download squad’s Package management 101

Package management refers to the way your distribution installs and configures (as well as manages and removes) software applications and libraries on your system. When Windows installs an .exe (which is the closest thing in Windows to a package) it usually places it in a single specific place within a directory. Linux installs across a few directories, leaving many new Linux users scratching their heads as to where their .rpm actually went. Most distributions install the executables in /usr/bin, and the libraries in /usr/lib. You may notice related files in /usr/share or /etc.

In short, you’ll want to let your package manager install and upgrade new software for you.  You don’t have to take my word for it, Thank You, Aptitude!

I’ve long believed that the easiest way to install software on a modern operating system is through a well-designed package manager connected to one or more carefully-maintained package repositories.


Fail2Ban + Mod_Security = Spammer Bouncer

Under debian, fail2ban’s configuration is in /etc/fail2ban/.  In the filter.d directory add the following file and name it apache-modsec.conf.

# Option:  failregex
# Notes.:  regex to match the password failure messages in the logfile. The
#          host must be matched by a group named "host". The tag "<HOST>" can
#          be used for standard IP/hostname matching.
# Values:  TEX
failregex = [[]client <HOST>[]] mod_security: Access denied with code 500
# Option:  ignoreregex
# Notes.:  regex to ignore. If this regex matches, the line is ignored.
# Values:  TEXT
ignoreregex =

Enable this filter by  adding the followin to jail.local:

enabled = true
port    = http
filter  = apache-modsec
logpath = /var/log/apache*/*error.log
maxretry = 4

Restart fail2ban, next time a spammer gets blocked by modsecurity 4 times, it will be recorded in /var/log/fail2ban.log

2007-02-07 11:52:45,024 fail2ban.actions: WARNING [apache-modsec] Ban

Debian, PHP5, and session garbage collection

Poking around, like you do on your server, I noticed that php session files weren’t being deleted, even after a few days. Turns out, PHP’s automatic garbase collection is disabled on Debian. Instead, its replaced by a cron script that cleans out the /var/lib/php5 directory, which is where sessions are saved by default. If you’re a security conscious PHP developer, you give each php application its own session save path. If you do that on a debian box, make sure you reenable garbase collection by setting seesion.gc_divisor to a positive integer.

; Define the probability that the 'garbage collection' process is started
; on every session initialization.
; The probability is calculated by using gc_probability/gc_divisor,
; e.g. 1/100 means there is a 1% chance that the GC process starts
; on each request.

; This is disabled in the Debian packages, due to the strict permissions
; on /var/lib/php5. Instead of setting this here, see the cronjob at
; /etc/cron.d/php5, which uses the session.gc_maxlifetime setting below
;session.gc_probability = 0
session.gc_divisor = 100

Also, the README.Debian file spell this out, shame on me for not reading it, but it should also contain instructions for handling the above situation.

    Session files are stored in /var/lib/php5.  For security purposes, this
directory is unreadable by non-root users. This means that php5 running
from apache, for example, will not be able to clean up stale session
files. Instead, we have a cron job run every 30 mins that cleans up
stale session files; /etc/cron.d/php5. You may need to modify how
often this runs, if you've modified session.gc_maxlifetime in your
php.ini; otherwise, it may be too lax or overly aggressive in cleaning
out stale session files.

Server Upgrade: OS basics

Debian AMD64 Setup

My Linux distribution of choice is Debian, although for the desktop I’d recommend Ubuntu. The server was first setup using the unofficial debian64 repositories. Since the amd64 architecture is now officially part of Debian, we moved to using an official repository. see Google groups. An unofficial repository contains /debian-amd64/ like so:

#deb sarge main contrib
#deb-src sarge main contrib

Since packages make it to stable at a fairly slow pace, we also want to use the testing branch. This is fairly straightforward, first update your /etc/apt/sources.list to use etch/testing by removing (or commenting out) other repositories and adding the following lines. If you are outside the USA, replace the .us. part with your country code.

deb testing main contrib
deb-src testing main contrib

Next, run apt-get to update the system, first clean the system, then update, and upgrade.

apt-get clean
apt-get update
apt-get dist-upgrade

Follow the onscreen instructions, when in doubt pick the default option or swithc to google and read up. If apt installs a new kernel, you’ll have to restart your server. Finally, to get up to date LAMP packages, you can use the dotdeb repositories by adding the following lines to your sources.list. We will need these later to install Apache2, MySQL, and PHP5.

deb stable all
deb-src stable all

Kernel Parameters

Since our last server crashed because of a very low max open files limit, its worth checking that this setting is not too low on the new server.

cat /proc/sys/fs/file-max

Currently this returns 100905, which should be more than enough open files for our expected traffic. See Debian kernel tuning for more info.

Enabling Hyperthreading

Finally, because the CPU is an Intel P4 with hyperthreading, you can try using an SMP kernel to enable the 2 “virtual” processors. While there seems to be some debate on the benefits of Intel’s hyperthreading, IBM produced some benchmarks on the 2.6.15 kernel which show some gains. You can install a new kernel with:

apt-get install linux-image-2.6.16-2-em64t-p4-smp

Make sure you install the correct one for your CPU, since there are a generic 64-bit kernel, AMD specific kernels, Intel specific kernels, as well as single and multi-processor versions for each.

That’s it for this part, if you restart you should still have a working server (I do). Still to do: basic system security and setting up a LAMP environment


Fun Saturday night

Bizarre. My self-signed SSL certificate I use to access my email using IMAP over SLL expired last week. The real significance is that we’ve been on this server for one year now. Compared to how painful working on the old Redhat 7.2 install we had back then, working with a Debian system has been a breeze.

So, I figured I’d just generate a new certificate, the instructions are readily googleable and I’d be playing a little Warcraft in no time. But no, after following the instructions Cyrus’s imap server wouldn’t start up. Eventually I figured out it was complaining about a missing lib related to the net-snmp package. A year ago, Cyrus 2.2 wasn’t in Debian testing yet, but it is now so it ended up being time to upgrade to the official packages.

The install was easy as usual, except for some disconcerting messages about my Cyrus databases needing an upgrade from DBD 3.2 to 4.2. To get the upgrade tool I installed the db4.2-utils package and ran the databases through the utility. I had my fingers crossed, with the spectre of data loss looming nearby (although I do have nightly backups that in theory I can recover from). The conversion worked well to, and all my email is in its place.

Feeling invincible, at this point I decided it was time to upgrade the kernel from the old stock 2.4 kernel to a newer 2.6 linux kernel. The only real reason is that I’d read Mysql performs noticeably better on the 2.6 kernels. Can you tell a difference? As far as I can tell, nothing is massively broken. If you notice anything odd though, please let me know!


Debian Desktop Destruction and Recovery

Luckily the install is very painless and mostly automated (easier than the last windows install I did). Despite all the critiques of linux installs (and yes, it did cross my mind that a new Mac would be nice), I find myself hunting for drivers and driver CDs (especially for motherboard subsystems) than I have to compared to Windows.

That said, I have to do the following to get the PC back into a useful state – install nvidia binary drivers, setup twinview for the dual monitors, setup apache2 virtual hosts for my local development sites, install mysqll4.1, install php5, install subversion. The most painful one will be recompiling the kernel to include mppe support so I can vpn into the network at Forum One and be productive on Wednesday.

Not sure what it says about me, or my comfort with linux nowadays compared to 3 years ago, but none of the above sound all that challenging. I think its mostly because apt is such a handy packaging system, debian developers do a good job maintaing packages, so that something like subversion is just an apt-get install subversion away.

Update: at 3am, I started installing at 11:30pm, I had a working system with all of the above. MMPE support is now in them main linux kernel tree so no more patching and kernel recompiles for me!


Hula & Debian

If, like me,  you have a Debian box and are itching to try out Hula, last week Hula packages made it into the Unstable branch.


Tweaking Firefox & Thunderbird in Linux

Both on my Debian PC’s at home and work, I’ve been annoyed that my Firefox and Thunderbird installations didn’t launch each other when I clicked on a link or email address. For example, at work, Evolution would come up if I clicked on a mailto link on a web page. Inside each program’s preferences there were no options for controlling this behavior. These preferences are set in the gnome-control-center. Each application will respect the preferred browser and preferred email programs defined in there. I forgot where I found this tip recently but I can’t take credit for discovering it on my own.


on Apt-y goodness

Jason has discovered the goodness of APT for managing installed software on a Linux box. You gotta respect a tool that lets you upgrade pretty much your entire distribution with one command: apt-get dist-upgrade .