Colorized Word Diffs

I’ve been finding myself doing a lot for copy and tech editting. I needed a way to annotate a PDF based on the changes I’d made to our markdown source. Trying to eyeball this was difficult, and I checked if there was a way to do word-by-word diffs based on SVN output. Remember, SVN’s diff command will show line-by-line differences. But there is a Perl script wdiff that will show you word by word diffs. With a bash alias, you can pipe svn output to it and see what’s been added/removed more clearly.  Once you install wdiff, either from source or via your distributions package manager, create an alias (I use bash) that will colorize its output:

alias cwdiff="wdiff -n -w $'\033[30;41m' -x $'\033[0m' -y $'\033[30;42m' -z $'\033[0m'"

wdiff compares two files by default but in can also parse the output of a diff file. By piping svn diff to it, you can see the changes word for word. In the example below, I’m reviewing the changes made by a specific commit and using less to page the output, the -r flag keeps the colors intact.

svn diff -r 267 -x -w 05.chapter.markdown | cwdiff -d | less -r

Words that were deleted will be in red, additions are in green.

color-diff

 Update:

Git has this behavior built in using:

git diff --color-words

Also, if you need to share your changes with a designer (who maybe needs to update a PDF…), with this ansi2html script from pixelbeat.org, you can generate HTML output showing your changes. I found that its useful to run the generated HTML through fold so that the lines wrap if you email it to someone.

svn diff foo.markdown | cwdiff -d | ~/bin/ansi2html.sh | fold -s > foo-updates.html

Finally, you can wrap all this functionality into a simple bash script that you can save anywhere. Pipe it the output of a svn diff or diff command, and get nice HTML as a result. It assumes ansi2html is somewhere in your executables $PATH.

#!/bin/bash
# take diff input (from svn diff too), colorize it, convert it to html

cat - | wdiff -n -w $'\033[30;41m' -x $'\033[0m' -y $'\033[30;42m' -z $'\033[0m' -d \
| ansi2html.sh | fold -s

Quick mysqldump snapshot script

I find myself needing this often on different servers, you may find it useful too. This script creates a timestamped dump of a mysql database to the current directory. Assumes it runs as a user who can connect to the database. You can set those credentials using the -u and -p command line switches for mysqldump

#!/bin/bash
# retrieve a database and gzip it

if [ $# -ne 1]
then
  echo "Usage: `basename $0` {database_name}"
  exit $E_BADARGS
fi

DB="$1"

DATEZ=`date +%Y-%m-%d-%H%M`
OUTFILE="$DB.$DATEZ.sql.gz";

echo "mysqldump for $DB to $OUTFILE"
sudo mysqldump --opt -e -C -Q $1 | gzip -c > $OUTFILE

Automating FTP uploads

I needed to automate copying files for a website that I was building. Since this site was hosted on an inexpensive shared hosting plan, I didn’t have the luxury of shell or rsync access to automate copying files from my local development environment to the host. The only option was FTP, and after wasting too much time manually tracking down which files I needed to update, I knew I needed an automated solution. Some googling led me to lftp, a command-line and scriptable FTP client. It should be available via your distribution’s repository. Once installed, you can use a script like the one below to automatically copy files.

{syntaxhighlighter BASH}
#!/bin/sh
# paths
HERE=`pwd`
SRC=”$HERE/web”
DEST=”~/www”
# login credentials
HOST=”ftp.commodity-hosting.com”
USER=”mysiteusername”
PASS=”supersecretpassword”
# FTP files to remote host
lftp -c “open $HOST
user $USER $PASS
mirror -X img/* -X .htaccess -X error_log –only-newer –reverse –delete –verbose $SRC $DEST

The script does the following:

  • Copy files from the local ./web directory to the remote ~/www directory.
  • Uses $HOST, $USER, $PASS to login, so make sure your script is readable, writeable, and executable only by you and trusted users.
  • the lftp command connects and copies the files. The -c switch specifies the commands to issue, one per line. The magic happens with the mirror command which will copy the files. Since we added the –only-newer and –reverse switches, this will upload only files which have changed.
  • You could be a little safer and remove the –delete switch, which will remove files from the destination which are not on your local machine.
  • You can use the -X to give it glob patterns to ignore. In this case, it won’t touch the img/ directory or the .htacess file.

If you’re still moving files over FTP manually, even with a good GUI, it’ll be worth your time to automate it and make it a quicker, less error-prone process.

Fix SSL timeouts with the Facebook PHP-SDK

I ran into SSL timeouts on in local development setup when I was re-factoring some integration code with facebook and using their SDK. It was tricky to diagnose, I was sure that my changes couldn’t be the cause, and I finally confirmed it by running our production codebase. Since it was having the same timeout error, I knew the bug had to be in an underlying layer.

For the record, I’m running this version of curl on my Archlinux box:

<code>curl 7.25.0 (x86_64-unknown-linux-gnu) libcurl/7.25.0<br /> OpenSSL/1.0.1 zlib/1.2.6 libssh2/1.4.0
</code>

I also got the error from the command line with

<code>curl "https://graph.facebook.com/oauth/access_token"
</code>

But it is fixed with

<code>curl --sslv3 "https://graph.facebook.com/oauth/access_token"
</code>

Debian Server

On a debian squeeze server, with the latest (4/3/2011) version of curl:

<code>curl 7.21.0 (x86_64-pc-linux-gnu) libcurl/7.21.0<br /> OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.15 libssh2/1.2.6
</code>

The timeout does not happen with either of the following commands:

<code>curl "https://graph.facebook.com/oauth/access_token"
curl --sslv3 "https://graph.facebook.com/oauth/access_token"
</code>

OS X

The time out does not happen on OS X which runs curl 7.21.4

Thoughts

So, this timeout only seems to affect users with very new version of curl. Fixing it requires adding a line to the Facebook PHP SDK, which while minor, you have to remember if you ever upgrade it. At the same time, this bug could come back and bite you down the road if your operating system sneaks in a newer version of curl. You can see a fork of the PHP SDK with this fix on github.

Other references:

  1. Facebook bug ticket
  2. Maybe related PHP bug

Make any printer AirPrint compatible

At home, we were missing the ability to print from our iPad or iPhones. While I’m not an OS zealot (anymore), I did upgrade to an iPhone last month, and have had an iPad for a while now. They’re very useful for casual computing, checking email, browsing. But if we needed to print something, it was a hassle to fire up a laptop or desktop computer to use our networked printer.

It turns out that Apple’s AirPrint uses DNS Service Discovery, an open standard.  There are programs out there for Windows and Mac to let you share a printer attached to that OS via Airprint.  It turns out, you can use it on Linux if you run the Avahi daemon and CUPS. Of course someone has already figured out how to do it: CUPS with Apple AirPrint, using a python script airprint-generate.

I was following the instructions, but the airprint-generate.py program would not generate the XML file to make it all work. Diving into the code, I saw that the printer had to be configured as a shared printer in CUPS, which makes total sense. I didn’t have it configured that way, since its a network printer, any other device could connect directly to the printer server. The setting can be changed through the cups web interface or by adding the line below to your printers.conf file. airprint-generate.py will now find your printer and generate the configuration file to add to your avahi services directory.

Shared Yes

The one downside to this setup is that the computer running cups + avahi has to be on for this to work. But, you could buy an inexpensive ARM device, like the upcoming Raspberry PI or another one listed here and build yourself a custom, Airprint compatible printer server.

Virginia Tech professor cuts PC energy usage

I came across this link about a Virginia Tech professor that wrote a program to reduce energy wasted by computers. On the face of it, it sounds a bit smarter than the usual power management techniques that shut off parts or all of a computer when its not being used. It claims up to 30% reduction in power, which I expect translates into noticeably longer times between batter charges, if you use it on a laptop.

He also likens his software to cruise control in cars. But while cruise control will ease up on the accelerator as a car gathers speed when descending, it “doesn’t look ahead and say, ‘Hey, there’s a hill coming,’” Cameron says. “It just reacts.” In contrast, his program seeks out patterns to determine when a computer will need more power. He says users report electricity savings of 30 percent.

I expected the program, named Granola and distributed by miserware, to be a Windows only affair. However, its available for both Windows and various distributions of Linux. The latter likely a result of wanting to reduce server power usage, an area where Linux has a sizeable presence.  There is even a community supported, AUR package of Granola for ArchLinux. Joy!

Fixing Realtek ethernet driver on Linux

When my 3 year old son pressed the sleep button on my keyboard, we discovered that it actually sends my Archlinux desktop to sleep. Who would have thunk it? What was more annoying is that when the machine woke up, the network connection did not come back.

Poking around in the log files, I saw lines like the following lines which lead me to believe it was a NetworkManager problem.

Oct 26 09:12:30 grouch NetworkManager[14296]: <info> (eth0): carrier now OFF (device state 70, deferring action for 4 seconds)
Oct 26 09:12:31 grouch NetworkManager[14296]: <info> (eth0): carrier now ON (device state 70)
Oct 26 09:12:32 grouch NetworkManager[14296]: <info> (eth0): carrier now OFF (device state 70, deferring action for 4 seconds)
Oct 26 09:12:34 grouch NetworkManager[14296]: <info> (eth0): carrier now ON (device state 70)
Oct 26 09:12:50 grouch NetworkManager[14296]: <warn> (eth0): DHCPv4 request timed out.
Oct 26 09:12:50 grouch NetworkManager[14296]: <info> (eth0): canceled DHCP transaction, DHCP client pid 15506
Oct 26 09:12:50 grouch NetworkManager[14296]: <info> Activation (eth0) Stage 4 of 5 (IP4 Configure Timeout) scheduled...
Oct 26 09:12:50 grouch NetworkManager[14296]: <info> Activation (eth0) Stage 4 of 5 (IP4 Configure Timeout) started...
Oct 26 09:12:50 grouch NetworkManager[14296]: <info> Activation (eth0) Stage 5 of 5 (IP Configure Commit) scheduled...
Oct 26 09:12:50 grouch NetworkManager[14296]: <info> Activation (eth0) Stage 4 of 5 (IP4 Configure Timeout) complete.
Oct 26 09:12:50 grouch NetworkManager[14296]: <info> Activation (eth0) Stage 5 of 5 (IP Configure Commit) started...
Oct 26 09:12:50 grouch NetworkManager[14296]: <info> Activation (eth0) Stage 5 of 5 (IP Configure Commit) failed (no IP configuration found)
Oct 26 09:12:50 grouch NetworkManager[14296]: <info> (eth0): device state change: ip-config -&gt; failed (reason 'ip-config-unavailable') [70 120 5]
Oct 26 09:12:50 grouch NetworkManager[14296]: <warn> Activation (eth0) failed.
Oct 26 09:12:50 grouch NetworkManager[14296]: <info> Activation (eth0) Stage 5 of 5 (IP Configure Commit) complete.
Oct 26 09:12:50 grouch NetworkManager[14296]: <info> (eth0): device state change: failed -&gt; disconnected (reason 'none') [120 30 0]
Oct 26 09:12:50 grouch NetworkManager[14296]: <info> (eth0): deactivating device (reason 'none') [0]
</info></info></info></warn></info></info></info></info></info></info></info></info></warn></info></info></info></info>

After a lot of googling, it turns out that the problem was the kernel was using the wrong driver for the network card. First to find out which type of Realtek card you have use the following command:

lspci | grep Real

If you see the following, you need to update the driver loaded by the kernel to one provided by Realtek.

01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)

If you are using Archiinux, its as simple as installing the AUR package for the r8168 module. Using the package, the module downloaded and compiled fine for me, with the latest 3.0 Linux kernel.

yaourt -S r8168

Finally, reboot and you should still have a working network with the added bonus that it will automatically reconnect when resuming.

A Sneak Peek at Upcoming Linux Mint 11

Linux Mint, an ubuntu derivative, posted a preview of what’s included in there next release, Linux Mint 11. It sounds like they are positioning themselves as a nice alternative for users who don’t like Unity or the new Gnome interface, and users who can’t run the newfangled UIs.

One of the most noteable features of version 11 will be the retention of GNOME 2.32. It will be the foundation for the same desktop layout users know and love, including Compiz and Metacity.

A Sneak Peek at Upcoming Linux Mint 11

Printing with CUPS and KDE4

Printing in linux has always been a particular headache for me, and probably many users.There is a myriad of systems that have to talk to each other to get it to work. You have CUPS which takes care of making your computer talk to the printer, various packages like foomatic and ghostscript that you may or may not need so that CUPS can talk to the printer itself. Then you have the printer configuration frontends under GNOME, KDE, or whatever desktop environment you might be running that, in theory, hook the applications you use up with CUPS.

For a couple of years now, I’ve been running ArchLinux with a KDE desktop. I’ve succesfully configured CUPS to print, but I could never actually print from an applicatoin, like Okular, KDE’s PDF viewer. Whenver I used KDE’s system printer configuration utility to add a printer, the options on the left simply listed “Other” with a field for my printers Device URI (yeah – what is that for a printer, exactly?). KDE could not talk to CUPS, even though CUPS was configured correctly, and didn’t have any access restrictions.

Poking around in /etc/cups/cups.conf, I saw the following lines:

# Socket disabled by default it makes KDE fail on CUPS
# Listen /var/run/cups/cups.sock

I uncommented the Listen line, and presto, my CUPS printer is visible in KDE! This comment must be for an older version, I’m running 4.6.2.

Using Dropbox on KDE

Dropbox is a handy service to automatically synchronize files between different computers.  Each computer requires a client that connects to the service and monitors the dropbox folder for changes.  It’s free for up to 2GB of storage and works well.

One of the cool things about Dropbox, is that they provide a native linux client but it integrates with the Gnome desktop environment.  While you can get it working, its a bit of a hassle.  However, I just found a KDE application that works with it, named KFilebox, formerly known as KDropbox. To install, you can grab the .deb or .rpm file and follow the instructions.  If you use ArchLinux, its available on AUR.