Building CandiData

This past weekend, my colleague and friend Sandy Smith participated in Election Hackathon 2012 (read his take of the hackathon). We built our first public Musketeers.me product, Candidata.me. This was my first hackathon, and it was exciting and exhausting to bring something to life in little more than 24 hours. Our idea combined a number of APIs to produce a profile for every candidate running for President or Congress in the United States. The seed of the idea was good enough that we were chosen among 10 projects to present it to the group at large on Sunday afternoon.

Under the Hood and Hooking Up with APIs

We used our own PHP framework, Treb, as our foundation. It provides routing by convention, controllers, db acccess, caching, and a view layer. Along the way, we discovered a small bug in our db helper function that failed because of the nuances of autoloading.

I quickly wrote up a base class for making HTTP Get requests to REST APIs. The client uses PHPs native stream functions for making the HTTP requests, which I’ve found easier to work with than the cURL extension. The latter is a cubmersome wrapper to the cURL fucntionality.  

To be good API clients, we cached the request responses in Memcached between an hour to a month, depending on how often we anticipated the API response to change.

Sandy also took on the tedious – but not thankless – task of creating a list of all the candidates that we imported into a simpl Mysql table. For each candidate, we could then pull in information such as

  • Polling data from Huffington Post’s Pollster API, which we then plotted using jqplot. Polls weren’t available for every race, so we had to manually match available polls to candidates.
  • Basic Biographical information from govtrack.us
  • Campaign Finance and Fact Checked statements from Washington Post’s APIs.
  • Latest News courtesy of search queries to NPR’s Story Api.
  • A simple GeoIP lookup on the homepage to populate the Congressional candidates when a user loads the page

Bootstrap for UI goodness.

I used this opportunity to check out Twitter’s Bootstrap framework. It let us get a clean design from the start, and we were able to use its classes and responsive grid to make the site look really nice on tablets and smartphones too. I found it a lot more feature filled than Skeleton, which is just a responsive CSS framework and lacks the advanced UI elements like navigation, drop downs, modals found in Bootstrap.

Improvements that we could make

We’ve already talked about a number of features we could add or rework to make the site better. Of course, given the shelf life this app will have after November 6th, we may not get to some of these.

  • Re-work the state navigation on the homepage so that it plays nice with the browser’s history. We did a simple ajax query on load, but a better way to do it would be to change the hash to contain the state “http://candidata.us/#VA”, and then pull in the list of candidates. This would also only initiate the geoip lookup if the hash is missing.
  • Add a simple way to navigate to opponents from a candidate’s page.
  • Allow users to navigate to other state races from a candidate’s page.
  • Get more candidate information, ideally something that can provide us Photos of each candidate. Other apps at the hackathon had this, but we didn’t find the API in time. Sunlight provides photos for Members of Congress.
  • Pull in statements made by a candidate via WaPo’s Issue API, maybe running it through the Trove API to pull out categories, people, and places mentioned in the statement.
  • Use the Trove API to organize or at least tag latest news stories and fact checks by Category.

Overall, I’m very happy with what we were able to build in 24 hours. The hackathon also exposed me to some cool ideas and approaches, particularly the visualizations done by some teams. I wish I’d had spent a little more time meeting other people, but my energy was really focused on coding most of the time.

Please check out CandiData.me and let me know what you think either via email or in the comments below.

What’s in your Project Management toolbox?

Matthew at DogStar describes his PM toolbox today, The Project Management Tool Box | Opensource, Nonprofits, and Web 2.0.  It’s a detailed and well organized list, and I think reflects a very practical approach. The first thing that strikes me, is the overwhelming amount of tools available to the would-be PM.  Certainly, there is no lack of tools out there.

You see, the general feeling is, there is no silver bullet. There is no grail of a tool that does everything a single Web Producer, Project Manager, Product Manager, or Content Manager might need or want. There is clearly a gap that is filled with a series of different products. This walked hand in hand with a desire to review processes at work and engage in course corrections. It is an excellent habit to follow – look what you are doing with a critical eye, analyse why you are doing it, and make changes as needed. I have worked across four different shops with a wide variety of different ways of practicing project management. I have used these methodologies and tools across ~ 50 different Drupal projects and another 25 or so custom PHP MySQL projects.

I could not agree more that its important to not be seduced into picking the one right tool for every situation.  It is a difficult tempation to resists, especially when you have tool makers pushing to sell you a solution. The best tool for the job isn’t the one that has the most features, its the one that you, and your team, end up using the most.

As I read the article, a thought that struck me is that sometimes, you don’t need ONE tool, you just need to make sure everyone has the right tools (and skills) to be productive and responsible. At work, we’re a tiny team of 3 who deal with day to day management of our Drupal site, unexpected requests on tight deadlines, and long term projects to build new features. Here’s a secret – we don’t have a central bug/ticket tracking tool. We can be productive simply with email, IM, code bearing, and face to face conversations. For big projects we use a whiteboard to wireframe, capture tasks, and track progress.  This works better than a more sophisticated technical solution that would impose a greater burder on our time.

What’s your experience with tools and grappling with finding the perfect tool?

Why I switched from GoDaddy

You would think something named “Domain Name Registration” would be boring service that the overwhelming majority of Internet users need not think about. However, GoDaddy thinks that may not be good for business, and their latest media blitz has caused me to rethink how I spend my money. I must confess I tolerated their titillating super bowl ads, mostly because I was too lazy to switch, I thought the process would be tedious and cumbersome (boy was I wrong, more on that further down).

If you aren’t aware, when they aren’t planning the next “too hot for TV” ad for the super bowl, CEO Bob Parsons likes to hunt and kill elephants. I’m not a die-hard PETA supporter, another group famous for using sex and controversy to drum up publicity, and more importantly donations. But, upon reflection, I’d rather not have the money I spend on DNS go toward stuff like this.

I’d chosen GoDaddy back when they were a cheap option and others spoke well of the service. Over the years, the company started offering other services, I suppose JUST being a registrar doesn’t make it rain enough to pay women to bounce around, I mean spokes model, for you on TV. As they added more services, they pioneered the art of making the checkout page a confusing mess of addons to trick you into purchasing more stuff. Furthermore, adding the “private” registration option which hides your contact details from WHOIS searches, adds $10/year to the cost, in the end I was really paying $20 to $25 per year for each domain.

Looking for a new registrar, I found Hover (that’s my affiliate link now, save 10%), which is recommended by Leo Laporte of TWIT fame. Their website, from the outset, is the antithesis of godaddy – clean, spartan, and useful. That was enough for me to sign up, and 2 nights ago I began transferring my domains over to them. Their pricing, starting at $15 for the most common TLDs (.com, .net, .org) includes private registration, which makes them more affordable too!

The entire process was very straightforward, and took about 2 hours to do over two nights. Hover provides a thorough step-by-step guide for transferring from godaddy to hover. Each transfer costs $10, and completed in about an hour. It generates a lot of email and confirmations, and godaddy will also send you emails asking you to reconsider, so be prepared to check your inbox frequently. One pleasant surprise, if you are using a 3rd party for managing the actual DNS records (yes, confusing lingo), hover automatically picks up and uses the same name servers.

I’m glad I’ve switched, not only will I save money in the long run, but I know I sent the most powerful message a customer can send by taking my money elsewhere.

Update: Use my affiliate link below to save 10%

Gmail Adds Nested Labels and Message Preview

Google can now render labels hierarchically.  I've been using a naming convention to get labels to sort the way I want, but this will further declutter my labels list.  You have to go to the Google Labs tab in settings to enable it for your account.

Nested Labels is just a cosmetic change that lets you create labels which are displayed hierarchically. If you enable this experiment and create a label like Mailing-Lists/Linux, you'll notice that Linux is displayed as a subfolder of Mailing-Lists. Unfortunately, all the other places that let you interact with labels show the label as Mailing-Lists/Linux

Gmail Adds Nested Labels and Message Preview

Duh – cheaper to outsoure email services.

Will we see more organizations outsourcing email services? Having run my own personal mail server for a few years, I’d be quite wary of purchasing and operating your own mail servers. Yes, if you own your mail servers, you can make them as secure and redudant as you want if price is no limit. But there’s always a budget, and it will always not be enough to provide the uptime, redundancy, and backups that someone like Google, Yahoo, or Microsoft can provide. Plus, you can’t really put a price on being able to sleep at night, free of worries that your server will be hacked, relay spam, or hard drive will fail.

If you are a non-profit, and even if you are not, Non-Profit Tech Blog wrote a three part series on setting up your organization with Google Apps for your Domain.

Have you outsourced email servers? What led to that decision? Any positive/negative feedback about email hosting providers?

Report: Gmail about one-third as expensive as hosted e-mail

Despite the confusion, a lot of companies realize that e-mail has become expensive for two simple reasons: spam and malware. Nearly half of those surveyed were evaluating off-site solutions because e-mail costs had risen, while another 30 percent were performing the evaluation as part of an upgrade or service consolidation process. Fully 85 percent of these companies were leaning towards moving some of the services off-site.

Google Calendar Launches

Google Calendar has launched if you want to check it out. There is also an overview of the features available. Will this be a compelling calendaring solution? After looking it over and checking out the website, I think it might be for me. I’ve setup DAV on apache before so that I can have a portable read/write calendar available online but it hasn’t become critical for me. Of course, I was also the kind of person all through school and college that kept track of due dates pretty much in my head (and that’s why professors also give you a syllabus). This might just stick, at least I’ll try it out and report on if it does in the near future. This summer is full of travel and events so its as good a time as any.

Some observations on the funcitonality

  • Drag in grid to create events (ala outlook)
  • You can subscribe to any public ical/ics file (doesn’t handle stuff behind a password protection)
  • You can invite friends to view your calendar (even if it is not public)
  • You can search through other public calendars
  • You can invite people to events in your calendar, but I don’t see an option for scheduling the time/date conflicts
  • Event reminders via email or sms(to cell phone)
  • Can import a calendar from iCal or csv export from MS Outlook
  • Integrates with gmail
  • You can invite people via email to an event and they can reply if they will attend or not, ala evite but without the spam
  • You can subscribe to your calendar to view it via Mozilla’s Calendar app or Apple’s iCalendar.
  • Event organizers can embed a little button on their own pages to prompt users to add an event to the user’s own google calendar ()

Som things Google did not find essential, for this release and possible any other:

 

  • Scheduling free/busy for an event across all participants
  • Synchronizing calendars with other devices. Of course, if you have iCalendar on apple, iSync should take care of synchronizing with your cell phone or PDA.

 

Fun Saturday night

Bizarre. My self-signed SSL certificate I use to access my email using IMAP over SLL expired last week. The real significance is that we’ve been on this server for one year now. Compared to how painful working on the old Redhat 7.2 install we had back then, working with a Debian system has been a breeze.

So, I figured I’d just generate a new certificate, the instructions are readily googleable and I’d be playing a little Warcraft in no time. But no, after following the instructions Cyrus’s imap server wouldn’t start up. Eventually I figured out it was complaining about a missing lib related to the net-snmp package. A year ago, Cyrus 2.2 wasn’t in Debian testing yet, but it is now so it ended up being time to upgrade to the official packages.

The install was easy as usual, except for some disconcerting messages about my Cyrus databases needing an upgrade from DBD 3.2 to 4.2. To get the upgrade tool I installed the db4.2-utils package and ran the databases through the utility. I had my fingers crossed, with the spectre of data loss looming nearby (although I do have nightly backups that in theory I can recover from). The conversion worked well to, and all my email is in its place.

Feeling invincible, at this point I decided it was time to upgrade the kernel from the old stock 2.4 kernel to a newer 2.6 linux kernel. The only real reason is that I’d read Mysql performs noticeably better on the 2.6 kernels. Can you tell a difference? As far as I can tell, nothing is massively broken. If you notice anything odd though, please let me know!

Pointers: running your own mail server

There are a bunch of reasons why you might decide that running your own mail server is something you want to do. You have a lot of free time and enjoy spending a lot of time at the command line reading howto guides and installation manuals. Hopefully, you know you are running a mail server and don’t have a Windows machine that’s been turned into a spam zombie.

Ok, so that may not sound like good reasons. There are some real benefits, you can give yourself unlimited email aliases, give your friends and family easier-to-remember addresses, and set up mailing lists to keep in touch with people.

If you’ve got a linux server, Postfix is one of the more popular mail transports. One of the things about mail server jargon is that there are a number of lego blocks that go into the mail chain. I won’t attempt to write a complete guide to setting up your server, instead I’ll point you at some useful links that I found helpful, and to boot not horrendously diffult to install.

  • Postfix takes care of receiving incoming mail and routing it to a local destination. I found the Postfix Anti-UCE Cheat Sheet useful for making sure I had configured the server correctly to make sure it is being used for good and not evil.
  • Amavisd-new is a perl script for plugging in virus scanners and spam blocks into the delivery chain. I use Clam Anti-virus, a Free virus scanner, and Spamassassin, to protect users from unwanted or dangerous mail messages.
  • Once you’re system is running, you might find that you’re looking at the mail log to make sure nothing is out of sorts. Download pflogsumm and schedule it to send you a report on how many messages are being delivered/sent/blocked and other useful metrics.

Tweaking Firefox & Thunderbird in Linux

Both on my Debian PC’s at home and work, I’ve been annoyed that my Firefox and Thunderbird installations didn’t launch each other when I clicked on a link or email address. For example, at work, Evolution would come up if I clicked on a mailto link on a web page. Inside each program’s preferences there were no options for controlling this behavior. These preferences are set in the gnome-control-center. Each application will respect the preferred browser and preferred email programs defined in there. I forgot where I found this tip recently but I can’t take credit for discovering it on my own.