HTTP... something

A survey of HTTP client-side libraries. Too much choice. And my 2 cents.

Note that these are in no particular order and that the list (still) isn’t complete. If you have something that deserves a mention here, let me know. I will include it for novelty, not for repetition.

http (2011), nestful (2010), righthttp (2010), simplehttp (2007), http_request.rb (2009), plain_http (2011)

I have a hard time seeing the differences between these libraries. There sure are some, but they might have well copied the API from one another.

  Http.get 'http://example.com'             # http
  Nestful.get 'http://example.com'          # nestful
  HTTP.get 'http://example.com'             # righttp
  SimpleHttp.get "http://www.example.com"   # simplehttp
  HttpRequest.get('http://www.github.com')  # http_request.rb
  PlainHTTP.get('http://www.google.com')    # plain_http

nestful has probably the most features geared towards JSON.

It looks as if everyone is scratching the same itch. And an itch that itches so bad, you can’t be bothered with google anymore. Some of these are better than others and nestful probably deserves its own category based on features, but there.

Typhoeus (2009)

Typhoeus is a mythical greek god with 100 fire breathing serpent heads.¹ In the case of typhoeus, a head stands metaphorically for a connection – this library allows more than one connection be made at the same time. The documentation at github has a nice ‘getting started’ tutorial that shows you how to do requests.

Typhoeus really will do parallel requests! If you’re someone who calls 100 backend services on every request (thanks, SOA!), you will need this in one of its forms. This library looks very feature complete and advanced, but as you might have guessed, this will also work:

  Typhoeus::Request.get('http://example.com')

An interesting variation here is that you can use Typhoeus as a DSL in your classes, giving them HTTP call perks. See this post from Paul Dix for an in-depth explanation.

httpadapter (2010)

This isn’t a HTTP speaking library, but rather it ‘translates HTTP request and response objects for various clients into a common representation’. The idea seems to be to create a client-side rack.²

http_connection (2009)

This library by RightScale.com seems to attempt to do reliable http connections that do connection retries. It doesn’t have as many features as others, but reliability seems to be something that others don’t address.

Is it something that needs to be handled by your HTTP library?

httpclient (2007)

This seems to be the most complete client like implementation of HTTP. It has things like cookie support. (storing cookies from one request to the next, like a browser does)

The samples are very informative and should be easy enough to pick up.

  client = HTTPClient.new
  client.get 'http://example.com'

httparty (2008)

I’ve never heard of this before today and it seems I am late to the .. well let’s not go there. This library is similar to Typhoeus in that it allows you to create special purpose classes that encapsulate HTTP functionality.

Here’s the simple example from the project page:

  class Twitter
    include HTTParty
    base_uri 'twitter.com'
    basic_auth 'username', 'password'
  end

  Twitter.post('/statuses/update.json', :query => 
    {:status => "It's an HTTParty and everyone is invited!"})

It’s been around for some time and even though it doesn’t seem to be as heavy as Typhoeus is on parallelization, it seems to allow terse special purpose HTTP request classes to be written in just a few lines.

net-https-wrapper (2009)

This library is special in its minimalism. Its (rather short) gemspec has the same amount of lines as its implementation. The author doesn’t seem to think we’d need #get, ever, so it only speaks #post or #put. Is it art?

http_spew (2011)

If one of these libraries gets outlawed, it’s this one. This seems to be the only library I’ve looked at that doesn’t bother with the responses that are sent by the server. Instead, it attempts to generate as much traffic as possible without reading things back. As the author rightfully remarks, this may get you banned from the internet. Or wait. No. Daily business.

fast_http (2010)

Art? Or just plain fast! HTTP? I cannot really tell from the documentation.

net-http-persistent (2010), persistent_http (2010)

These libraries are not about the API, but about making HTTP connections persistent. This is also something a browser does and probably should be a feature of the base library.

persistent_http will allow pooling connections instead of keeping one connection per thread.

net-http-pipeline (2010)

Allows HTTP/1.1 request pipelining:

  require 'net/http/pipeline'

  Net::HTTP.start 'localhost' do |http|
    req1 = Net::HTTP::Get.new '/'
    req2 = Net::HTTP::Get.new '/'
    req3 = Net::HTTP::Get.new '/'

    http.pipeline [req1, req2, req3] do |res|
      puts res.code
      puts res.body[0..60].inspect
      puts
    end
  end

HTTPal (2007)

Similar to httpclient, this keeps cookies and referrers for you. A simple library that seems to either do its job or is abandoned – no releases since the initial few.

httpi (2010)

A common interface to several HTTP backends: This library speaks curb, httpclient and net-http. The idea is that it gives you a simple interface to remember and allows all of these backends to be swapped out, depending on your needs.

  HTTPI.get 'http://www.example.com'    # will work

Conclusion

I’ve used httpclient (for its cookie caps) and httpi (by accident) in the past. No gripes with those. Also, I must have used net-http (stdlib) at some point. The fact that I didn’t write one of these libraries must mean that it didn’t bother me as much as it did others.

I’ve added Typhoeus and HTTParty to the list of libraries I would use. The API idea behind these makes sense and used in the right place, it will simplify things. Typhoeus certainly comes in handy when making hundreds of requests to backends.

Interesting to know about: net-http-persistent and net-http-pipeline.

Write another one?

No.

Perhaps we should all clean up our respective acts and:

• Use google before using one of the many project skeleton generators that exist. (That is a list to be made another day…)

• Merge some of these? Make better libraries instead of more?

And a parting note

I could make these kind of lists for many problems we Rubyists solve. The authors of these libraries are not to blame. (or rather, they are only to blame for their part of the problem) The Ruby community as a whole is currently experiencing a severe shortness of attention span. If it is not new, it is probably crap, right?

No. We should stop the proliferation of new libraries and start working good ideas into the libraries we already have. Also, a good telling sign whether you need to release a library is: Is the library bigger than the Gemspec?

¹ http://www.pauldix.net/2009/05/breath-fire-over-http-in-ruby-with-typhoeus.html

² http://httpadapter.rubyforge.org/api/