node.js http.get hangs after 5 requests to remote site

Question

I'm writing a simple api endpoint to determine if my server is able to reach the internet. It works great, but after 5 requests (exactly 5, every time) the request hangs. Same thing happens when I switch Google to Hotmail.com, which makes me think that this is something on my end. Do I need to close the http.get requests? I was under the impression that this function closes the requests automatically.

// probably a poor assumption, but if Google is unreachable its generally safe to say     that the server can't access the internet
// using this client side in the dashboard to enable/disable internet resources

app.get('/api/internetcheck', function(req, res) {
console.log("trying google...");
    http.get("http://www.google.com", function(r){
        console.log("Got status code!: " +r.statusCode.toString());
        res.send(r.statusCode.toString());
        res.end();
        console.log("ended!"); 
    }).on('error', function(e) {
        console.log("Got error: " + e.message);
    });
});

Answer 1

Here's the reason of the "exactly 5": https://nodejs.org/docs/v0.10.36/api/http.html#http_agent_maxsockets

Internally, the http module uses an agent class to manage HTTP requests. That agent will, by default, allow for a maximum of 5 open connections to the same HTTP server.

In your code, you don't consume the actual response sent by Google. So the agent assumes that you're not done with the request, and will keep the connection open. And so after 5 requests, the agent won't allow you to create a new connection anymore and will start waiting for any of the existing connections to complete.

The obvious solution would be to just consume the data:

http.get("http://www.google.com", function(r){
  r.on('data', function() { /* do nothing */ });
  ...
});

If you run into the problem that your /api/internetcheck route is called a lot, so you need to allow for more than 5 concurrent connections, you can either up the connection pool size, or just disable the agent completely (although you would still need to consume the data in both cases);

// increase pool size
http.globalAgent.maxSockets = 100;

// disable agent
http.get({ hostname : 'www.google.com', path : '/', agent : false }, ...)

Or perhaps use a HEAD request instead of GET.

(PS: in case http.get generates an error, you should still end the HTTP response by using res.end() or something like that).

NOTE: in Node.js versions >= 0.11, maxSockets is set to Infinity.