python - Tornado "error: [Errno 24] Too many open files" error

Question

I've worked with Tornado quite a bit, but this is the first time I've run into this sort of error. I've been working on a very basic URL shortener. URLs are put into the database by a different application, this one just reads the URLs from a MongoDB store and redirects the clients. After I'd written the basic code I set up a simple 'Siege' test against it, after about 30 seconds of running siege (run with siege -c 64 -t 5m -r 1 http://example.com/MKy against 4 application threads) I started getting 500 responses. Looking in the error log I saw this;

ERROR:root:500 GET /MKy (127.0.0.1) 2.05ms
ERROR:root:Exception in I/O handler for fd 4
Traceback (most recent call last):
  File "/opt/python2.7/lib/python2.7/site-packages/tornado-2.1-py2.7.egg/tornado/ioloop.py", line 309, in start
  File "/opt/python2.7/lib/python2.7/site-packages/tornado-2.1-py2.7.egg/tornado/netutil.py", line 314, in accept_handler
  File "/opt/python2.7/lib/python2.7/socket.py", line 200, in accept
error: [Errno 24] Too many open files
ERROR:root:Uncaught exception GET /MKy (127.0.0.1)
HTTPRequest(protocol='http', host='shortener', method='GET', uri='/MKy', version='HTTP/1.0', remote_ip='127.0.0.1', body='', headers={'Host': 'shortener', 'Accept-Encoding': 'gzip', 'X-Real-Ip': '94.23.155.32', 'X-Forwarded-For': '94.23.155.32', 'Connection': 'close', 'Accept': '*/*', 'User-Agent': 'JoeDog/1.00 [en] (X11; I; Siege 2.66)'})
Traceback (most recent call last):
  File "/opt/python2.7/lib/python2.7/site-packages/tornado-2.1-py2.7.egg/tornado/web.py", line 1040, in wrapper
  File "main.py", line 58, in get
  File "main.py", line 21, in dbmongo
  File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/connection.py", line 349, in __init__
  File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/connection.py", line 510, in __find_master
  File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/connection.py", line 516, in __try_node
  File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/database.py", line 301, in command
  File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/collection.py", line 441, in find_one
  File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/cursor.py", line 539, in loop
  File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/cursor.py", line 560, in _refresh
  File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/cursor.py", line 620, in __send_message
  File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/connection.py", line 735, in _send_message_with_response
  File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/connection.py", line 591, in __stream
  File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/connection.py", line 200, in get_stream
  File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/connection.py", line 559, in __connect
AutoReconnect: could not connect to [('127.0.0.1', 27017)]

Important (I guess);

error: [Errno 24] Too many open files

The code; (It's very simple)

import tornado.ioloop
import tornado.web
import tornado.escape
import apymongo
import time
import sys

#Useful stuff (Connect to Mongo)
class setup(tornado.web.RequestHandler):
    def dbmongo(self):
        if not hasattr(self, '_dbmongo'):
            self._dbmongo = apymongo.Connection("127.0.0.1", 27017)
        return self._dbmongo
        

#Basic method to lookup URLs from Mongo and redirect accordingly
class expand(setup):
    @tornado.web.asynchronous
    def get(self, url):
        self.mongo = self.dbmongo()
        
        #Lookup the URL
        cursor = self.mongo.rmgshortlinks.links.find_one({'short':url}, self.direct)
    
    def direct(self, response):
        if response == None:
            self.send_error(404)
            self.finish()
            return
        
        link = tornado.escape.url_unescape(response['long'])
        
        #Bounce the client
        self.write("<!DOCTYPE html><html><head><meta charset="UTF-8" /><meta http-equiv="refresh" content="0;URL="+link+""</head><body><a href=""+link+"">Click Here</a></body></html>")
        self.finish();


#Define the URL routes
application = tornado.web.Application([
    (r"/([a-zA-Z0-9]+)", expand)
])

#Start the server
if __name__ == "__main__":
    listening_port = int(sys.argv[1])
    
    if listening_port > 0:
        application.listen(listening_port)
        tornado.ioloop.IOLoop.instance().start()
    else:
        sys.stderr.write("No port specified!")

The dev server I'm using has 8 cores and 64GB memory, running RedHat Enterprise Linux 5 and Python 2.6. I've never had these sorts of issues with Tornado/Async Mongo applications before.

Probably useful information;

[root@puma ~]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31374
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31374
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

(open files is only set to 1024 but I would have thought that's more than enough)

Is Tornado/Apymongo not closing the connections properly? The applications sit behind NGINX but connect using HTTP, Apymongo should be connecting via TCP but might be using sockets. Even so it should be sharing/pooling connections shouldn't it?

Edit

As suggested, moved the app onto one of our testing servers with a max open files limit of 61440, same error after about 30 seconds of running in siege.

Answer 1

Very simple, the RequestHandler object is instantiated for every request. Which means that the cached object that you're saving is on the RequestHandler (e.g. expand) object.

If you were to add a simple "print 'CREATED!'" to the dbmongo(...) function you would see that it's created on every GET request.

What you need to do is attach the handler to the class object, or a "global" as needed, though the best case is to put it on the Tornado Application object.

Simple:

class setup(tornado.web.RequestHandler):
    @classmethod
    def dbmongo(cls):
        if not hasattr(cls, '_dbmongo'):
            cls._dbmongo = apymongo.Connection("127.0.0.1", 27017)
        return cls._dbmongo

Second approach is just to make it a global in your file:

dbmongo_connection = None
def dbmongo():
    if not dbmongo_connection:
        dbmongo_connection = apymongo.Connection("127.0.0.1", 27017)
    return dbmongo_connection

Both of these have the same problem which is that if you have lots of classes that want to use the DB connection, it's harder to share it. Since the DB is a shared entity you probably want one for your whole application.

class MongoMixin(object):
    def mongodb(self):
        if not hasattr(self.application, 'mongodb'):
            self.application.mongodb = apymongo.Connection(self.application.settings.get("mongohost", "127.0.0.1"), 27017)
        return self.application.mongodb

class expand(tornado.web.RequestHandler, MongoMixin):
    def get(self):
       db = self.mongodb()