node.js - What is the right way to deal with Mongodb connections?

Question

I try node.js with mongodb (2.2.2) together using the native node.js drive by 10gen.

At first everything went well. But when coming to the concurrency benchmarking part, a lot of errors occured. Frequent connect/close with 1000 concurrencies may cause mongodb reject any further requests with error like:

Error: failed to connect to [localhost:27017]

Error: Could not locate any valid servers in initial seed list

Error: no primary server found in set

Also, if a lot of clients shutdown without explicit close, it'll take mongodb minutes to detect and close them. Which will also cause similar connection problems. (Using /var/log/mongodb/mongodb.log to check the connection status)

I have tried a lot. According to the manual, mongodb don't have connection limitation, but poolSize option seems to have no effects to me.

As I have only worked with it in node-mongodb-native module, I'm not very sure what eventually caused the problem. What about the performance in other other languages and drivers?

PS: Currently, using self maintained pool is the only solution I figured out, but using it can not can not solve the problem with replica set. According to my test, replica set seems take much less connections then standalone mongodb. But have no idea why this happens.

Concurrency test code:

var MongoClient = require('mongodb').MongoClient;

var uri = "mongodb://192.168.0.123:27017,192.168.0.124:27017/test";

for (var i = 0; i < 1000; i++) {
    MongoClient.connect(uri, {
        server: {
            socketOptions: {
                connectTimeoutMS: 3000
            }
        },
    }, function (err, db) {
        if (err) {
            console.log('error: ', err);
        } else {
            var col = db.collection('test');
            col.insert({abc:1}, function (err, result) {
                if (err) {
                    console.log('insert error: ', err);
                } else {
                    console.log('success: ', result);
                }
                db.close()
            })
        }
    })
}

Generic-pool solution:

var MongoClient = require('mongodb').MongoClient;
var poolModule = require('generic-pool');

var uri = "mongodb://localhost/test";

var read_pool = poolModule.Pool({
    name     : 'redis_offer_payment_reader',
    create   : function(callback) {
        MongoClient.connect(uri, {}, function (err, db) {
            if (err) {
                callback(err);
            } else {
                callback(null, db);
            }
        });
    },
    destroy  : function(client) { client.close(); },
    max      : 400,
    // optional. if you set this, make sure to drain() (see step 3)
    min      : 200, 
    // specifies how long a resource can stay idle in pool before being removed
    idleTimeoutMillis : 30000,
    // if true, logs via console.log - can also be a function
    log : false 
});


var size = [];
for (var i = 0; i < 100000; i++) {
    size.push(i);
}

size.forEach(function () {
    read_pool.acquire(function (err, db) {
        if (err) {
            console.log('error: ', err);
        } else {
            var col = db.collection('test');
            col.insert({abc:1}, function (err, result) {
                if (err) {
                    console.log('insert error: ', err);
                } else {
                    //console.log('success: ', result);
                }
                read_pool.release(db);
            })
        }
    })
})

Answer 1

Since Node.js is single threaded you shouldn't be opening and closing the connection on each request (like you would do in other multi-threaded environments.)

This is a quote from the person that wrote the MongoDB node.js client module:

“You open do MongoClient.connect once when your app boots up and reuse the db object. It's not a singleton connection pool each .connect creates a new connection pool. So open it once an[d] reuse across all requests.” - christkv https://groups.google.com/forum/#!msg/node-mongodb-native/mSGnnuG8C1o/Hiaqvdu1bWoJ