node.js - Using node ddp-client to insert into a meteor collection from Node

Question

I'm trying to stream some syslog data into Meteor collections via node.js. It's working fine, but the Meteor client polling cycle of ~10sec is too long of a cycle for my tastes - I'd like it be be ~1 second.

Client-side collection inserts via console are fast and all clients update instantly, as it's using DDP. But a direct MongoDB insert from the server side is subject to the polling cycle of the client(s).

So it appears that for now I'm relegated to using DDP to insert updates from my node daemon.

In the ddp-client package example, I'm able to see messages I've subscribed to, but I don't see how to actually send new messages into the Meteor collection via DDP and node.js, thereby updating all of the clients at once...

Any examples or guidance? I'd greatly appreciate it - as a newcomer to node and Meteor, I'm quickly hitting my limits.

Answer 1

Ok, I got it working after looking closely at some code and realizing I was totally over-thinking things. The protocol is actually pretty straight forward, RPC ish stuff.

I'm happy to report that it absolutely worked around the server-side insert delay (manual Mongo inserts were taking several seconds to poll/update the clients).

If you go through DDP, you get all the real-time(ish) goodness that you've come to know and love with Meteor :)

For posterity and to hopefully help drive some other folks to interesting use cases, here's the setup.

Use Case

I am spooling some custom syslog data into a node.js daemon. This daemon then parses and inserts the data into Mongo. The idea was to come up with a real-timey browser based reporting project for my first Meteor experiment.

All of it worked well, but because I was inserting into Mongo outside of Meteor proper, the clients had to poll every ~10 seconds. In another SO post @TimDog suggested I look at DDP for this, and his suggestion looks to have worked perfectly.

I've tested it on my system, and I can now instantly update all Meteor clients via a node.js async application.

Setup The basic idea here is to use the DDP "call" method. It takes a list of parameters. On the Meteor server side, you export a Meteor method to consume these and do your MongoDB inserts. It's actually really simple:

Step 1: npm install ddp

Step 2: Go to your Meteor server code and do something like this, inside of Meteor.methods:

Meteor.methods({

'push': function(k,v) { // k,v will be passed in from the DDP client.
    console.log("got a push request")
    var d = {};
    d[k] = parseInt(v);
    Counts.insert(d, function(err,result){ // Now, simply use your Collection object to insert.
        if(!err){ 
            return result
            }else{
            return(err)
            }
        });
     }
  });

Now all we need to do is call this remote method from our node.js server, using the client library. Here's an example call, which is essentially a direct copy from the example.js calls, tweaked a bit to hook our new 'push' method that we've just exported:

ddpclient.call('push', ['hits', '1111'], function(err, result) {
console.log('called function, result: ' + result);
})

Running this code inserts via the Meteor server, which in turn instantly updates the clients that are connected to us :)

I'm sure my code above isn't perfect, so please chime in with suggestions. I'm pretty new to this whole ecosystem, so there's a lot of opportunity to learn here. But I do hope that this helps save some folks a bit of time. Now, back to focusing on making my templates shine with all this real-time data :)