MongoDB Replica Set for Oplog Tailing

What is oplog tailing?

This is the wrong question to ask in the first place. Before we answer this we need to know…

What is a MongoDB Replica Set?

This basically means that you do not run only one MongoDB process but many and every MonogDB process always have the exact same data in their databases. This is quite important for production environments. If you only have one MongoDB process running and it crashes your whole application would not be able to run anymore.

So we want to run multiple MongoDB processes in a replica set and one process of the replica set is the so called primary. All other MongoDB processes are called secondary. Reads and writes from a client typically only go to the primary process of the replica set. There are more options than just primary and secondaries but we will only look at this configuration. We also want to have one primary with exactly two secondaries. This way we would run three MongoDB processes which is a good amount for redundancy. If the primary becomes unavailable the replica set holds an election and one of the secondaries becomes the new primary. So this is good so far, but…

What is oplog tailing?

We now have three MongoDB processes running (I’ll show you how this works in practice in this article, too) and each of these have the same data set, we know that now. But how on earth do the secondaries know exactly what the data set of the primary looks like, sync their own data set and all of this in real time? This is where the oplog comes into play. The oplog is the operations log and is a collection that keeps track of all operations that modifies data. If someone inserts a new document to a collection this operation will stored into the oplog collection. The secondary processes simply copy the changes in the oplog from the primary to their own oplog collection and update their data set. And this oplog is exactly what we need and one of the reasons why we need the replica set because now our Meteor Application or multiple Meteor Application can watch the oplog as well and inform clients that rely on the data immediately.

Convert a Standalone to a Replica Set

1. Shut down the standalone mongod instance.
2. Restart the instance. Use the --replSet option to specify the name of the new replica set.
   For example, the following command starts a standalone instance as a member of a new replica set named rs0. The command uses the standalone’s existing database path of/srv/mongodb/db0:

   mongod --port 27017 --dbpath /srv/mongodb/db0 --replSet rs0
   

   If your application connects to more than one replica set, each set should have a distinct name. Some drivers group replica set connections by replica set name.
   For more information on configuration options, see Configuration File Options and the mongodmanual page.
3. Connect to the mongod instance.
4. Use rs.initiate() to initiate the new replica set:

   rs.initiate()
   

   The replica set is now operational.
   To view the replica set configuration, use rs.conf(). To check the status of the replica set, use rs.status().
5.  Now try inserting some thing into the database.
6. rs0:PRIMARY> db.inventory.insert({item: "ABC1",category:"clothing",details:{model:"14Q3",manufacturer:"MS"}})
7. WriteResult({ "nInserted" : 1 })
   rs0:PRIMARY> use local
   switched to db local
   rs0:PRIMARY> db.oplog.rs.find()
   { "ts" : Timestamp(1423633740, 1), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "o" : { "msg" : "initiating set" } }
   { "ts" : Timestamp(1423635276, 1), "h" : NumberLong("-6734526260774436847"), "v" : 2, "op" : "i", "ns" : "inventory.inventory", "o" : { "_id" : ObjectId("54daf34be24350068efb399f"), "item" : "ABC1", "category" : "clothing", "details" : { "model" : "14Q3", "manufacturer" : "MS" } } }
   rs0:PRIMARY> db.oplog.rs.find()
   { "ts" : Timestamp(1423633740, 1), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "o" : { "msg" : "initiating set" } }
   { "ts" : Timestamp(1423635276, 1), "h" : NumberLong("-6734526260774436847"), "v" : 2, "op" : "i", "ns" : "inventory.inventory", "o" : { "_id" : ObjectId("54daf34be24350068efb399f"), "item" : "ABC1", "category" : "clothing", "details" : { "model" : "14Q3", "manufacturer" : "MS" } } }

Create oplog User

then the first member should be the master and this is our mongod process running on port 27017 we connected to.

We do not really need to, but we check if there are users already:

show users

For a clean mongodb there should not be one. This command shows the users depending on which database you are using. We want to create our user in the special ‘admin’ database. So we switch to this database by:

msrs:PRIMARY> use admin

Again, there will be no users, too. Now we add the oplogger into the admin database.

msrs:PRIMARY> db.addUser({user:'oplogger',pwd:'YOUR_PASSWORD',roles:[],otherDBRoles:{local:["read"]}})

If you type now ‘show users’ we will see the oplogger. The oplogger has the right to read everything that is written to the local database. Every mongod instance has its own local database and in this database there is the oplog in the local.oplog.rs collection. In this collection is every operation logged and if you inspect it (e.g. with RoboMongo) you will see a document that looks like this:

{
    "ts" : Timestamp(1389778589, 1),
    "h" : -6184987159182693880,
    "v" : 2,
    "op" : "i",
    "ns" : "admin.system.users",
    "o" : {
        "_id" : ObjectId("52d6569ddfa5a7b581dbe963"),
        "user" : "oplogger",
        "pwd" : "552d22594bb530f7c56c087830f9e08d",
        "roles" : [],
        "otherDBRoles" : {
            "local" : [ 
                "read"
            ]
        }
    }
}

This was the insert (“op”:”i”) of our ‘oplogger’ user.

Sudoku solver

report: report
Code : Downaload