Skip to content
This repository has been archived by the owner on Oct 23, 2023. It is now read-only.

Read MongoDB oplogs from a sharded cluster

Notifications You must be signed in to change notification settings

WW-Digital/mongo-oplog-reader

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

75 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

mongo-oplog-reader

Read MongoDB oplogs from a sharded cluster.

Build Status

  • Supports multiple processes for high availability
  • Emits event after document has been written to the majority of members of the shard's replica set
  • Resumes from the last emitted event of the replica set (in the case of process crash/stop/restart)
  • Ability to add/remove shards/hosts without restarting

Notes:

  • The order of the oplog events is not guaranteed to be strictly chronological if there are multiple workers per oplog

Requirements

  • Redis 3.2+

Install

npm i mongo-oplog-reader redis

Usage

import MongoOplogReader from 'mongo-oplog-reader';
import redis from 'redis';

const redisClient = redis.createClient();

const connectionStrings = [
  'mongodb://shard0-primary/local',
  'mongodb://shard0-secondary0/local',
  'mongodb://shard0-secondary1/local',
  'mongodb://shard1-primary/local',
  'mongodb://shard1-secondary0/local',
  'mongodb://shard1-secondary1/local'
];

const reader = new MongoOplogReader({
  redisClient,
  workersPerOplog: 1 // total # of redundant workers per oplog (respected across all processes)
});
reader.filter(oplogDocument => oplogDocument.op !== 'u'); // ignore 'update' operations
reader.setConnectionStrings(connectionStrings);
reader.onEvent(data => {
  // return a promise to apply backpressure on the oplog stream to prevent a 
  // build-up of in-memory stream buffering while performing slower async operations
  return somethingAsync(data);
});
reader.start();

License

MIT