Full offline mode (createSubscribeQuery)

[zzph]

Hi,

I’m trying to create a fully offline experience, not only storing the ops to send- but reading data from local storage (indexdb) when there’s no connection to the server.

My guess is something like:

1. for every item, to store the following in local storage:

{
data,
src: connection.id,
v: version,
seq: connection.seq,
c: collection,
d: id,
}

2. When unable to connect to the server, read items from local storage, recreating each one as a “doc” with normal properties (submitOp etc)

Is this plausible- can i simply recreate each “doc”?

Do you have any suggestion on implementation?

[alecgibson]

If you want full offline support, including queries (which are usually handled by the DB, so not very offline-friendly!), then I have wondered in the past about running an instance of the ShareDB server in the browser (since it is, after all, JavaScript). It would have query support thanks to sharedb-mingo-memory.

The tricky part is that you'd still need to figure out how to sync your in-memory DB state with indexdb, which isn't a solved problem as far as ShareDB goes, and I sadly don't have any time to work on that personally.

[zzph]

Hmm, interested. I'm keen to spend time figuring it out.

Could elaborate how I'd "mock" the results coming through on connection from indexDB (as opposed to remotely)? Where is a good starting point?

I looked into custom middleware (db adapter) but I'm unclear where to "inject" it

import { Connection } from 'sharedb/lib/client'
...
const connection = new Connection(Socket, someCustomDb?)

[alecgibson]

You could probably start by just copy-pasting and/or forking sharedb-mingo-memory. You can then extend its functionality to actively write to indexdb every time it writes to memory (and to fetch from indexdb on initialisation, or if something isn't already loaded into memory).

You'd register it on your "backend" (which is actually running on the server) like any other DB adapter.

I forgot to mention that if you get this working fully offline, there's still some work to do once you come back online to reconcile your local database with the real database, since your document histories will have diverged. I haven't thought too hard on that yet, but I guess we can cross that bridge once you get your local DB running properly 😅

[zzph]

Thanks- I'm a little confused how I'd run it on the 'backend' though- this would have to be client side (browser) offline mode.

Did you mean to run the server code in my browser (somehow connecting a socket to it)?

  const backend = new ShareDB({
    db: new MyIndexDBFork(),
  })

const webSocketServer = new WebSocket()
  webSocketServer.on('connection', (webSocket) => {
    const stream = new WebSocketJSONStream(webSocket)
    backend.listen(stream)
  }

  const connection = backend.connect()
  
  //  use as normal `sharedb/lib/client` mode?

[zzph]

FWIW,

I made an alternate approach too, which was to re-created the Doc class all together when offline. Kind of works, except that the submitOp event doesn't seem to update the data (may be my incorrect use of js classes). Is this a bad approach? Any idea why my submitOp isn't 'updating' the data on my object?

export class DocMock {
  constructor(collection, data) {
    this.connection = { canSend: false }
    this.collection = collection
    this.data = data
    this.id = data.id
  }

  submitOp(ops, cb) {
    this.data = getItemWithOps(this.data, ops)
    this.emit('op', ops, true)
    cb(null)
  }

  on(eventType, cb) {
    this[`_on${eventType}`] = cb;
  }

  emit(eventType, a, b) {
    if (this[`_on${eventType}`]) {
      this[`_on${eventType}`](a, b);
    }
  }
}

[alecgibson]

Did you mean to run the server code in my browser

Yes, this is exactly what I mean. If you run the server code in your browser, out-of-the-box you can already call backend.connect() with no arguments and it will create an in-memory StreamSocket and return a Connection instance.

[zzph]

I've had a go, firstly needing to npm install stream as it couldn't be found.

However, I get the error TypeError: Cannot read properties of undefined (reading 'call') at new ServerStream. Any ideas why?

  const backend = new ShareDB()
  const connection = backend.connect()

[alecgibson]

I don't think you want stream (it's last update was 8yrs ago); we're using stream-browserify

[zzph]

I don't think you want stream (it's last update was 8yrs ago); we're using stream-browserify

I tried installing stream-browsrify but still got the same error.

Hmm seems it's still using stream, no?

Eg here:

sharedb/lib/op-stream.js

Line 2 in e29936c

var Readable = require('stream').Readable;

[alecgibson]

Yes, you'll need to include this in whatever you're using to package your JS. For example, if using Webpack 5: https://webpack.js.org/configuration/resolve/#resolvefallback

[zzph]

Hmm I'm using plain js (no bundler), i can't figure it out I thought exports would do it in package.json

[alecgibson]

Unfortunately you currently need some sort of bundler for ShareDB. See #255

[zzph]

As an alternative, can you offer any suggestion on the below example? It work well, except the submitOps method inside the class isn't actually 'changing' the data in the object itself (object is inside a react createContext and it works fine with normal ShareDB objects)

Is there another event I need to emit other than 'submitOp'? Is there some "special" way the Doc is updated?

export class DocMock {
constructor(collection, data) {
this.connection = { canSend: false }
this.collection = collection
this.data = data
this.id = data.id
}

submitOp(ops, cb) {
this.data = getItemWithOps(this.data, ops)
this.emit('op', ops, true)
cb(null)
}

on(eventType, cb) {
this[_on${eventType}] = cb;
}

emit(eventType, a, b) {
if (this[_on${eventType}]) {
this[_on${eventType}](a, b);
}
}