Scratchpad:Remote Sync

= Remote Sync = It has multiple times been suggested to add some kind of remote synchronization feature to OpenLP in order to be able to keep multiple installations of OpenLP in sync. The suggested solution has so far been to move the OpenLP data folder to a synchronized folder (Dropbox, Google Drive, OneDrive, etc). While this can work fine, there are some problem with this approach:
 * If multiple users uses OpenLP at the same time the behavior is undefined... (expect OpenLP to hang and/or crash)
 * If conflicts arises there is no way to handles this.

On this page one or more solutions will be described, and maybe in time implemented...

= Solutions = So far 2 solutions has been proposed:
 * Synchronization using remote database accessed using REST webservices.
 * Synchronization using a synchronized folder (Dropbox, Google Drive, OneDrive, etc).

Below the 2 solutions is described in more detail. If both (or more) solutions should be implemented it would an advantages to implement a general "Remote Sync" plugin with multiple backends.

General sync features
Some general features are needed regardless of which solution is chosen.

Conflict handling
Conflicts could be handled in multiple ways, and we could let the user decide between multiple ways to resolve them:
 * Latest change is kept. (For webservice: If we know the current time of the server and the time of the latest remote change, and the local time and time of latest local change, then it should be easy to calculate which is newer.)
 * Manually resolve using a window similar to the one used when handling duplicated songs.
 * Local/remote always "wins".

Background thread to handle synchronization
When a change is made to a song/author/etc, the change is pushed to a sync-queue that the background thread look at and then push to the "sync-backend".

The thread also looks for remote changes and merge them into the local OpenLP DB. If conflicts arise the user is asked to take action, or perhaps a preselected way to resolve the conflict is used.

When syncing remote entries we need to make sure that ingoing and outgoing syncing handles conflict. An example of the problem could be that we have changes to an entry in the outgoing queue waiting to be synced. We then receive an ingoing sync which overwrite the local entry. The entry in the outgoing queue is now push to the remote, which means that the old version of the entry (which locally has been overwrite) is distributed to remote sync.

Global entry id
Since songs/authors/etc can be added by multiple OpenLP instances a global id for entries are needed to identify entries. Assigning a UUID to each entry which is saved both locally and remote would solve this.

Synchronization using remote database accessed using REST webservices
The remote (server side) implementation would be accessible through a REST interface which would fairly closely resemble the DB layout of OpenLP, probably using some simple existing framework. The reference implementation of the webserice will most likely be done in Python. There would probably also need to be some kind of graphical user interface to handle users (login/password).

At regular intervals the webservice must be asked if any changes has occurred. Perhaps this could be done using websockets?

We should probably have a look at U1DB, which looks to support what we need, and it is written in Python2. Unfortunately it seems to be a largely abandoned project, but very interesting!

Synchronization using files
This is not the same approach as placing OpenLPs datafolder in a synchronized folder! Instead files that can be used to synchronized multiple OpenLP databases is placed in a synchronized folder and OpenLP then read these files.

On each change to songs/authors/etc the updated entry is placed in the sync'ed folder like this: /tablename/uuid.json. While writing to the file it should be named /tablename/uuid.json.tmp and then renamed after writing is finished to avoid syncing partial files.

To get notified about file changes a crossplatform python library exists: https://pypi.python.org/pypi/watchdog though it is not available as packages on all major Linux distributions.

Synchronization using a web API
Dropbox, Google Drive, etc. offers web APIs which would make it possible for us to sync OpenLP songs and other data. The APIs can be used to handle files, and easily find updated data.

DropBoxs Datastore allows for database-like storage which could be used to store songs etc. A problem with Dropbox Datastore is that it cannot be shared among users, it is private to the user.

OwnCloud allows for OwnCloud-apps to get and store data from external connections, which could be used as a datastore, but we would then have to develop a OwnCloud app.

DropBox API: https://www.dropbox.com/developers/datastore

Google Drive API: https://developers.google.com/drive/v2/reference/

OneDrive REST API: http://msdn.microsoft.com/en-us/library/dn659752.aspx

OwnCloud App documentation: http://doc.owncloud.org/server/7.0/developer_manual/app/index.html