ButeoSynchronizationFramework
|
The Buteo Synchronization framework provides a generic framework for creating sync plugins that can perform sync operations across devices, PC, Cloud. The framework provides the capability to manage the lifecycle of a sync plugin. It also provides scheduling, bearer sensing (USB, BT, Network), support for both client and server plugins and also storage plugins. The client plugin gets loaded for the instant of the sync session and gets unloaded thereafter. A server plugin stays alive until the sync engine is alive. The storage plugin provides a generic interface for fetching the objects, changed objects. The storage plugin can be used across multiple plugins and avoid duplication of code.
<table id="toc" class="toc" summary="Contents"><tr><td><div id="toctitle"><h2>Contents</h2> </div>
Personal computing, cloud computing and mobile computing have increasingly made users to store their personal data in personal computers, Internet based cloud services and mobile devices. The kind of personal data varies from normal text files to a varied range of media data like pictures (JPEG, GIF, etc.), music files (mp3, wav, etc.), movie files (wmv, mp4, etc.), data file (.doc, .ppt, .pdf etc.) and non-file formats like vcard (format for contact card), vcal (format for calendar entry) which has contacts and calendar information. As the data gets more and more distributed, it become the more important to keep the data in sync among the various computing entities.Various protocols(SyncML, MTP, REST etc.) have evolved to keep the user data in sync among the entities.
These various protocols work over different kinds of transports (like Bluetooth, USB, IP etc.). So it is no more enough that a particular mobile device supports only a single protocol to keep the user data in sync. It also so happens that two different protocols are also used to synchronize the same kind of data (for example, SyncML and REST API can be used to synchronize calendar data). The choice of the protocol to use largely depends on the kind of target service from where the data is fetched. Also, the device UI has to provide a coherent view to the end-user without having to expose the internal protocols being used to keep the user data in sync. In order to handle the various synchronization protocols, it is important to have a framework that dynamically chooses the protocol to use for a particular sync session and also provide various kinds of platform based services to the sync session. The Buteo Synchronization Framework tries to create a unified architecture for any kind of synchronization protocol and also enables application developers to create a unified user interface to merge all the synchronization services in the device (here the device can be a PC, mobile device or any other computing device).
The Buteo SyncFW does not provide any synchronization capabilities by itself. It is only the plug-ins and the corresponding protocol stacks that provide the synchronization capabilities.
Note: The name Buteo is the Latin name for a Hawk (more from here: http://en.wikipedia.org/wiki/Buteo)
Following diagram shows the end-user usecases for Buteo Sync solution.
The high level end-application use-cases are taking into consideration while creating the architecture
The main usecases are to allow a 3'rd party synchronization application developer to create synchronization services and deploy to the framework and to provide various other features like deploy/undeploy synchronization plug-ins, develop storage plug-ins for already existing synchronization services etc.
Following diagram depicts some more usecases of the framework
The Buteo Synchronization framework architecture is created taking into consideration the extensibility and scalability of the solution for current and future use cases. Effort is made to make use of the existing components as mush as possible.
A plug-in based architecture is the logical choice to satisfy the requirement of the ability to deploy/undeploy sync services in the device. The plug-in manager forms the core concept of the Buteo SyncFW. Even though the component is named as plug-in manager, the component does not perform any handling of the plug-ins itself. It is the sync daemon that performs the actual functionality of loading/unloading of the plugins. These plugins could also be called as sync agents, since they handle the sync sessions by using the corresponding sync protocols.
In the synchronization world, typically, there are two kinds of sync services - one that acts as a client and initiates a sync towards a sync server; the typical scenario is where a device sync’s PIM data with an online service (like Ovi.com) and another that acts as a sync server that accepts incoming sync requests or one which has to have a persistent connection towards a sync service. In order to satisfy these two kinds of modes, two kinds of plug-ins are designed. One is a Client Plugin and another is a Server Plugin. A client plugin is loaded on demand either by a GUI application or through a scheduled sync session and is unloaded once is sync is completed. On the other hand, a server plugin is always loaded for accepting incoming sync requests and is never unloaded. In terms of interface methods, the only difference between these plugins would be a “listen” method, which the server plug-in would have. The client/server role decision is made at the time of writing of the plugin and is defined the plugin configuration file. Since the Buteo SyncFW is a generic architecture for MeeGo platform, which is not just targetted for mobile devices, the server functionality could as well be used for a netbook kind of usecase, where the sync service in the netbook acts as a server and the Buteo SyncFW acts as a client.
Another plugin that is defined is the storage plugin. Typically every sync service would involve synchronization of different kinds of user data, the formats of which are profoundly different. A sync protocol like OMA DS SyncML would be able to synchronize different kinds of data with different formats, which would mean that the framework should be able to handle synchronization of these different kinds of data formats. The concept of “Storage Plugin” is created to cater to this kind of usecases. Backend storage is defined as a component in the platform that holds the user personal data. Some examples are contacts, calendar, SMS, MMS, music files, audio files, photos etc. Storage plug-in would be loaded along with the corresponding client or server plug-in. The decision of which storage plug-in to load is made based on the profile information of the deployed service and also the protocol request for the storage. More information about a profile is described in the following sections. This kind of plug-in based mechanism for storage provides a scalable architecture where it is possible to provide many storage plug-ins for the same kind of backend storage. The following figure describes the context of the plugins and the plugin manager.
The main functionality of storage plug-in would be to obtain the raw data in the protocol message and perform a transformation of the data to the format suitable for the backend storage in the device. For example a SyncML message would provide the Contact information in the form of a vcard, while the storage in the device could be in the form of a SQLite database with an API layer. In this case, the storage plug-in would obtain the vcard from the protocol, use the API layer over the Contacts database and store the vcard entry to the database. A reverse mechanism would be used to retrieve the vcard from the storage and return it to the target sync entity.
Apart from the implementation interfaces, a plugin should also have a configuration file that defines the properties of the plugin, like, whether the plug-in should be visible or hidden to the user, the storage plugins that client/server plugin should use, the target service address, like the ovi sync URL etc. The plugin configuration file has only static information, which is read by the synchronization daemon and used accordingly. The configuration file could also provide extensions, which could specifically be used by the plug-in, but not the framework.
Profile Management is quite important both from managing the deployed plug-ins as well as holds the information to be displayed to the end-user. From the end-user point of view, a profile is defined as something that provides information about the synchronization service that the user has synchronized with. A profile has only dynamic information that is created using the plug-in configuration and other information that is generated after a sync has been initiated. For example, if the user has synchronized his data with Ovi.com, then a profile entry would be created with information like the databases the sync session was initiated, time of sync, sync status etc. A profile is also used to display the sync status in a user-friendly manner in a GUI application.
The profile information is stored persistently, so that it could be used across sync sessions and also be available for any GUI application to query the sync status. The profile is the only object that moves across all the synchronization entities (daemon, GUI’s etc.). Some of the properties of a profile can be modified by the user and some properties are only modifiable by the framework. Some of the identified properties of a profile manager are as under:
Another important component of Buteo SyncFW is the synchronization scheduler. The Scheduler is responsible for scheduling synchronization sessions and also for handling parallel synchronizations. It has a queue mechanism to queue the incoming sync requests. The periodic scheduling of the Sync sessions is handled by using an alarm kind of functionality. The frequency is set by the end-user through a well-known GUI interface. The frequency is defined as a repeat event like everyday, a particular day every week, the hour setting etc. This event is converted to an alarm event and the alarm is triggered whenever the timer expired.
One of the main properties of a sync protocol is to provide data consistency. Even though the backend datastores may provide data consistency through database ACID properties simultaneous synchronization sessions might result in inconsistent results in the end-user data. For example, if sync session1 adds 10 data elements, and a simultaneous sync session2 might result in the addition of the same set of data from a different source, resulting in duplication of data. The framework provides a functionality, wherein simultaneous sync requests that access the same backend datastore are queued and are executed in the first-in-first-out order. In the above example, the two sync sessions are queued and are executed one-after-the-other. More details on how this is achieved is explained in the low-level design.
Accounts&SSO (http://wiki.meego.com/Single_sign-on) is a component in MeeGo platform that provides a one-stop shop for the end-users to configure online accounts. Examples of online acconts are Ovi, Google etc. Most of the online services have Single Sign On enabled which enables the user to enter the credentials only once and be able to access all the services (like Ovi Music, Ovi Sync, Ovi Sharing etc.) without having to re-enter the credentials again and again. The Accounts&SSO component on the device is the counterpart of the online service accounts management. This subsystem also stores the user credentials in a secure location which is not readable by a non-root user. Since the Buteo SyncFW supports synchronization of user data with online services, it needs to be integrated to the Accounts&SSO subsystem to fetch the SSO token and provide it to the online sync service.
The Accounts&SSO subsystem provides a pluggable interface to hook-in a new online account service. The pluggable component uses a XML file definition similar to Buteo FW. The Buteo SyncFW uses the account identifier defined in the accounts XML definition to link the sync plug-in with the corresponding account. The plug-in developer is also given the option to not use to the Accounts&SSO subsystem, but rather provide the credentials in the service XML.
The synchronization daemon is the only always running process in the sync framework. It provides the functionality of loading/unloading the plug-ins, sending d-bus signals, handle profile management (like creating profiles, deleting them etc.) integration with Accounts, hooking up with the hardware layer to register for interested signals (like bluetooth availability, USB and network connectivity etc). The daemon would also load the server plug-ins which would allow external devices to connect to the sync service in the device and perform synchronization. Following is the component context diagram of synchronization daemon depicting the various components that it interacts with. The synchronization daemon is the central component in the framework and is responsible for managing the various states of the framework.
In the above figure, the boxes in blue represent the Buteo SyncFW components and the boxes in yellow represent the components in MeeGo. The framework provides handlers for each of the external component services that it uses. The main functionality of the daemon is:
Following is the state machine diagram of the framework, of which the daemon forms the central component Following are the various states and activities that occur in the synchronization framework:
As part of the framework, the solution provides an API for 3'rd party developers to create plug-ins for the MeeGo platform. The framework provides two kinds of APIs:
For most of the synchronization services, it is important to initiate the sync on the availability of the underlying transport. For example, when the device is connected to a PC via USB, when bluetooth is switched on in the device, on the availability of the network (wifi/GPRS/3G..) etc. In order to support these usecases, the framework hooks into the hardware notifications services in the device, like Context Framework, Hardware Abstraction Layer (HAL) etc. The Context Framework provides context aware information like Bluetooth availability, network availability etc. The USB connectivity is obtained using HAL. Whenever these events happen, the hardware hookup adapter notifies the daemon to take the appropriate action. The daemon checks all the plugin profiles that have registered to be invoked whenever a particular transport is available and loads those plugins. Once loaded, the plug-ins check if the transport suits their requirements, perform their work and exit, after which they are unloaded by the daemon. There could be some server plugins that are always loaded as long as the underlying transport is available.
The device state based sync is mainly related to handling of the scenarios where sync cannot continue because of low-device-resource conditions. Examples of low resource conditions are less disk space, low memory, low-battery level etc. The framework makes use of Context framework to fetch information related to some of the device state variables, like battery level, but for other conditions like disk space, low memory the native Linux methods should be used. Details are available in the low-level design chapter. Note that if the sync is initiated from the UI, most of these checks should be done in the UI itself to avoid round-trip checks by the engine. This is a performance improvement measure
Another important feature of any sync solution is to initiate sync whenever the user changes any of his/her personal data. The general term for this is change based sync. There could be two methodologies used to support this kind of feature:
1.If the backend datastores have the ability to send notification signals whenever the database is modified. But it cannot be expected that each datastore supports the notification mechanism. This leads to the next possible option 2.The synchronization daemon polls the various databases at periodic intervals of time to know if some changes have been done to the data- stores. If this methodology is chosen, the timing of the interval becomes quite important. In this methodology, the sync cannot be real-time, since if:
the change in the datastore happens at t0, and the periodic datastore check interval is at t1, then the sync would occur only after every n*t1, where n is the periodicity of the sync
This is a reasonable limitation, since the user might not expect a real-time sync operation. One of the possibilities to implement this functionality is to provide a plug-in that would be loaded in a periodic interval and which would be able to check the backend datastores for any changes. The scheduling functionality of the sync can be used to periodically load this plugin. This design avoids the sync framework to directly access the backend datastores.
Most of the synchronization protocols support a feature called “fast sync” [Ref: TBD] wherein the consecutive sync’s after the first sync only synchronize the changes from the previous synchronization. The changes include the data that was added/modified/deleted. The protocol implementations make use of the backend datastores capability to return the list of added/modified and deleted items. Though most of the datastores are able to provide information related to added and modified items, they do not keep track of the deleted items and purge them for good. In such cases, handling the list of deleted items becomes the job of the synchronization service.
Since this is a feature that is required by most of the storage plug-ins, the framework could as well provide an interface which the storage plug-ins would implement and store a list of deleted items. Couple of mechanisms exists to keep track of the deleted items:
Apart from a profile being created whenever the user enables the account or synchronizes with a target device, at some points of time, it could also be deleted by the user. Any profile specific data in the sync framework could be handled by the framework itself, but most of the times, the protocol stacks and the plugins also maintain the corresponding profile specific data. For example, the SyncML stack maintains the anchor information and the id mappings, which have to be purged when the corresponding profile is deleted. The SyncFW cannot directly delete this information from the protocol stack, since the data is protocol specific. A good solution to handle this case is to provide a plug-in interace method (say “cleanup”), which every plug-in has to implement to cleanup the plug-in and stack specific data. In order to invoke this method, whenever the user performs a “delete” operation, the plug-in is loaded and the “cleanup” method is invoked. This is a clean approach to handle the plug-in specific operations and shows the strength of the extendability of the plug-in method.
The framework provides a d-bus API [Developer API] that the client applications can use to interact with the synchronization daemon. This works out as long as the just use simple d-bus API methods like “startSync”, “stopSync” etc., but for complex API methods like “fetchProfile”, “getSyncLogs” the UI clients have to unmarshall the complex d-bus message, parse the message and then take an appropriate action. The more then number of clients towards the framework, the more code replication there is. In order to avoid this, it makes more sense to provide a synchronization framework client library, which the applications can use to interact with the framework. The framework would be responsible for performing the task of un-marshalling the d-bus message, parsing the message and then signal the UI component to fetch the result. More detailed information is provided in further sections.
Following is the system context diagram
Accounts & SSO subsystem (for accounts integration) that belongs to the “Personal Services” layer Bluetooth (for Bluetooth related information) that belongs to the “Comms Services” layer Qt Core (for using the Qt core API classes) (QtAPI) that belongs to the visual services The d-bus messaging services (for providing the d-bus services) (FreedesktopDbus) that belongs to the “kernel” layer
Interface Name | Interface Type | Description |
---|---|---|
libaccounts | Dynamic link library | The Accounts&SSO library that provides the functionality of accounts registration |
libdbus | Dynamic link library | The d-bus library that is used to use as well as provide the d-bus interface for the framework |
libbluetooth | d-bus | The BlueZ d-bus API is used to get notifications about availability of bluetooth |
Interface Name | Interface Type | Description |
---|---|---|
libbuteosyncfw | Dynamic link library | The library provides the framework functionality of Buteo. The API provided by this library includes support for writing plugins, managing profiles and obtaining sync results. |
Framework d-bus API | d-bus | Apart from the library libbuteosyncfw, this d-bus API is for applications to interact with the synchronization daemon. An application using this API will have to understand d-bus and unmarshall and parse the d-bus messages itself. |
Profile XML API | XML file | A plug-in could be deployed to the framework with a default profile XML file. The format of the profile XML file also forms as part of the API |
Buteo syncfw version 0.1.0 supported only dynamic link library plugins which the framework loads into the same process memory as msyncd. This archicture has a problem that if any one of the plugin misbehaves (crashes, for example), msyncd would also crash and there is a probability that it would never recover. To avoid such situations, an out of process plugin architecture was devised and implemented. In this architecture, each of the plugins would be running as separate processes and msyncd process would interact with each of the processes to handle the sync life cycle and operations.
This new architecture is devised with the following absolute requirements:
Keeping the above requirements in context, the following architecture is devised, the class diagram of which is depicted below.
The classes in blue belong to the framework, while the classes in pink to the plugin. Two concrete classes, OOPClientPlugin and OOPServerPlugin (that inherit from ClientPlugin and ServerPlugin respectively) are created which act as the interface between the plugin and the framework. These classes will be responsible for performing any back-forth conversion of the data transferred. These classes also ensure that the interface between msyncd and the plugins remain the same. In dll based plugin, the .so file is opened using dlsym and the binary is loaded into memory and then the ClientPlugin object is created. In this mechanism, the plugin process is started and then the OOPClientPlugin object is created, which then talks to the process plugin over d-bus.
D-bus with its inbuilt capability to marshall data, invoke signals/slots, seamless API is chosen as the communication mechanism between the plugins and msyncd. A dbus interface is created for communication between msyncd and the plugins. This interface is same as the interface provided by both ClientPlugin and ServerPlugin. Using the xml interface description (common for both client and server plugin), dbus client proxy and server adaptor code is generated (using qdbusxml2cpp tool).
For facilitating that none of the plugin code changes and also to commonalize the code across all the plugins, the class PluginServiceObj (that implmenets the dbus skeleton adaptor class) is created. This class ensures that none of the existing plugin code has to be changed and also performs any transformation of the data between msyncd and the plugins (over dbus). This class is a peer class of OOPClientPlugin and OOPServerPlugin. Apart from this class, the plugin_main.cpp provides the main() function required to generate an executable binary. This class takes the plugin name and the profile name as arguments and then initializes the PluginServiceObj and registers it as a dbus service.
All signals emitted by the plugin (and msyncd) are automatically relayed by Qt dbus. The life cycle of a process plugin is similar to that of a dll plugin.
To convert an existing dll plugin to a process plugin, only a few settings need to be added. Described here Writing Out of process plugins