Redundancy Active-Active

Redundancy solution allows multiple RapidOSS servers to syncronize and share data :

• There may be multiple servers, each gathering main data from external sources ( Database , Smarts or others ) .
• Users may interact with any of the servers, user data, user interface related data and other required data are reflected to other RapidOSS servers.
• There is no master for data, all data is duplicated on each server. There is only master for some management operations ( Notification System and alike ).
• If any of the servers fail the others will continue to work and syncronize data. A new server can be added at any time which will synchronize with the existing servers.
• Can be used for redundancy and load balancing if there are so many users interacting with RapidOSS front-end UI.
  

  The solution supports multiple servers, however quality tests are made with two RapidOSS servers. Solution is guaranteed to work with two RapidOSS servers.

  Installation and Customization part covers general aspects of Redundancy Solution, for further information after reading Installation part please refer to Under The Hood part below.

1. Installation and Customization

After installing RapidOSS on each server ,

• Please first install all other solutions that you need,
• Copy files under RS_HOME/RapidServer/solutions/ActiveActiveRedundancy to RS_HOME/RapidServer/RapidSuite. Files under ActiveActiveRedundancy/grails-app/domain and files under  ActiveActiveRedundancy/operations will overwrite the existing ones in RapidSuite : you may take a backup before copying the solution.
• Optional Model Configuration : By default solution synchronizes user interface, notification system and maintenance model data. If there is a need to synchronize other model data, those models should also be configured.
• Server List Configuration : createRedundancyConnections script should be changed, each server should have list of other servers in these script. 
• After Optional Model Configuration and Server List Configuration, RapidOSS server should be started and createRedundancyConnections script should be run.
• After createRedundancyConnections script is run , each server will start to syncronize with others.

2 Optional Model Configuration

Default models listed below are already configured for synchronization. The model configuration should be done only when other models need to be synchronized.

Models who does not have key properties can not be synchronized with Redundancy solution, id property can not be used for Redundancy because id generation is different among different RapidOSS servers

2.a Default Configured Models

By default Redundancy solution syncronizes data changes on the following models :

User Interface Data :

• search.SearchQueryGroup :  used by Saved Search Query Tree on left of the events, historical events, inventory tabs & notification system also
• search.SearchQuery : usage is same as SearchQueryGroup, contains the saved queries
• ui.GridView : used on all Grids on events, historical events, inventory, service view, device view tabs
• ui.GridColumn : usage is same as GridColumn , contains the columns for the views
• ui.ComponenConfig : used for every component in user interface , keeps polling interval information for each component
• ui.map.MapGroup : used by Saved Map Treee on left of the maps tab,
• ui.map.TopoMap  : usage is same as MapGroup, contains saved map information.

Notification System Data :

• message.RsMessageRule : used by Notification Tab and Notification System , contains user saved notification rules
• auth.RsUserInformation : used by Notification System , contains user destination information for email , jabber etc.
• RsLookup messageGenerator lookup instances, this is not configurable is a must for synchronization of Notification System.

Maintenance Solution Data :

• RsInMaintenace : Contains information of active devices in maintenance
• RsInMaintenanceSchedule  : Contains information of device maintenance Schedule for future.
  

  By default auth.RsUser and auth.Group models are not synchronized. Please apply the Patch in  Support Issue http://www.ifountain.org/jira/browse/SUP-31 in order to synchronize this models.

  
  

  Some bugs have been fixed related with updating an object on the same server and renaming the object. Please apply the Patch in Support Issue http://www.ifountain.org/jira/browse/SUP-32 in order to fix these problems. This patch also contains User and Group model synchronization support.

2.b Configuring Model Operations For Synchronization

• This step should be done before starting RapidOSS. If RapidOSS is running, after operation change, operations can be reloaded from "Admin UI > Reload Tab > Select The Model > Click on Reload Button"
• This configuration step is already done for the models listed in 2.a Default Configured Models part, this step is only needed if extra models need to be configured.
• Operation files are under RapidSuite/operations/   , sample :  RapidSuite/operations/message/RsMessageRuleOperations.groovy
• Following afterInsert,afterUpdate,afterDelete triggers should be added to each model operation file, if models already contains these trigger methods, code should be merged :
  

  class RsMessageRuleOperations extends com.ifountain.rcmdb.domain.operation.AbstractDomainOperation {
     //	changed for Active-Active Redundancy
      def afterInsert()
      {
  	application.RapidApplication.getUtility("RedundancyUtility").objectInAfterInsert(this.domainObject);
      }
      def afterUpdate(params)
      {
  	application.RapidApplication.getUtility("RedundancyUtility").objectInAfterUpdate(this.domainObject);
      }
      def afterDelete()
      {
  	application.RapidApplication.getUtility("RedundancyUtility").objectInAfterDelete(this.domainObject);
      }
      //change ended

  
  

  If server is running during operation change, operations can be reloaded from "Admin UI > Reload Tab > Select The Model > Click on Reload Button"

2.c Id Relation Configuration

• This step should be done before starting RapidOSS. If RapidOSS is running, after script change, script can be reloaded from "Admin UI > Scripts Tab > Reload updatedObjects script"
• This configuration step is already done for the models listed in 2.a Default Configured Models part, this step is only needed if extra models need to be configured.
• Model relations are syncronized between servers successfuly, there is no extra configuration for model relations.
• But object id is not same between RapidOSS servers, a SearchQuery with name : "Critical Events" will have the same key "Critical Events" but will have different id among RapidOSS servers.
• So if the id of any model instance is saved in any syncronization model, the id has to be converted between RapidOSS servers.
• Id conversion for id relations automatically done by Redundancy solution after appropriate configuration done.

Sample : message.RsMessageRule model contains userId and searchQueryId , when this object is transferred to another RapidOSS server , userID and searchQueryId will change .

Id relation is easily configured in RapidSuite/scripts/updatedObjects.groovy script as follows:

ID_RELATION_MAPPING["message.RsMessageRule.userId"]=auth.RsUser;
ID_RELATION_MAPPING["message.RsMessageRule.searchQueryId"]=search.SearchQuery;
ID_RELATION_MAPPING["auth.RsUserInformation.userId"]=auth.RsUser;
ID_RELATION_MAPPING["auth.ChannelUserInformation.userId"]=auth.RsUser;

The above code states that:

message.RsMessageRule model userId property contains the id of auth.RsUser model instance.

message.RsMessageRule model searchQueryId property contains the id of search.SearchQuery model instance.

3 Server List Configuration

• This step should be done before starting RapidOSS.  Then server should be started and createRedundancyConnections script should be run.
• If server is running , scripts should be reloaded from "Admin UI > Script Tab >createRedundancyConnections ", then createRedundancyConnections script should be run.
• After createRedundancyConnections script is run, syncronize scripts will be able to retrieve data from the servers in the list.
• createRedundancyConnections script on each server should be modified so that each server contains list of other servers , server names should start with "redundancy"
• As a sample RapidOSS instances on 192.168.1.10 and 192.168.1.20 will syncronize with each other :
• createRedundancyConnections script on 192.168.1.10 should be modified as follows :

...
redundancy_servers.add([name:"redundancy_1_20",baseUrl:"http://192.168.1.20:12222/RapidSuite/"]);

• createRedundancyConnections script on 192.168.1.20 should be modified as follows :
  

  ...
  redundancy_servers.add([name:"redundancy_1_10",baseUrl:"http://192.168.1.10:12222/RapidSuite/"]);

  
  
  

Under The Hood

Tracking local changes on each Server :

After model operations configured for isLocal property as in the Installation part :

• When an object is locally created or updated on a RapidOSS server, an entry is saved to UpdatedObjects model.RapidOSS automatically saves last update time to rsUpdatedAt property.
• When an object is locally deleted on a RapidOSS server, an entry is saved to DeletedObjects model.  RapidOSS automatically saves last update time to rsUpdatedAt property.
• The other servers can track model instances from UpdatedObjects model with a range query rsUpdatedAt : give me your local updated objects updated after  time X .
• When an object is created / updated on other server, no UpdatedObjects entry will be saved,
• When an object is delete on other server, no DeletedObjects entry will be saved.
• By distinguishing local and remote updated and deletes, data transfer recursion among RapidOSS server is prevented.
• For an object there can be only single UpdatedObjects or DeletedObjects entry on a server :
  • When an object is added/updated , if there is any DeletedObjects entry for that object exists from past it will be deleted.
  • When an object is deleted, if there is any UpdatedObjects entry for that object exists from past it will be deleted.
  • When an object is remotely added/updated, if there is any UpdatedObjects entry for that object exists from past it will be deleted.

Data provider for changes , updatedObjects script :

updatedObjects scripts gives information on updated and deleted objects together with their relations and id relations ( described in installation part )  in the following format :

 <Objects total='1' offset='0' rsUpdatedAt='1266499889000'>
	<Object alias='auth.RsUserInformation' id='4444441' isLocal='true' rsInsertedAt='1266499889000' rsOwner='p' rsUpdatedAt='1266499889000' type='email0' userId='5555'>
    	    <IdRelatedObject relationName='userId' alias='auth.RsUser' searchQuery='username:"(user1)" ' />
    	    <RelatedObject relationName='rsUser' alias='auth.RsUser' searchQuery='username:"(user1)" ' />
        </Object>
</Objects>

updatedObjects script gives data of the requrested model in searchIn parameter. But when searchIn is given as "RealUpdatedObjects" it searches UpdatedObjects model but returns the real object instance SearchQuery, RsMessageRule etc. searchIn : RealUpdatedObjects is used by synchronizeUpdatedObjects.

updatedObjects scripts gives the last rsUpdatedAt from last row on the header row of the xml definition by supposing sort is rsUpdatedAt, asc/ For RealUpdatedObjects it gives rsUpdatedAt of UpdatedObjects object , not the real objects rsUpdatedAt. Because UpdatedObjects entry for an object is created a little time later, in order not to miss updates or do duplicate updates this is done.

A sample call to remote updatedObjects script via http , from synronizeUpdatedObjectObjects script:

def requestParams=[:];
requestParams.login="rsadmin";
requestParams.password="changeme";
requestParams.format="xml";
requestParams.sort="rsUpdatedAt";
requestParams.order="asc";
requestParams.searchIn="RealUpdatedObjects";
requestParams.max="100";
requestParams.query="rsUpdatedAt:[1234567789 TO *] ";
requestParams.withRelations="true";
def searchUrl="script/run/updatedObjects";
def xmlResult=ds.doRequest(searchUrl,requestParams);

A sample call to remote updatedObjects script via http , from synronizeDeletedObjects script:

def requestParams=[:];
requestParams.login="rsadmin";
requestParams.password="changeme";
requestParams.format="xml";
requestParams.sort="rsUpdatedAt";
requestParams.order="asc";
requestParams.searchIn="DeletedObjects";
requestParams.max="100";
requestParams.query="rsUpdatedAt:[1234567789 TO *]";
def searchUrl="script/run/updatedObjects";
def xmlResult=ds.doRequest(searchUrl,requestParams);

Scheduled Synchronizer For Inserts & Updates,  synchronizeUpdatedObjects script :

synchronizeUpdatedObjects is a scheduled script :

• For each remote server it processes the updated objects after the last retrieved object from that server.
• After changes are done rsUpdatedAt value of last updated remote object ( rsUpdated value in top xml header row ) is saved to model RsLookup.
• Calls remote updatedObjects script with searchIn: RealUpdatedObjects .
• Script is scheduled to run at every 60 seconds. You may change schedule time from "Admin UI> Script Tab > synchronizeUpdatedObjects script > edit "

Scheduled Synchronizer For Deleted Objects, synchronizeDeletedObjects script

synchronizeDeletedObjects is a scheduled script :

• For each remote server it processes the DeletedObjects after the last retrieved object from that server,
• After deletes are done rsUpdatedAt value of last updated remote object ( rsUpdated value in top xml header row )  is saved to model RsLookup.
• Calls remote updatedObjects script with searchIn: DeletedObjects.
• Script is scheduled to run at every 60 seconds with a start delay of 15 seconds. You may change schedule time from "Admin UI> Script Tab > synchronizeDeletedObjects script > edit "
  

Scheduled Cleaner For Expired Deleted Objects, clearExpiredDeletedObjects script

 clearExpiredDeletedObjectsis a scheduled script :

• Runs on each server and deleted DeletedObjects instances which are older then 12 hours.
• Script is scheduled to run at every 12 hours.  You may change schedule time from "Admin UI> Script Tab > clearExpiredDeletedObjectsis script > edit "
  

Master Switching Mechanism , redundancyMasterSwitcher script :

Information on Master RapidOSS Server :

• All RapidOSS servers retrieves data from each other, synchronized  model data is same across all RapidOSS servers.
• There is only one difference between master and slave RapidOSS servers, master RapidOSS server runs the Notification System , processes events and sends them to users.

redundancyMasterSwitcher script is a scheduled script , on each run it works as follows :

• Script does nothing if no RapidOSS servers defined and returns error on this case.
• By default  RedundancyLookup model isMaster entry does not exist :
• If RedundancyLookup model, isMaster value is true it does nothing 
• If RedundancyLookup model, isMaster value does not exist or false :* It searches other ROSS instances,
• If it can not access or find any master ROSS server, it makes local server master by running enableLocalMaster script.
• If it finds any remote master server and RedundancyLookupisMaster value does not exist , it makes local server slave by running disableLocalMaster script. It syncs messageGenerator lookup values from Master for notification system.
  

Making Local Server Master, enableLocalMaster script :

Enables messageGenerator script and notification connector sender scripts. Saves isMaster : true to RedundancyLookup model.

• This script is only runned by scheduled script redundancyMasterSwitcher.
• You may run this script on demand on trouble cases which will make local server master.
  • Before doing so be sure that disableLocalMaster script is runned other master save, make sure that no other master server exist.
  • Be sure that RsLookup messageGenerator values are same as the last active Master. If messageGenerator values are older than the last Master server , then some of the messages will be sent again.
  • For more information on NotificationSystem lookup values please refer to Managing Notification System, part 2.a Activating Message Generator Script.
    

Making Local Server Slave, disableLocalMaster script :

Disables messageGenerator script and notification connector sender scripts. Saves isMaster : false to RedundancyLookup model.

Be aware that on the next run of redundancyMasterSwitcher script ( periodic ): it will make the local server master again if it can not find any remote master server