Replicate Multiple IBM Cloud Object Storage Accounts via the CData Sync CLI

CData Sync for IBM Cloud Object Storage is a stand-alone application that provides solutions for a variety of replication scenarios such as replicating sandbox and production instances into your database. Both Sync for Windows and Sync for Java include a command-line interface (CLI) that makes it easy to manage multiple IBM Cloud Object Storage connections. In this article we show how to use the CLI to replicate multiple IBM Cloud Object Storage accounts.

Configure IBM Cloud Object Storage Connections

You can save connection and email notification settings in an XML configuration file. To replicate multiple IBM Cloud Object Storage accounts, use multiple configuration files. Below is an example configuration to replicate IBM Cloud Object Storage to SQLite:

Windows

<?xml version="1.0" encoding="UTF-8" ?>
<CDataSync>
 <DatabaseType>SQLite</DatabaseType>
 <DatabaseProvider>System.Data.SQLite</DatabaseProvider>
 <ConnectionString>ApiKey=myApiKey;CloudObjectStorageCRN=MyInstanceCRN;Region=myRegion;OAuthClientId=MyOAuthClientId;OAuthClientSecret=myOAuthClientSecret;</ConnectionString>
 <ReplicateAll>False</ReplicateAll>
 <NotificationUserName></NotificationUserName>
 <DatabaseConnectionString>Data Source=C:\my.db</DatabaseConnectionString>
 <TaskSchedulerStartTime>09:51</TaskSchedulerStartTime>
 <TaskSchedulerInterval>Never</TaskSchedulerInterval>
</CDataSync>

Java

<?xml version="1.0" encoding="UTF-8" ?>
<CDataSync>
<DatabaseType>SQLite</DatabaseType>
 <DatabaseProvider>org.sqlite.JDBC</DatabaseProvider>
 <ConnectionString>ApiKey=myApiKey;CloudObjectStorageCRN=MyInstanceCRN;Region=myRegion;OAuthClientId=MyOAuthClientId;OAuthClientSecret=myOAuthClientSecret;</ConnectionString>
 <ReplicateAll>False</ReplicateAll>
 <NotificationUserName></NotificationUserName>
 <DatabaseConnectionString>Data Source=C:\my.db</DatabaseConnectionString>
</CDataSync>

Register a New Instance of Cloud Object Storage

If you do not already have Cloud Object Storage in your IBM Cloud account, follow the procedure below to install an instance of SQL Query in your account:

1. Log in to your IBM Cloud account.
2. Navigate to the

Connecting using OAuth Authentication

There are certain connection properties you need to set before you can connect. You can obtain these as follows:

API Key

To connect with IBM Cloud Object Storage, you need an API Key. You can obtain this as follows:

1. Log in to your IBM Cloud account.
2. Navigate to the Platform API Keys page.
3. On the middle-right corner click "Create an IBM Cloud API Key" to create a new API Key.
4. In the pop-up window, specify the API Key name and click "Create". Note the API Key as you can never access it again from the dashboard.

Cloud Object Storage CRN

If you have multiple accounts, specify the CloudObjectStorageCRN explicitly. To find the appropriate value, you can:

• Query the Services view. This will list your IBM Cloud Object Storage instances along with the CRN for each.
• Locate the CRN directly in IBM Cloud. To do so, navigate to your IBM Cloud Dashboard. In the Resource List, Under Storage, select your Cloud Object Storage resource to get its CRN.

Connecting to Data

You can now set the following to connect to data:

• InitiateOAuth: Set this to GETANDREFRESH. You can use InitiateOAuth to avoid repeating the OAuth exchange and manually setting the OAuthAccessToken.
• ApiKey: Set this to your API key which was noted during setup.
• CloudObjectStorageCRN (Optional): Set this to the cloud object storage CRN you want to work with. While the connector attempts to retrieve this automatically, specifying this explicitly is recommended if you have more than Cloud Object Storage account.

When you connect, the connector completes the OAuth process.

1. Extracts the access token and authenticates requests.
2. Saves OAuth values in OAuthSettingsLocation to be persisted across connections.

Configure Queries for Each IBM Cloud Object Storage Instance

Sync enables you to control replication with standard SQL. The REPLICATE statement is a high-level command that caches and maintains a table in your database. You can define any SELECT query supported by the IBM Cloud Object Storage API. The statement below caches and incrementally updates a table of IBM Cloud Object Storage data:

REPLICATE Objects;

You can specify a file containing the replication queries you want to use to update a particular database. Separate replication statements with semicolons. The following options are useful if you are replicating multiple IBM Cloud Object Storage accounts into the same database:

You can use a different table prefix in the REPLICATE SELECT statement:

REPLICATE PROD_Objects SELECT * FROM Objects 

Alternatively, you can use a different schema:

REPLICATE PROD.Objects SELECT * FROM Objects

Run Sync

After you have configured the connection strings and replication queries, you can run Sync with the following command-line options:

Windows

IBMCloudObjectStorageSync.exe -g MyProductionIBMCloudObjectStorageConfig.xml -f MyProductionIBMCloudObjectStorageSync.sql

Java

java -Xbootclasspath/p:c:\sqlitejdbc.jar -jar IBMCloudObjectStorageSync.jar -g MyProductionIBMCloudObjectStorageConfig.xml -f MyProductionIBMCloudObjectStorageSync.sql