Load PingOne Data to a Database Using Embulk

Embulk is an open source bulk data loader. When paired with the CData JDBC Driver for PingOne, Embulk easily loads data from PingOne to any supported destination. In this article, we explain how to use the CData JDBC Driver for PingOne in Embulk to load PingOne data to a MySQL dtabase.

With built-in optimized data processing, the CData JDBC Driver offers unmatched performance for interacting with live PingOne data. When you issue complex SQL queries to PingOne, the driver pushes supported SQL operations, like filters and aggregations, directly to PingOne and utilizes the embedded SQL engine to process unsupported operations client-side (often SQL functions and JOIN operations).

Configure a JDBC Connection to PingOne Data

Before creating a bulk load job in Embulk, note the installation location for the JAR file for the JDBC Driver (typically C:\Program Files\CData\CData JDBC Driver for PingOne\lib).

Embulk supports JDBC connectivity, so you can easily connect to PingOne and execute SQL queries. Before creating a bulk load job, create a JDBC URL for authenticating with PingOne.

To connect to PingOne, configure these properties:

• : The region where the data for your PingOne organization is being hosted.
• : The type of authentication to use when connecting to PingOne.
• Either (required when using the default PingOne domain) or , configured as described below.

Configuring WorkerAppEnvironmentId

is the ID of the PingOne environment in which your Worker application resides. This parameter is used only when the environment is using the default PingOne domain (auth.pingone). It is configured after you have created the custom OAuth application you will use to authenticate to PingOne, as described in Creating a Custom OAuth Application in the Help documentation.

First, find the value for this property:

1. From the home page of your PingOne organization, move to the navigation sidebar and click Environments.
2. Find the environment in which you have created your custom OAuth/Worker application (usually Administrators), and click Manage Environment. The environment's home page displays.
3. In the environment's home page navigation sidebar, click Applications.
4. Find your OAuth or Worker application details in the list.
5. Copy the value in the Environment ID field. It should look similar to:
   

   WorkerAppEnvironmentId='11e96fc7-aa4d-4a60-8196-9acf91424eca'

Now set to the value of the Environment ID field.

Configuring AuthorizationServerURL

is the base URL of the PingOne authorization server for the environment where your application is located. This property is only used when you have set up a custom domain for the environment, as described in the PingOne platform API documentation. See Custom Domains.

Authenticating to PingOne with OAuth

PingOne supports both OAuth and OAuthClient authentication. In addition to performing the configuration steps described above, there are two more steps to complete to support OAuth or OAuthCliet authentication:

• Create and configure a custom OAuth application, as described in Creating a Custom OAuth Application in the Help documentation.
• To ensure that the driver can access the entities in Data Model, confirm that you have configured the correct roles for the admin user/worker application you will be using, as described in Administrator Roles in the Help documentation.
• Set the appropriate properties for the authscheme and authflow of your choice, as described in the following subsections.

OAuth (Authorization Code grant)

Set to OAuth.

Desktop Applications

Get and Refresh the OAuth Access Token

After setting the following, you are ready to connect:

• : GETANDREFRESH. To avoid the need to repeat the OAuth exchange and manually setting the each time you connect, use .
• : The Client ID you obtained when you created your custom OAuth application.
• : The Client Secret you obtained when you created your custom OAuth application.
• : The redirect URI you defined when you registered your custom OAuth application. For example: https://localhost:3333

When you connect, the driver opens PingOne's OAuth endpoint in your default browser. Log in and grant permissions to the application. The driver then completes the OAuth process:

1. The driver obtains an access token from PingOne and uses it to request data.
2. The OAuth values are saved in the location specified in , to be persisted across connections.

The driver refreshes the access token automatically when it expires.

For other OAuth methods, including Web Applications, Headless Machines, or Client Credentials Grant, refer to the Help documentation.

Built-in Connection String Designer

For assistance in constructing the JDBC URL, use the connection string designer built into the PingOne JDBC Driver. Either double-click the JAR file or execute the jar file from the command-line.

java -jar cdata.jdbc.pingone.jar

Fill in the connection properties and copy the connection string to the clipboard.

👁 Using the built-in connection string designer to generate a JDBC URL (Salesforce is shown.)

Below is a typical JDBC connection string for PingOne:

jdbc:pingone:AuthScheme=OAuth;WorkerAppEnvironmentId=eebc33a8-xxxx-4f3a-yyyy-d3e5262fd49e;Region=NA;OAuthClientId=client_id;OAuthClientSecret=client_secret;InitiateOAuth=GETANDREFRESH;

Load PingOne Data in Embulk

After installing the CData JDBC Driver and creating a JDBC connection string, install the required Embulk plugins.

Install Embulk Input & Output Plugins

1. Install the JDBC Input Plugin in Embulk.
   https://github.com/embulk/embulk-input-jdbc/tree/master/embulk-input-jdbc

embulk gem install embulk-input-jdbc

2. In this article, we use MySQL as the destination database. You can also choose SQL Server, PostgreSQL, or Google BigQuery as the destination using the output Plugins.
   https://github.com/embulk/embulk-output-jdbc/tree/master/embulk-output-mysql

   embulk gem install embulk-output-mysql
   

With the input and output plugins installed, we are ready to load PingOne data into MySQL using Embulk.

Create a Job to Load PingOne Data

Start by creating a config file in Embulk, using a name like pingone-mysql.yml.

1. For the input plugin options, use the CData JDBC Driver for PingOne, including the path to the driver JAR file, the driver class (e.g. cdata.jdbc.pingone.PingOneDriver), and the JDBC URL from above
2. For the output plugin options, use the values and credentials for the MySQL database

Sample Config File (pingone-mysql.yml)

in:
	type: jdbc
	driver_path: C:\Program Files\CData[product_name] 20xx\lib\cdata.jdbc.pingone.jar
	driver_class: cdata.jdbc.pingone.PingOneDriver
	url: jdbc:pingone:AuthScheme=OAuth;WorkerAppEnvironmentId=eebc33a8-xxxx-4f3a-yyyy-d3e5262fd49e;Region=NA;OAuthClientId=client_id;OAuthClientSecret=client_secret;InitiateOAuth=GETANDREFRESH;
	table: "[CData].[Administrators].Users"
out: 
	type: mysql
	host: localhost
	database: DatabaseName
	user: UserId
	password: UserPassword
	table: "[CData].[Administrators].Users"
	mode: insert

After creating the file, run the Embulk job.

embulk run pingone-mysql.yml

After running the the Embulk job, find the Salesforce data in the MySQL table.

Load Filtered PingOne Data

In addition to loading data directly from a table, you can use a custom SQL query to have more granular control of the data loaded. You can also perform increment loads by setting a last updated column in a SQL WHERE clause in the query field.

in:
	type: jdbc
	driver_path: C:\Program Files\CData[product_name] 20xx\lib\cdata.jdbc.pingone.jar
	driver_class: cdata.jdbc.pingone.PingOneDriver
	url: jdbc:pingone:AuthScheme=OAuth;WorkerAppEnvironmentId=eebc33a8-xxxx-4f3a-yyyy-d3e5262fd49e;Region=NA;OAuthClientId=client_id;OAuthClientSecret=client_secret;InitiateOAuth=GETANDREFRESH;
	query: "SELECT Id, Username FROM [CData].[Administrators].Users WHERE [RecordId] = 1"
out: 
	type: mysql
	host: localhost
	database: DatabaseName
	user: UserId
	password: UserPassword
	table: "[CData].[Administrators].Users"
	mode: insert

More Information & Free Trial

By using CData JDBC Driver for PingOne as a connector, Embulk can integrate PingOne data into your data load jobs. And with drivers for more than 200+ other enterprise sources, you can integrate any enterprise SaaS, big data, or NoSQL source as well. Download a 30-day free trial and get started today.