ETL PingOne in Oracle Data Integrator

Leverage existing skills by using the JDBC standard to connect to PingOne: Through drop-in integration into ETL tools like Oracle Data Integrator (ODI), the CData JDBC Driver for PingOne connects real-time PingOne data to your data warehouse, business intelligence, and Big Data technologies.

JDBC connectivity enables you to work with PingOne just as you would any other database in ODI. As with an RDBMS, you can use the driver to connect directly to the PingOne APIs in real time instead of working with flat files.

This article covers a JDBC-based ETL -- PingOne to Oracle. After reverse engineering a data model of PingOne entities, you will create a mapping and select a data loading strategy -- since the driver supports SQL-92, this last step can easily be accomplished by selecting the built-in SQL to SQL Loading Knowledge Module.

Install the Driver

To install the driver, copy the driver JAR (cdata.jdbc.pingone.jar) and .lic file (cdata.jdbc.pingone.lic), located in the installation folder, into the ODI appropriate directory:

• UNIX/Linux without Agent: ~/.odi/oracledi/userlib
• UNIX/Linux with Agent: ~/.odi/oracledi/userlib and $ODI_HOME/odi/agent/lib
• Windows without Agent: %APPDATA%\Roaming\odi\oracledi\userlib
• Windows with Agent: %APPDATA%\odi\oracledi\userlib and %APPDATA%\odi\agent\lib

Restart ODI to complete the installation.

Reverse Engineer a Model

Reverse engineering the model retrieves metadata about the driver's relational view of PingOne data. After reverse engineering, you can query real-time PingOne data and create mappings based on PingOne tables.

1. In ODI, connect to your repository and click New -> Model and Topology Objects. 👁 Create a New Model
   
2. On the Model screen of the resulting dialog, enter the following information:
   • Name: Enter PingOne.
   • Technology: Select Generic SQL (for ODI Version 12.2+, select Microsoft SQL Server).
   • Logical Schema: Enter PingOne.
   • Context: Select Global.
   👁 Configuring the Model
   
3. On the Data Server screen of the resulting dialog, enter the following information:
   • Name: Enter PingOne.
   • Driver List: Select Oracle JDBC Driver.
   • Driver: Enter cdata.jdbc.pingone.PingOneDriver
   • URL: Enter the JDBC URL containing the connection string.

     To connect to PingOne, configure these properties:

     • : The region where the data for your PingOne organization is being hosted.
     • : The type of authentication to use when connecting to PingOne.
     • Either (required when using the default PingOne domain) or , configured as described below.

     Configuring WorkerAppEnvironmentId

     is the ID of the PingOne environment in which your Worker application resides. This parameter is used only when the environment is using the default PingOne domain (auth.pingone). It is configured after you have created the custom OAuth application you will use to authenticate to PingOne, as described in Creating a Custom OAuth Application in the Help documentation.

     First, find the value for this property:

     1. From the home page of your PingOne organization, move to the navigation sidebar and click Environments.
     2. Find the environment in which you have created your custom OAuth/Worker application (usually Administrators), and click Manage Environment. The environment's home page displays.
     3. In the environment's home page navigation sidebar, click Applications.
     4. Find your OAuth or Worker application details in the list.
     5. Copy the value in the Environment ID field. It should look similar to:
        

        WorkerAppEnvironmentId='11e96fc7-aa4d-4a60-8196-9acf91424eca'

     Now set to the value of the Environment ID field.

     Configuring AuthorizationServerURL

     is the base URL of the PingOne authorization server for the environment where your application is located. This property is only used when you have set up a custom domain for the environment, as described in the PingOne platform API documentation. See Custom Domains.

     Authenticating to PingOne with OAuth

     PingOne supports both OAuth and OAuthClient authentication. In addition to performing the configuration steps described above, there are two more steps to complete to support OAuth or OAuthCliet authentication:

     • Create and configure a custom OAuth application, as described in Creating a Custom OAuth Application in the Help documentation.
     • To ensure that the driver can access the entities in Data Model, confirm that you have configured the correct roles for the admin user/worker application you will be using, as described in Administrator Roles in the Help documentation.
     • Set the appropriate properties for the authscheme and authflow of your choice, as described in the following subsections.

     OAuth (Authorization Code grant)

     Set to OAuth.

     Desktop Applications

     Get and Refresh the OAuth Access Token

     After setting the following, you are ready to connect:

     • : GETANDREFRESH. To avoid the need to repeat the OAuth exchange and manually setting the each time you connect, use .
     • : The Client ID you obtained when you created your custom OAuth application.
     • : The Client Secret you obtained when you created your custom OAuth application.
     • : The redirect URI you defined when you registered your custom OAuth application. For example: https://localhost:3333

     When you connect, the driver opens PingOne's OAuth endpoint in your default browser. Log in and grant permissions to the application. The driver then completes the OAuth process:

     1. The driver obtains an access token from PingOne and uses it to request data.
     2. The OAuth values are saved in the location specified in , to be persisted across connections.

     The driver refreshes the access token automatically when it expires.

     For other OAuth methods, including Web Applications, Headless Machines, or Client Credentials Grant, refer to the Help documentation.

     Built-in Connection String Designer

     For assistance in constructing the JDBC URL, use the connection string designer built into the PingOne JDBC Driver. Either double-click the JAR file or execute the jar file from the command-line.

     java -jar cdata.jdbc.pingone.jar
     

     Fill in the connection properties and copy the connection string to the clipboard.

     👁 Using the built-in connection string designer to generate a JDBC URL (Salesforce is shown.)
     

     Below is a typical connection string:

     jdbc:pingone:AuthScheme=OAuth;WorkerAppEnvironmentId=eebc33a8-xxxx-4f3a-yyyy-d3e5262fd49e;Region=NA;OAuthClientId=client_id;OAuthClientSecret=client_secret;InitiateOAuth=GETANDREFRESH;
     

   👁 Configuring the Data Server
   
4. On the Physical Schema screen, enter the following information:
   • Name: Select from the Drop Down menu.
   • Database (Catalog): Enter CData.
   • Owner (Schema): If you select a Schema for PingOne, enter the Schema selected, otherwise enter PingOne.
   • Database (Work Catalog): Enter CData.
   • Owner (Work Schema): If you select a Schema for PingOne, enter the Schema selected, otherwise enter PingOne.
   👁 Configuring the Physical Schema
   
5. In the opened model click Reverse Engineer to retrieve the metadata for PingOne tables. 👁 Reverse Engineer the Model
   

Edit and Save PingOne Data

After reverse engineering you can now work with PingOne data in ODI. To view PingOne data, expand the Models accordion in the Designer navigator, right-click a table, and click View data. 👁 Viewing the data.

Create an ETL Project

Follow the steps below to create an ETL from PingOne. You will load [CData].[Administrators].Users entities into the sample data warehouse included in the ODI Getting Started VM.

1. Open SQL Developer and connect to your Oracle database. Right-click the node for your database in the Connections pane and click new SQL Worksheet.

   Alternatively you can use SQLPlus. From a command prompt enter the following:

   sqlplus / as sysdba
   

2. Enter the following query to create a new target table in the sample data warehouse, which is in the ODI_DEMO schema. The following query defines a few columns that match the [CData].[Administrators].Users table in PingOne:

   CREATE TABLE ODI_DEMO.TRG_[CDATA].[ADMINISTRATORS].USERS (USERNAME NUMBER(20,0),Id VARCHAR2(255));
   

3. In ODI expand the Models accordion in the Designer navigator and double-click the Sales Administration node in the ODI_DEMO folder. The model is opened in the Model Editor.
4. Click Reverse Engineer. The TRG_[CDATA].[ADMINISTRATORS].USERS table is added to the model.
5. Right-click the Mappings node in your project and click New Mapping. Enter a name for the mapping and clear the Create Empty Dataset option. The Mapping Editor is displayed.
6. Drag the TRG_[CDATA].[ADMINISTRATORS].USERS table from the Sales Administration model onto the mapping.
7. Drag the [CData].[Administrators].Users table from the PingOne model onto the mapping.
8. Click the source connector point and drag to the target connector point. The Attribute Matching dialog is displayed. For this example, use the default options. The target expressions are then displayed in the properties for the target columns.
9. Open the Physical tab of the Mapping Editor and click [CDATA].[ADMINISTRATORS].USERS_AP in TARGET_GROUP.
10. In the [CDATA].[ADMINISTRATORS].USERS_AP properties, select LKM SQL to SQL (Built-In) on the Loading Knowledge Module tab. 👁 SQL-based access to PingOne enables you to use standard database-to-database knowledge modules.
    

You can then run the mapping to load PingOne data into Oracle.