How to work with Zuora Data in Apache Spark using SQL

Apache Spark is a fast and general engine for large-scale data processing. When paired with the CData JDBC Driver for Zuora, Spark can work with live Zuora data. This article describes how to connect to and query Zuora data from a Spark shell.

The CData JDBC Driver offers unmatched performance for interacting with live Zuora data due to optimized data processing built into the driver. When you issue complex SQL queries to Zuora, the driver pushes supported SQL operations, like filters and aggregations, directly to Zuora and utilizes the embedded SQL engine to process unsupported operations (often SQL functions and JOIN operations) client-side. With built-in dynamic metadata querying, you can work with and analyze Zuora data using native data types.

Install the CData JDBC Driver for Zuora

Download the CData JDBC Driver for Zuora installer, unzip the package, and run the JAR file to install the driver.

Start a Spark Shell and Connect to Zuora Data

1. Open a terminal and start the Spark shell with the CData JDBC Driver for Zuora JAR file as the jars parameter:

   $ spark-shell --jars /CData/CData JDBC Driver for Zuora/lib/cdata.jdbc.zuora.jar
   

2. With the shell running, you can connect to Zuora with a JDBC URL and use the SQL Context load() function to read a table.

   Zuora uses the OAuth standard to authenticate users. See the online Help documentation for a full OAuth authentication guide.

   Configuring Tenant property

   In order to create a valid connection with the provider you need to choose one of the Tenant values (USProduction by default) which matches your account configuration. The following is a list with the available options:

   • USProduction: Requests sent to https://rest.zuora.com.
   • USAPISandbox: Requests sent to https://rest.apisandbox.zuora.com"
   • USPerformanceTest: Requests sent to https://rest.pt1.zuora.com"
   • EUProduction: Requests sent to https://rest.eu.zuora.com"
   • EUSandbox: Requests sent to https://rest.sandbox.eu.zuora.com"

   Selecting a Zuora Service

   Two Zuora services are available: Data Query and AQuA API. By default ZuoraService is set to AQuADataExport.

   DataQuery

   The Data Query feature enables you to export data from your Zuora tenant by performing asynchronous, read-only SQL queries. We recommend to use this service for quick lightweight SQL queries.

   Limitations
   • The maximum number of input records per table after filters have been applied: 1,000,000
   • The maximum number of output records: 100,000
   • The maximum number of simultaneous queries submitted for execution per tenant: 5
   • The maximum number of queued queries submitted for execution after reaching the limitation of simultaneous queries per tenant: 10
   • The maximum processing time for each query in hours: 1
   • The maximum size of memory allocated to each query in GB: 2
   • The maximum number of indices when using Index Join, in other words, the maximum number of records being returned by the left table based on the unique value used in the WHERE clause when using Index Join: 20,000

   AQuADataExport

   AQuA API export is designed to export all the records for all the objects ( tables ). AQuA query jobs have the following limitations:

   Limitations
   • If a query in an AQuA job is executed longer than 8 hours, this job will be killed automatically.
   • The killed AQuA job can be retried three times before returned as failed.

   Built-in Connection String Designer

   For assistance in constructing the JDBC URL, use the connection string designer built into the Zuora JDBC Driver. Either double-click the JAR file or execute the jar file from the command-line.

   java -jar cdata.jdbc.zuora.jar
   

   Fill in the connection properties and copy the connection string to the clipboard.

   👁 Using the built-in connection string designer to generate a JDBC URL (Salesforce is shown.)
   

   Configure the connection to Zuora, using the connection string generated above.

   scala> val zuora_df = spark.sqlContext.read.format("jdbc").option("url", "jdbc:zuora:OAuthClientID=MyOAuthClientId;OAuthClientSecret=MyOAuthClientSecret;Tenant=USProduction;ZuoraService=DataQuery;InitiateOAuth=GETANDREFRESH;").option("dbtable","Invoices").option("driver","cdata.jdbc.zuora.ZuoraDriver").load()
   

3. Once you connect and the data is loaded you will see the table schema displayed.

4. Register the Zuora data as a temporary table:

   scala> zuora_df.registerTable("invoices")

5. Perform custom SQL queries against the Data using commands like the one below:

   scala> zuora_df.sqlContext.sql("SELECT Id, BillingCity FROM Invoices WHERE BillingState = CA").collect.foreach(println)

   You will see the results displayed in the console, similar to the following:

   👁 Data in Apache Spark (Salesforce is shown)
   

Using the CData JDBC Driver for Zuora in Apache Spark, you are able to perform fast and complex analytics on Zuora data, combining the power and utility of Spark with your data. Download a free, 30 day trial of any of the hundreds of CData JDBC Drivers and get started today.