VOOZH about

URL: https://www.cdata.com/kb/tech/databricks-cloud-datapine.rst

⇱ Build Databricks-Connected Visualizations in datapine

Build Databricks-Connected Visualizations in datapine

👁 Dibyendu Datta
Dibyendu Datta
Lead Technology Evangelist
Use CData Connect AI and datapine to build visualizations and dashboards with access to live Databricks data.

datapine is a browser-based business intelligence platform. When paired with the CData Connect AI, you get access to your Databricks data directly from your datapine visualizations and dashboards. This article describes connecting to Databricks in CData Connect AI and building a simple Databricks-connected visualization in datapine.

CData Connect AI provides a pure SQL Server interface for Databricks, allowing you to query data from Databricks without replicating the data to a natively supported database. Using optimized data processing out of the box, CData Connect AI pushes all supported SQL operations (filters, JOINs, etc.) directly to Databricks, leveraging server-side processing to return the requested Databricks data quickly.

About Databricks Data Integration

Accessing and integrating live data from Databricks has never been easier with CData. Customers rely on CData connectivity to:

  • Access all versions of Databricks from Runtime Versions 9.1 - 13.X to both the Pro and Classic Databricks SQL versions.
  • Leave Databricks in their preferred environment thanks to compatibility with any hosting solution.
  • Secure authenticate in a variety of ways, including personal access token, Azure Service Principal, and Azure AD.
  • Upload data to Databricks using Databricks File System, Azure Blog Storage, and AWS S3 Storage.

While many customers are using CData's solutions to migrate data from different systems into their Databricks data lakehouse, several customers use our live connectivity solutions to federate connectivity between their databases and Databricks. These customers are using SQL Server Linked Servers or Polybase to get live access to Databricks from within their existing RDBMs.

Read more about common Databricks use-cases and how CData's solutions help solve data problems in our blog: What is Databricks Used For? 6 Use Cases.


Getting Started


Configure Databricks Connectivity for datapine

Connectivity to Databricks from datapine is made possible through CData Connect AI. To work with Databricks data from datapine, we start by creating and configuring a Databricks connection.

  1. Log into Connect AI, click Sources, and then click Add Connection
    2. 👁 Adding a Connection
  2. Select "Databricks" from the Add Connection panel
    3. 👁 Selecting a data source
  3. Enter the necessary authentication properties to connect to Databricks.

    To connect to a Databricks cluster, set the properties as described below.

    Note: The needed values can be found in your Databricks instance by navigating to Clusters, and selecting the desired cluster, and selecting the JDBC/ODBC tab under Advanced Options.

    • Server: Set to the Server Hostname of your Databricks cluster.
    • HTTPPath: Set to the HTTP Path of your Databricks cluster.
    • Token: Set to your personal access token (this value can be obtained by navigating to the User Settings page of your Databricks instance and selecting the Access Tokens tab).
    👁 Configuring a connection (Salesforce is shown)
  4. Click Save & Test
  5. Navigate to the Permissions tab in the Add Databricks Connection page and update the User-based permissions. 👁 Updating permissions

Add a Personal Access Token

When connecting to Connect AI through the REST API, the OData API, or the Virtual SQL Server, a Personal Access Token (PAT) is used to authenticate the connection to Connect AI. It is best practice to create a separate PAT for each service to maintain granularity of access.

  1. Click on the Gear icon () at the top right of the Connect AI app to open the settings page.
  2. On the Settings page, go to the Access Tokens section and click Create PAT.
  3. Give the PAT a name and click Create. 👁 Creating a new PAT
  4. The personal access token is only visible at creation, so be sure to copy it and store it securely for future use.

With the connection configured and a PAT generated, you are ready to connect to Databricks data from datapine.

Connecting to Databricks from datapine

Once you configure your connection to Databricks in Connect AI, you are ready to connect to Databricks from datapine.

  1. Log into datapine
  2. Click Connect to navigate to the "Connect" page
  3. Select MS SQL Server as the data source
  4. In the Integration step, fill in the connection properties and click "Save and Proceed"
    • Set the Internal Name
    • Set Database Name to the name of the connection we just configured (e.g. Databricks1)
    • Set Host / IP to "tds.cdata.com"
    • Set Username to your Connect AI username (e.g. [email protected])
    • Set Password to the corresponding PAT
    • Set Database Port to "14333"
    👁 Configuring the connection to CData Connect AI
  5. In the Data Schema step, select the tables and fields to visualize and click "Save and Proceed" 👁 Selecting tables and fields to visualize (Salesforce is shown)
  6. In the References step, define any relationships between your selected tables and click "Save and Proceed" 👁 Defining foreign key relationships
  7. In the Data Transfer step, click "Go to Analyzer"

Visualize Databricks Data in datapine

After connecting to CData Connect AI, you are ready to visualize your Databricks data in datapine. Simply select the dimensions and measures you wish to visualize!

👁 Visualizing data in datapine (Salesforce is shown)

Having connect to Databricks from datapine, you are now able to visualize and analyze real-time Databricks data no matter where you are. To get live data access to hundreds of SaaS, Big Data, and NoSQL sources directly from datapine, try CData Connect AI today!