Automate BigQuery Integration Tasks from PowerShell

The CData Cmdlets for BigQuery are standard PowerShell cmdlets that make it easy to accomplish data cleansing, normalization, backup, and other integration tasks by enabling real-time and bidirectional access to BigQuery.

Getting Started

---

PowerShell Cmdlets or ADO.NET Provider?

The Cmdlets are not only a PowerShell interface to BigQuery, but also an SQL interface; this tutorial shows how to use both to create, retrieve, update, and delete BigQuery data. We also show examples of the ADO.NET equivalent, which is possible with the CData ADO.NET Provider for Google BigQuery. To access BigQuery data from other .NET applications, like LINQPad, use the CData ADO.NET Provider for Google BigQuery.

Once you have acquired the necessary connection properties, accessing BigQuery data in PowerShell can be enabled in three steps.

Google uses the OAuth authentication standard. To access Google APIs on behalf of individual users, you can use the embedded credentials or you can register your own OAuth app.

OAuth also enables you to use a service account to connect on behalf of users in a Google Apps domain. To authenticate with a service account, register an application to obtain the OAuth JWT values.

In addition to the OAuth values, specify the DatasetId and ProjectId. See the "Getting Started" chapter of the help documentation for a guide to using OAuth.

PowerShell

1. Install the module:

   Install-Module GoogleBigQueryCmdlets

2. Connect:

   $googlebigquery = Connect-GoogleBigQuery -DataSetId "$DataSetId" -ProjectId "$ProjectId" -InitiateOAuth "$InitiateOAuth"
   

3. Search for and retrieve data:

   $shipcity = "New York"
   $orders = Select-GoogleBigQuery -Connection $googlebigquery -Table "Orders" -Where "ShipCity = `'$ShipCity`'"
   $orders
   

   You can also use the Invoke-GoogleBigQuery cmdlet to execute SQL commands:

   $orders = Invoke-GoogleBigQuery -Connection $googlebigquery -Query 'SELECT * FROM Orders WHERE ShipCity = @ShipCity' -Params @{'@ShipCity'='New York'}
   

ADO.NET

1. Load the provider's assembly:

   [Reflection.Assembly]::LoadFile("C:\Program Files\CData\CData ADO.NET Provider for Google BigQuery\lib\System.Data.CData.GoogleBigQuery.dll")
    

2. Connect to BigQuery:

    
   $conn= New-Object System.Data.CData.GoogleBigQuery.GoogleBigQueryConnection("DataSetId=MyDataSetId;ProjectId=MyProjectId;InitiateOAuth=GETANDREFRESH;")
   $conn.Open()
   

3. Instantiate the GoogleBigQueryDataAdapter, execute an SQL query, and output the results:

   $sql="SELECT OrderName, Freight from Orders"
   
   $da= New-Object System.Data.CData.GoogleBigQuery.GoogleBigQueryDataAdapter($sql, $conn)
   $dt= New-Object System.Data.DataTable
   $da.Fill($dt)
   
   $dt.Rows | foreach {
   	Write-Host $_.ordername $_.freight
   }
    

Update BigQuery Data

PowerShell

Update-GoogleBigQuery -Connection $GoogleBigQuery -Columns @('OrderName','Freight') -Values @('MyOrderName', 'MyFreight') -Table Orders -Id "MyId"

ADO.NET

$cmd = New-Object System.Data.CData.GoogleBigQuery.GoogleBigQueryCommand("UPDATE Orders SET ShipCity='New York' WHERE Id = @myId", $conn)
$cmd.Parameters.Add((New-Object System.Data.CData.GoogleBigQuery.GoogleBigQueryParameter("@myId","10456255-0015501366")))
$cmd.ExecuteNonQuery()

Insert BigQuery Data

PowerShell

Add-GoogleBigQuery -Connection $GoogleBigQuery -Table Orders -Columns @("OrderName", "Freight") -Values @("MyOrderName", "MyFreight") 

ADO.NET

$cmd = New-Object System.Data.CData.GoogleBigQuery.GoogleBigQueryCommand("INSERT INTO Orders (ShipCity) VALUES (@myShipCity)", $conn)
$cmd.Parameters.Add((New-Object System.Data.CData.GoogleBigQuery.GoogleBigQueryParameter("@myShipCity","New York")))
$cmd.ExecuteNonQuery()

Delete BigQuery Data

PowerShell

Remove-GoogleBigQuery -Connection $GoogleBigQuery -Table "Orders" -Id "MyId"

ADO.NET

$cmd = New-Object System.Data.CData.GoogleBigQuery.GoogleBigQueryCommand("DELETE FROM Orders WHERE Id=@myId", $conn)
$cmd.Parameters.Add((New-Object System.Data.CData.GoogleBigQuery.GoogleBigQueryParameter("@myId","001d000000YBRseAAH")))
$cmd.ExecuteNonQuery()

CodeProject