How to copy data from google secops platform using ADF

Hi @zmsoft

To copy data from Google SecOps (Chronicle) into Azure Data Factory (ADF), there is currently no native Google SecOps connector available in ADF.

The recommended approach depends on how the SecOps data is exposed.

If the Google SecOps data is exported to Google Cloud Storage (GCS), Azure Data Factory can ingest the data using the Google Cloud Storage connector.

Prerequisite: Ensure that Google SecOps is configured to export data to a customer-managed GCS bucket (for example using Google SecOps Data Export APIs). Without exporting the data first, ADF cannot directly access Google SecOps data.

Steps:

1. Prepare GCS access

• Enable interoperability access for the GCS bucket
• Create a service account with appropriate storage permissions
• Generate HMAC credentials (Access Key ID and Secret Access Key)

1. Create Linked Service in ADF

• Create a Google Cloud Storage linked service
• Configure:
  • Service URL: https://storage.googleapis.com
    • HMAC credentials
      • Integration Runtime (Azure IR or Self-hosted IR if required)
      • Validate the connection

1. Create Dataset

• Configure the bucket and folder/file path
• Use wildcard patterns if required

1. Create Copy Pipeline

• Source: Google Cloud Storage dataset
• Sink: Azure Blob Storage / ADLS Gen2 / Azure SQL / Synapse
• Configure mappings and transformations if needed

1. Execute and Monitor

• Trigger the pipeline
• Monitor execution and performance in ADF

Alternative Approach: If GCS export is not available, ADF can use REST connector integration with Google SecOps APIs. This approach requires handling authentication, pagination, and API limitations and is generally less suitable for large-scale ingestion scenarios.

Recommendation: For production scenarios, exporting Google SecOps data to GCS and ingesting through ADF is typically the most scalable and operationally manageable approach.

Reference:
https://learn.microsoft.com/en-us/azure/data-factory/connector-google-cloud-storage?tabs=data-factory
https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-overview
https://learn.microsoft.com/en-us/azure/data-factory/connector-overview

Please let us know if you have any questions or concerns.

Hey zmsoft,

It sounds like you want to bring data out of a Google SecOps-hosted bucket into Azure via Data Factory. The easiest path is to treat your SecOps data as files in Google Cloud Storage (GCS) and use ADF’s Google Cloud Storage connector (S3-API compatible). Here’s a high-level recipe:

1. Prepare your GCS side • Enable interoperability on the GCS bucket where your SecOps data lives. • Create a Google service account with at least storage.objects.get and storage.objects.list roles (and storage.buckets.list if you want UI browsing). • Generate an HMAC access key (AccessKeyID + SecretAccessKey).
2. Create a GCS Linked Service in ADF • In the Azure portal, open your Data Factory → Manage → Linked services → New. • Search “Google Cloud Storage (S3 API)” and select it. • Fill in: – serviceUrl: https://storage.googleapis.com – accessKeyId / secretAccessKey (or reference Key Vault secrets) – choose your Integration Runtime (Azure IR or Self-hosted IR if needed) • Test the connection and Create.
3. Build your source dataset • In ADF Authoring, add a new Dataset → choose Google Cloud Storage. • Point to your bucket and folder or file path. You can use wildcards (e.g. /logs/**/.json).
4. Build your destination (sink) dataset • This could be Blob Storage, ADLS Gen2, SQL Database, Synapse, etc. • Configure the linked service and file/folder/table details accordingly.
5. Author the Copy Activity • In a pipeline, drop in a Copy Data activity. • Source: point to your GCS dataset. • Sink: point to your destination dataset. • (Optional) Adjust schema mappings, enable staged copy, filter rows, etc.
6. Run & Monitor • Trigger the pipeline manually or schedule it. • Check Monitoring to see throughput, errors, performance stats.

Pro tip: If you just want a quick wizard to scaffold everything—including integration runtime, linked services, datasets and pipeline—launch the Copy Data tool in the ADF UI (aka Copy Wizard). It’ll walk you through source, sink, mappings and scheduling without hand-coding JSON.

Hope this gives you a clear path forward—let me know if you need more details!

Reference list

• Copy Data Tool (Wizard)

https://docs.microsoft.com/azure/data-factory/copy-data-tool

• ADF Copy Activity overview

https://docs.microsoft.com/azure/data-factory/copy-activity-overview

• Supported connectors in Azure Data Factory

https://docs.microsoft.com/azure/data-factory/connector-overview

• Copy data from Google Cloud Storage using ADF

https://docs.microsoft.com/azure/data-factory/connector-google-cloud-storage?wt.mc_id=knowledgesearch_inproduct_azure-cxp-community-insider#prerequisites

–and–

https://docs.microsoft.com/azure/data-factory/connector-google-cloud-storage?wt.mc_id=knowledgesearch_inproduct_azure-cxp-community-insider#create-a-linked-service-to-google-cloud-storage-using-ui