12,173 questions with Azure Data Factory tags
How to implement a dynamic trigger interval in ADF without creating infinite loops or pipeline concurrency issues?
Hello everyone, I am currently working on optimizing an existing data pipeline in Azure Data Factory (ADF) and I am facing an architectural challenge regarding dynamic scheduling. Here is my current setup: The pipeline is currently triggered by a fixed…
ADF Trigger Schedules
Can we schedule the trigger for once in last working day of the month?
Does the Azure Data Factory HubSpot connector v2.0 (Preview) support incremental load?
The HubSpot connector 1.0 supported incremental loads by means of passing a query. The new 2.0 connector does not support queries. Does that mean the new connector does not support incremental loads, or is there some other option? Fetching all of the…
How to break ForEach loop in ADF
Hi everyone, I have a ForEach activity containing a Copy Data task. The Problem: By default, if one iteration fails, the ForEach continues to trigger all other iterations. In my case, if the first one fails, the following ones are guaranteed to fail as…
Data Flow source test connection to Cosmos DB fails with InvalidSubscriptionId error via Managed VNet Private Endpoint
Azure Data Factory Data Flow fails to connect to Azure Cosmos DB (SQL API) when using Managed Virtual Network with a Managed Private Endpoint. The Copy Activity works correctly with the same linked service and private endpoint configuration. Error…
Not able to create ADF due regional constraint in student free tier account
Not able to create Data Factory due to the below error-
Azure Data Factory Private endpoint for Portal Endpoint
Hi Team, We are looking for recommendations and best practices around configuring the portal private endpoint for Azure Data Factory in a multi-VNET environment. Current setup: We have multiple ADF instances owned by separate teams. Each ADF has its…
Azure Data Factory Copy activity failing due DNS resolution issue
We have a Data Factory in which our Data Engineers have created a copy activity to pull data from a SQL server which is hosted inside a Azure VM and this VM also act as a SHIR and the target is a storage account and we the same VM as the SHIR. The VM and…
Azure Data Factory Dataverse to SQL - Timeout failures when creating table.
I'm getting repeated and consistent timeout failures during the DataverseToSQL pipeline step of the following guide. The process fails due to a timeout when attempting to create a table on the SQL server. What can I do to resolve this? …
Autoupdate on SHIR seems to have re-nabled and forced a restart of the SHIR service
On April 27th we had an issue with batch jobs that caused delays on key business process, which drew quite a bit of attention. After some initial investigation it was found that the overnight jobs failed because the SHIR they were running on had the SHIR…
Timestamp being logged incorrectly some of the time.
I have an ADF pipeline that starts by writing utcnow to a string pipeline variable, then performs operations with assorted casts to different form factors with formatdatetime, including several script blocks that log to a snowflake log table, all based…
Data factory - Azure Key Vault (Preview) Activity - bug?
The Azure Key Vault Activity in data factory (in preview) does not do anything - settings tab just says azure-key-vault-activity-settings works! Is this expected?
Any changes happened in the backend of Parse JSON Mapping data flow in ADF in Feb Week 2
Hi All, Is anyone else faced an issue with Json array not being parsed correctly in the ADF Mapping data flows. Earlier the string worked fined and able to parse the columns but it now shows Null Json Array is like [{"abc":"def",…
Schema Enforcement Strategy for Salesforce to Blob to Data Flow Pipelines
We have a common Azure Data Factory pipeline that extracts Salesforce data via SOQL, stages it as JSON in Blob Storage, and then loads it through Mapping Data Flows. We observed a case where a field such as BillingPostalCode was interpreted as integer…
Can a single ADF pipeline trigger multiple Azure ML pipelines in parallel when their inputs and outputs are independent?
I have an Azure Data Factory pipeline that orchestrates a multi-stage classification workflow. Right now every activity is chained one after another with success dependencies, so the whole pipeline runs strictly sequentially and takes a long time…
Azure Data Factory: Inquiry on New Undocumented Property 'trustModeClaimForMi'
It looks like PG added a new undocumented property on data factory resource properties: "trustModeClaimForMi": { "value": "Enabled", "editable": true } Any ideas about this and any ETA when…
How to copy data from google secops platform using ADF
Hi there, How to copy data from google secops platform using ADF Thanks & Best regards, zmsoft
Some of the CDS connection copy activities in the Azure Data Factory pipelines have been stuck in the queue for a long time.
Today, the copy activities using the CDS connection have been stuck in the queue for a long time. Previously, these activities used to complete within minutes, but today they have been in the queue state for 3–4 hours.
ADF Pipeline issues [Sev 2 ] [MAQ Software] - [ADF Pipeline Execution Delay – Pipelines Persistently Queued During Refresh Window]
Issue Description: We are currently facing an issue where multiple ADF pipelines are remaining in the Queued state for an extended duration during 5/29 refresh process. Pipelines that normally complete within a few minutes are now queued for more than 2…
Storage firewall blocking Snowflake stage access for ADF Oracle to Snowflake pipeline
Hi Azure Team, Our ADF pipeline (Oracle to Snowflake) is failing during Snowflake load with stage access error. Observed error: Operation on target Oracle to Snowflake_2024_v1 failed with SnowflakeOperationError. Failure using stage area. Status Code:…