Changelog folder created by Synapse Link

Deshmukh, Niyati 40 Reputation points

Hi,

Synapse Link is writing timestamped folders to Blob Storage from D365 as expected. The documentation states that consumers should not process the folder currently referenced in the changelog file.

I would like to clarify the behavior of files within that folder. While the folder is listed in the changelog and is still being populated:

Are only new files added to that folder?

Or can files that have already been written to the folder also be modified, overwritten, or appended to?

The reason I am asking is that I plan to implement event-based ingestion. A newly created file would trigger downstream processing, even though the containing folder may still be receiving additional files. I want to understand whether previously written files can change after their creation, or whether they can be considered final and immutable once written.

0 comments No comments
Sign in to comment

2 answers

  1. Pilladi Padma Sai Manisha 10,190 Reputation points • Microsoft External Staff • Moderator

    Hi @Deshmukh, Niyati
    Based on the public documentation, Synapse Link creates timestamped folders for incremental updates and advises consumers not to process the folder currently referenced in the changelog until it is no longer active.

    However, the documentation does not explicitly state whether files already written within the active folder are guaranteed to be immutable, or whether they may be modified while the export is still in progress.

    Because there is no documented immutability guarantee at the file level, the safest approach is to treat the entire active folder as incomplete and only process folders that are no longer referenced as the current folder in the changelog. This helps avoid race conditions in event-driven ingestion scenarios.

    If your design depends on individual file immutability, I would recommend validating the behavior in a test environment or opening a Microsoft support case for confirmation, as the public documentation primarily describes folder-level consumption behavior rather than file-level guarantees.

    Reference: Analyze incremental updates of your Dataverse data with Azure Synapse Link for Dataverse

    1. Pilladi Padma Sai Manisha 10,190 Reputation points • Microsoft External Staff • Moderator

      Hi @Deshmukh, Niyati
      Following up to see if the below answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

    Sign in to comment

  2. AI answer

    Within the current timestamped incremental folder created by Azure Synapse Link for Dataverse, files are not guaranteed to be immutable while that folder is still the “current folder” referenced in Changelog/changelog.info.

    Key documented behaviors:

    1. The Synapse Link service may be continuously updating data in the current folder. This is why consumers are explicitly advised not to process data from the current folder and instead to:
      • “Consume data from previous time stamped folders only” to avoid read–write conflicts with the Synapse Link service.
    2. Because the service can still be writing to the current folder, files in that folder must be treated as mutable until the folder is no longer marked as current in changelog.info. This includes the possibility of:
      • New files being added.
      • Existing files being updated/overwritten/appended as part of the ongoing batch publishing.
    3. Once a folder is no longer the current folder (i.e., changelog.info points to a newer timestamp), the previous folder can be treated as stable for consumption. The guidance is to build pipelines that:
      • Detect the latest non-current timestamped folder.
      • Process only those folders.
      • Optionally delete obsolete incremental folders older than 24 hours after processing, to avoid conflicts with system leases.

    For event-based ingestion, this means:

    • A file-created event in the current folder should not immediately trigger downstream processing that assumes immutability.
    • Instead, design the pipeline to trigger on or poll completed (non-current) timestamp folders, not on individual file events inside the current folder.

    References:

    AI-generated content may be incorrect. Read our transparency notes for more information.
Sign in to answer

Your answer