SQL Snapshot Replication Fails with BCP EOF / NULL Errors (Works in Transactional Replication)
Issue:
We have Snapshot Replication on a production server that was working fine but recently started failing with the below error:
Agent message code 20037. The process could not bulk copy into table "XXX".
Message: End of file reached, terminator missing or field data incomplete
Message:Batch send failed
Message: Failed to send batch after max errors
More details:
Publisher & Distributor: Microsoft SQL Server 2017 (RTM-CU31-GDR) (KB5084818) - 14.0.3525.1 (X64) Standard Edition on Windows Server 2016 Datacenter
Subscriber:
- Microsoft SQL Server 2022 (RTM-CU24-GDR) (KB5083252) - 16.0.4250.1 (X64) Standard Edition on Windows Server 2022 Datacenter
- Microsoft SQL Server 2017 (RTM-CU31-GDR) - 14.0.3525.1 (X64) Standard Edition on Windows Server 2016 Datacenter
What we tried:
- Dropped table on subscriber + deleted snapshot folder + reinitialized β worked temporarily
- Issue reoccurred; same steps no longer fix it
- Created new Snapshot Replication β same failure for this table
- Created Transactional Replication with same table β works fine
Issue occurs only in Snapshot Replication
Findings:
β’ Suspected issue with datetime2(7) and SNAC 11.0 (BCP native mode)
β’ Snapshot Agent appears to generate multiple .bcp files (parallel processing)
Quick Solution Workaround (working):
- To avoid generating multiple .bcp files per table (parallel processing), we added a row filter (WHERE 1 = 1) to the affected articles.
Questions:
- Is this a known issue with Snapshot Replication multi-threaded BCP?
- Does SNAC 11.0 have limitations with datetime2(7)?
- As we cannot add the same filter to every table (due to scale), will upgrading to OLE DB / ODBC Driver 17 or 18 resolve this?
- Is there a way to disable BCP partitioning globally (without using row filters)?
-
Erland Sommarskog 134.7K Reputation points β’ MVP β’ Volunteer Moderator
I'm afraid that to troubleshoot this, we would need a repro. That is, a scripts that creates a database, a table in the database and populates it with data. Before sharing the script with us, you should verify that you get the issues you mention. Since this started to happen recently, there is probably something in your data that has crept in recently.
-
Parag Nimbhore 5 Reputation points
Hi Erland,
Thanks for your response.
We understand the need for a reproducible script. However, we would like to share a few additional observations which may help narrow this down:
- The same table replicates successfully using Transactional Replication, but consistently fails under Snapshot Replication.
- The issue is resolved when we force Snapshot Agent to generate a single BCP file by adding a row filter (
WHERE 1 = 1), which disables parallel BCP generation. - This suggests the issue may be related to multi-threaded BCP file generation rather than purely data content.
- The error pattern (
EOF / terminator missing / NULL insert) appears consistent with incomplete or misaligned BCP data, possibly due to parallel file generation.
Given this behavior, we are trying to understand:
- Whether there are known issues with parallel BCP generation in Snapshot Replication
- Or if this could be related to SNAC 11.0 limitations (especially with datetime2(7))
Any guidance on Snapshot Agent behavior, BCP partitioning, or known limitations would be very helpful.
Thanks again for your time
-
Erland Sommarskog 134.7K Reputation points β’ MVP β’ Volunteer Moderator
This suggests the issue may be related to multi-threaded BCP file generation rather than purely data content.
Then again, you say that "was working fine but recently started failing", which suggests that there is something special in your environment/data that triggers this behaviour. I'm not ruling out that there is a bug of some sort, but I'm not going to set up something out of thin air to troubleshoot. To that end, I need to have something which is known to fail.
Rather than relying on volunteer support in peer-to-peer forum, it may be better to open a support case with Microsoft. When you work with a support engineer you can work directly with the case at hand, and you don't have to spend to time to produce a repro. On the other hand, it can certainly be more costly. Then again, if Microsoft agrees that this is a bug in the product, the case should be of no charge to you. (Although a pesky detail here is that the publisher is a version that is in extended support.)
-
Akhil Gajavelly 1,830 Reputation points β’ Microsoft External Staff β’ Moderator
Hi @Parag Nimbhore ,
Just checking in Erland has provided the best possible guidance here given the scenario.
Since this behavior started recently and isnβt reproducible in a controlled script, it does point to an environment/data-specific edge case. At this stage, a Microsoft support case would be the most effective way to investigate further.
https://support.serviceshub.microsoft.com/supportforbusiness/createIf you do identify a minimal repro later, feel free to share the community would be happy to take another look.
Thanks,
Akhil. -
Akhil Gajavelly 1,830 Reputation points β’ Microsoft External Staff β’ Moderator
Hi @Parag Nimbhore ,
Following up to see if you had a chance to review the earlier guidance.
At this point, without a reproducible script or additional diagnostic details, there may not be much more the community can isolate from this side. As suggested earlier, opening a Microsoft support case would likely be the best path for deeper investigation into the environment-specific behavior.
If you later manage to capture a minimal repro or additional error details, please feel free to share them here and the community can revisit the issue.
Thanks,
Akhil.
Sign in to comment
1 answer
-
David Yard 0 Reputation points
The WHERE 1=1 workaround confirming single-file BCP resolves it is a strong diagnostic signal β this points to a race condition or data alignment issue in parallel BCP file generation under Snapshot Replication, not a data content problem per se.
To answer your specific questions:
Parallel BCP partitioning β Snapshot Agent splits large articles into multiple .bcp files when MaxBcpThreads or article row counts trigger parallel processing. There is no global switch to disable this without row filters or reducing MaxBcpThreads to 1 in the Snapshot Agent profile, which is worth trying before upgrading drivers.
SNAC 11.0 and datetime2(7) β SNAC has known limitations with high-precision temporal types in native BCP mode. This is a legitimate suspect, especially if the table recently had datetime2 columns added or populated with non-null high-precision values.
Driver upgrade β Switching to ODBC Driver 17 or 18 is the right long-term move and has resolved similar BCP encoding issues for others. It won't disable partitioning but handles datetime2 correctly in native mode.
Immediate thing to test β Set MaxBcpThreads = 1 in the Snapshot Agent profile for this publication and retest without the row filter. If that stabilizes it, you've isolated parallel BCP as the cause and can escalate to Microsoft with that confirmation.
