Skip to content

Make FailureDetails.Properties optional for DataContractSerializer - #1431

Merged
Bernd Verst (berndverst) merged 5 commits into
Azure:mainfrom
WilliamHBonney:failuredetails-optionalfield
Nov 12, 2026
Merged

Bernd Verst (berndverst) merged 5 commits into
Azure:mainfrom
WilliamHBonney:failuredetails-optionalfield

Conversation

@WilliamHBonney

Copy link
Copy Markdown
Contributor

Problem

FailureDetails is a [Serializable] type that does not implement ISerializable. DataContractSerializer therefore treats every field as required. #1237 added the Properties auto-property, so payloads persisted before that change no longer deserialize. This affects Netherite commit logs and instance history, for example.

System.Runtime.Serialization.SerializationException: Error in line 1 position ...
Expecting element '_x003C_Properties_x003E_k__BackingField'.

This blocks upgrading apps that use a DataContractSerializer-based backend (for example Netherite) past DurableTask.Core 3.4.0 while existing task hubs contain failed activities, sub-orchestrations or orchestrations. Newer Microsoft.Azure.Functions.Worker.Extensions.DurableTask.AzureManaged packages require Core 3.8.0+, so side-by-side DTS + Netherite hosts hit this when upgrading.

Fix

Mark the Properties backing field [field: OptionalField]. DataContractSerializer then deserializes older payloads with Properties = null, and newer payloads still round-trip.

  • The serialized shape is unchanged, so data written by the new version is identical.
  • Implementing ISerializable instead was avoided because it changes the DCS wire format.
  • Newtonsoft.Json is unaffected: the [JsonConstructor] already defaults a missing properties to null.

I diffed the DCS-relevant types between Core 3.3.0 and 3.11.2. This was the only newly required member. The other additions are optional [DataMember]s or new rewind-only types and enum values.

Tests

Added FailureDetailsSerializationTests:

  • A TaskFailedEvent with nested FailureDetails and the Properties elements removed deserializes. Without the fix this fails with the error above.
  • FailureDetails with Properties round-trips through DataContractSerializer.

Passes on net8.0 and net48 together with the existing CustomExceptions / ExceptionHandling tests.

FailureDetails is [Serializable] without ISerializable, so DataContractSerializer treats every field as required. Data persisted before the Properties property was added (e.g. Netherite commit logs and history) fails to deserialize with 'Expecting element _x003C_Properties_x003E_k__BackingField'. Mark the backing field [OptionalField] so older payloads deserialize with Properties = null.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 4bf274ab-99ca-4766-831e-75f351459122
Copilot AI balanced review requested due to automatic review settings November 10, 2026 16:14
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The regression tests are under uppercase Test/ instead of the referenced lowercase test/ directory, so they are not compiled or discovered on case-sensitive checkouts.

1 open issue
What changed in this PR

This PR restores compatibility with older FailureDetails payloads used by DataContractSerializer-based backends, without changing the serialized shape.

Changes:

  • Marks the Properties backing field optional and documents why.
  • Adds tests for legacy nested failures and round-tripping populated properties.
File Description
Test/​DurableTask.Core.Tests/​FailureDetailsSerializationTests.cs Adds legacy-deserialization and properties round-trip tests.
src/​DurableTask.Core/​FailureDetails.cs Allows deserialization when the Properties field is absent.

Review effort: Balanced

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 4bf274ab-99ca-4766-831e-75f351459122
Copilot AI balanced review requested due to automatic review settings November 10, 2026 16:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block the change restoring compatibility with legacy persisted failure data.

1 open issue

Review effort: Balanced

@WilliamHBonney

Copy link
Copy Markdown
Contributor Author

 @microsoft-github-policy-service rerun .

@WilliamHBonney
WilliamHBonney marked this pull request as draft November 10, 2026 17:20
@WilliamHBonney
WilliamHBonney marked this pull request as ready for review November 10, 2026 17:20
@WilliamHBonney

Copy link
Copy Markdown
Contributor Author

@microsoft-github-policy-service agree company="Microsoft"

@WilliamHBonney

Copy link
Copy Markdown
Contributor Author

@microsoft-github-policy-service rerun

@WilliamHBonney

Copy link
Copy Markdown
Contributor Author

@microsoft-github-policy-service agree

Copilot AI balanced review requested due to automatic review settings November 11, 2026 10:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.

@WilliamHBonney

Copy link
Copy Markdown
Contributor Author

Davide Montanari (@davidemontanari) Bernd Verst (@berndverst) This resolves a breaking change from 2025 that means the netherite backend can't be upgraded. Could you have a look please?

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are recorded for the change restoring compatibility with legacy FailureDetails payloads.

No open issues 🎉

Review effort: Balanced

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: de587369-3d13-49fa-8654-999bf350e3c4
Copilot AI balanced review requested due to automatic review settings November 12, 2026 08:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified for the change restoring compatibility with older FailureDetails payloads.

No open issues 🎉

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings November 12, 2026 10:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the change restoring compatibility with legacy FailureDetails payloads.

No open issues 🎉

Review effort: Balanced

@berndverst Bernd Verst (berndverst) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found no breaking change or blocking defect introduced by this PR at revision 3177e39. This is a narrowly scoped backward-compatibility fix, and I would consider it safe to merge.

The production change adds [field: OptionalField] to the existing backing field. It does not remove the field, stop serializing it, or change the public property.

Impact across the repository

Area Assessment
Core / downstream SDK consumers Public signatures, constructors, field layout and assembly identity remain unchanged. No source or binary compatibility break identified.
Azure Storage — default mode Newtonsoft.Json output and deserialization remain unchanged. The field attribute does not affect its normal property-based contract.
Azure Storage — DataContract mode Its opt-in DataContractJsonSerializer path also benefits: missing historical Properties fields become acceptable. Existing output remains unchanged.
Service Bus Normal JSON messages are unchanged. The legacy DCS reader becomes more backward-compatible.
Service Fabric JSON proxy contracts remain unchanged; DCS-based persisted object graphs receive the same compatibility improvement.
Emulator, ApplicationInsights, samples and solutions No API adaptations, dependency changes, target-framework changes or project-reference changes are required.

The same conclusion applies to activity failures, sub-orchestration failures, orchestration completion records and orchestration state: they all reference the same FailureDetails type. Nested failures benefit as well.

Actual side effects and caveats

  • Missing data is now accepted intentionally. Historical payloads deserialize with Properties = null instead of throwing. This is consistent with the property's existing nullable contract. Other required fields remain required, and retry/non-retriable behavior is unchanged.
  • Exported XML schema changes slightly. The backing-field element becomes optional (minOccurs="0"). Schema-generation or schema-diff tooling can observe this, but serialized XML and JSON remain unchanged—including null, empty and populated dictionaries.
  • Rolling upgrades remain asymmetric for historical data. Patched output remains readable by the pre-PR version. However, an unpatched worker still cannot read historical records missing Properties; this PR neither rewrites those records nor fixes readers that have not been upgraded.
  • This does not certify every older-Core-to-latest upgrade. Unrelated version changes, custom serializers and arbitrary objects stored inside Properties retain their existing compatibility constraints.

Bottom line: this relaxes reading without changing writing. I found no repo SDK, project or solution that needs a corresponding change because of this PR.

@berndverst
Bernd Verst (berndverst) merged commit fac86dd into Azure:main Nov 12, 2026
3 checks passed
Bernd Verst (berndverst) added a commit that referenced this pull request Nov 12, 2026
… fix (#1436)

* Prepare compatibility patch releases

Bump Core and five companion packages to ship the FailureDetails deserialization compatibility fix from #1431.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Exclude Service Fabric and Emulator from patch release

Limit coordinated patch releases to Core, Azure Storage, Service Bus, and Application Insights.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Bernd Verst <beverst@microsoft.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants