Skip to content

Crash (UAF) on Teardown after Upload #1391

Description

Describe your environment.

PAL implementation: CPP11
SDK version: EVT-3.4.317.0

Steps to reproduce.
We make use of the SDK for a lightweight telemetry client we build which simply sends all events in one shot. I seem to be hitting issues where events are not uploaded and remain "offline" when looking at the mat_debug logs it looks as if the API is not behaving properly is it appears the SDK does not properly shutdown as i do not see the final MATSDK.PAL|Shut down in the mat-debug logs.

The way we use the SDK is LogEvent in batch and then during our app teardown is call UploadNow and FlushAndTeardown.

Ive posted a sample of our destructor which performs Upload once we've finished all of our LogEvent calls.

TelemetryManager::~TelemetryManager() noexcept
{
    status_t status = m_logManager->UploadNow();
    if (STATUS_SUCCESS != status)
    {
        OsConfigLogError(m_log, "Telemetry upload during shutdown failed. status=%d", status);
    }
    std::this_thread::sleep_for(std::chrono::microseconds(10)); // Minimal sleep required for upload to be triggered properly
    m_logManager->FlushAndTeardown();
    OsConfigLogInfo(m_log, "Telemetry shutdown complete.");
}

Without the sleep there it appears to not teardown properly (my log is missing the final Telemetry shutdown complete) but with the sleep there, it works just fine and the mat-debug logs also show the final MATSDK.PAL|Shut down.

mat-debug.log (with sleep)

2025-11-12T22:55:53.317Z|00008792|W|MATSDK|upload  = 819 ms
2025-11-12T22:55:53.317Z|00008792|W|MATSDK|abort   = 0 ms
2025-11-12T22:55:53.317Z|00008792|W|MATSDK|stop    = 1 ms
2025-11-12T22:55:53.317Z|00008792|W|MATSDK|worker  = 0 ms
2025-11-12T22:55:53.317Z|00008792|W|MATSDK|storage = 0 ms
2025-11-12T22:55:53.317Z|00008792|D|EventsSDK.LogManager|Telemetry system stopped
2025-11-12T22:55:53.317Z|00008792|D|EventsSDK.RecordStats|RecordStats destroyed: 13
2025-11-12T22:55:53.317Z|00008792|D|EventsSDK.RecordStats|RecordStats destroyed: 14
2025-11-12T22:55:53.317Z|00008792|D|EventsSDK.RecordStats|RecordStats destroyed: 15
2025-11-12T22:55:53.317Z|00008792|D|EventsSDK.RecordStats|RecordStats destroyed: 16
2025-11-12T22:55:53.317Z|00008792|D|EventsSDK.RecordStats|RecordStats destroyed: 17
2025-11-12T22:55:53.317Z|00008792|D|EventsSDK.StorageHandler|LOCKGUARD   lockin at /workspaces/azure-osconfig/build2/adapters/mc/telemetry-external-prefix/src/telemetry-external-build/lib/1dscpplib-prefix/src/1dscpplib/lib/offline/OfflineStorageHandler.cpp:62
2025-11-12T22:55:53.317Z|00008792|D|EventsSDK.StorageHandler|LOCKGUARD   locked at /workspaces/azure-osconfig/build2/adapters/mc/telemetry-external-prefix/src/telemetry-external-build/lib/1dscpplib-prefix/src/1dscpplib/lib/offline/OfflineStorageHandler.cpp:62
2025-11-12T22:55:53.317Z|00008792|D|EventsSDK.LogManager|LOCKGUARD   lockin at /workspaces/azure-osconfig/build2/adapters/mc/telemetry-external-prefix/src/telemetry-external-build/lib/1dscpplib-prefix/src/1dscpplib/lib/api/LogManagerImpl.cpp:855
2025-11-12T22:55:53.317Z|00008792|D|EventsSDK.LogManager|LOCKGUARD   locked at /workspaces/azure-osconfig/build2/adapters/mc/telemetry-external-prefix/src/telemetry-external-build/lib/1dscpplib-prefix/src/1dscpplib/lib/api/LogManagerImpl.cpp:855
2025-11-12T22:55:53.317Z|00008792|D|MATSDK.PAL|Shutting down...
2025-11-12T22:55:53.317Z|00008792|I|MATSDK.PAL|queue item=0x7ffc4600e8e0
2025-11-12T22:55:53.317Z|00008792|D|MATSDK.PAL|LOCKGUARD   lockin at /workspaces/azure-osconfig/build2/adapters/mc/telemetry-external-prefix/src/telemetry-external-build/lib/1dscpplib-prefix/src/1dscpplib/lib/pal/WorkerThread.cpp:81
2025-11-12T22:55:53.317Z|00008792|D|MATSDK.PAL|LOCKGUARD   locked at /workspaces/azure-osconfig/build2/adapters/mc/telemetry-external-prefix/src/telemetry-external-build/lib/1dscpplib-prefix/src/1dscpplib/lib/pal/WorkerThread.cpp:81
2025-11-12T22:55:53.317Z|00008793|D|MATSDK.PAL|LOCKGUARD   lockin at /workspaces/azure-osconfig/build2/adapters/mc/telemetry-external-prefix/src/telemetry-external-build/lib/1dscpplib-prefix/src/1dscpplib/lib/pal/WorkerThread.cpp:189
2025-11-12T22:55:53.317Z|00008793|D|MATSDK.PAL|LOCKGUARD   locked at /workspaces/azure-osconfig/build2/adapters/mc/telemetry-external-prefix/src/telemetry-external-build/lib/1dscpplib-prefix/src/1dscpplib/lib/pal/WorkerThread.cpp:189
2025-11-12T22:55:53.317Z|00008792|I|MATSDK.PAL|Shut down
(END)

mat-debug.log (no sleep)

2025-11-12T23:08:49.395Z|00010035|I|MATSDK|offline records=340, pending uploads=0
2025-11-12T23:08:49.395Z|00010035|D|EventsSDK.MemoryStorage|LOCKGUARD   lockin at /workspaces/azure-osconfig/build2/adapters/mc/telemetry-external-prefix/src/telemetry-external-build/lib/1dscpplib-prefix/src/1dscpplib/lib/offline/MemoryStorage.cpp:467
2025-11-12T23:08:49.395Z|00010035|D|EventsSDK.MemoryStorage|LOCKGUARD   locked at /workspaces/azure-osconfig/build2/adapters/mc/telemetry-external-prefix/src/telemetry-external-build/lib/1dscpplib-prefix/src/1dscpplib/lib/offline/MemoryStorage.cpp:467
2025-11-12T23:08:49.395Z|00010035|D|EventsSDK.Storage|LOCKGUARD   lockin at /workspaces/azure-osconfig/build2/adapters/mc/telemetry-external-prefix/src/telemetry-external-build/lib/1dscpplib-prefix/src/1dscpplib/lib/offline/OfflineStorage_SQLite.cpp:925
2025-11-12T23:08:49.395Z|00010035|D|EventsSDK.Storage|LOCKGUARD   locked at /workspaces/azure-osconfig/build2/adapters/mc/telemetry-external-prefix/src/telemetry-external-build/lib/1dscpplib-prefix/src/1dscpplib/lib/offline/OfflineStorage_SQLite.cpp:925
2025-11-12T23:08:49.497Z|00010035|D|EventsSDK.MemoryStorage|LOCKGUARD   lockin at /workspaces/azure-osconfig/build2/adapters/mc/telemetry-external-prefix/src/telemetry-external-build/lib/1dscpplib-prefix/src/1dscpplib/lib/offline/MemoryStorage.cpp:467
2025-11-12T23:08:49.499Z|00010035|D|EventsSDK.MemoryStorage|LOCKGUARD   locked at /workspaces/azure-osconfig/build2/adapters/mc/telemetry-external-prefix/src/telemetry-external-build/lib/1dscpplib-prefix/src/1dscpplib/lib/offline/MemoryStorage.cpp:467
2025-11-12T23:08:49.499Z|00010035|D|EventsSDK.Storage|LOCKGUARD   lockin at /workspaces/azure-osconfig/build2/adapters/mc/telemetry-external-prefix/src/telemetry-external-build/lib/1dscpplib-prefix/src/1dscpplib/lib/offline/OfflineStorage_SQLite.cpp:925
2025-11-12T23:08:49.499Z|00010035|D|EventsSDK.Storage|LOCKGUARD   locked at /workspaces/azure-osconfig/build2/adapters/mc/telemetry-external-prefix/src/telemetry-external-build/lib/1dscpplib-prefix/src/1dscpplib/lib/offline/OfflineStorage_SQLite.cpp:925
2025-11-12T23:08:49.500Z|00010035|I|MATSDK|offline records=340, pending uploads=0
2025-11-12T23:08:49.500Z|00010035|D|EventsSDK.MemoryStorage|LOCKGUARD   lockin at /workspaces/azure-osconfig/build2/adapters/mc/telemetry-external-prefix/src/telemetry-external-build/lib/1dscpplib-prefix/src/1dscpplib/lib/offline/MemoryStorage.cpp:467
2025-11-12T23:08:49.500Z|00010035|D|EventsSDK.MemoryStorage|LOCKGUARD   locked at /workspaces/azure-osconfig/build2/adapters/mc/telemetry-external-prefix/src/telemetry-external-build/lib/1dscpplib-prefix/src/1dscpplib/lib/offline/MemoryStorage.cpp:467
2025-11-12T23:08:49.500Z|00010035|D|EventsSDK.Storage|LOCKGUARD   lockin at /workspaces/azure-osconfig/build2/adapters/mc/telemetry-external-prefix/src/telemetry-external-build/lib/1dscpplib-prefix/src/1dscpplib/lib/offline/OfflineStorage_SQLite.cpp:925
2025-11-12T23:08:49.500Z|00010035|D|EventsSDK.Storage|LOCKGUARD   locked at /workspaces/azure-osconfig/build2/adapters/mc/telemetry-external-prefix/src/telemetry-external-build/lib/1dscpplib-prefix/src/1dscpplib/lib/offline/OfflineStorage_SQLite.cpp:925
(END)

Ive also come across the article How to Not Crash on Shutdown in case of static LogManager instance
which sounds eerily familiar to this issue but even after using the suggestion (Flush, PauseActivity, WaitPause and finally FlushAndTeardown), the behavior does not change.

What is the expected behavior?
I expect, that even without the sleep, the application shuts down properly and mat-debug logs contain the MATSDK.PAL|Shut down trace at the end.

What is the actual behavior?
Without the sleep, it appears to crash the SDK

Additional context.
N/A

Activity

  1. bmehta001 commented on Sep 24, 2026

    @bmehta001
    Contributor

    Hi Ahmed Messaoud (@AhmedBM), Likely addressed by #1520: teardown and upload completion paths now drain/cancel requests safely, and the PR adds an in-flight-upload teardown regression test. The storage flush path also signals completion when shutdown prevents a pending flush from running. These changes target the reported UploadNow() then FlushAndTeardown() race, though they do not reproduce the original customer crash or prove the same root cause. Closing as likely resolved; please reopen with a current stack trace and logs if teardown still fails without the workaround sleep.

  2. self-assigned this
    on Sep 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions