Skip to content

Background Thread Correlation

aryehcitron@gmail.com edited this page Nov 5, 2026 · 22 revisions

Background processing — change feed processors, message handlers, hosted services, timer triggers — runs on threads where the test framework's execution context is unavailable. This page explains how to correlate those operations back to the originating test.

Event-driven architectures? If your SUT is triggered entirely by consuming messages (no direct HTTP calls from the test), see Event-Driven Architecture Testing for the complete end-to-end guide. This page covers the lower-level mechanisms used under the hood.

Parallel test execution? If you need parallel-safe background correlation (v2.36.0+), see Parallel-Safe Background Correlation for TestCorrelationStore — a thread-safe alternative to GlobalFallback that requires no production code changes.


The Problem

CurrentTestInfo.Fetcher (xUnit3, and equivalents in other frameworks) uses TestContext.Current.Test to determine the active test. This only works on the test runner thread — any background thread, task continuation, or callback will see TestContext.Current.Test as null, and the fetcher will throw InvalidOperationException, which the resolver takes as "no test here". Every framework's fetcher does so outside a test; xUnit v2's and NUnit's since 4.13.1.

Common scenarios where this occurs:

Scenario Why TestContext is null
Cosmos DB Change Feed Processor Polls on a dedicated background thread
Azure Functions timer triggers Executes on a thread pool thread
Hosted services (IHostedService) Runs on a separate thread
Event handlers (e.g. AfterPublish) May fire on thread pool threads
Read-through cache refresh HTTP calls from a caching layer's background refresh
Task.Run() / thread pool work Any work dispatched to the thread pool

Symptoms

  • Diagnostic report shows a large number of "Unknown" entries
  • Sequence diagrams for tests that trigger background processing are missing operations
  • Service Bus publishes triggered by change feed events don't appear in diagrams

The scenario's calls and the host's (3.17.0)

A web host built inside a test (a WebApplicationFactory in the test body, or a per-test fixture) inherits the test's execution context, and every hosted service, timer and message consumer the host starts inherits it in turn. Their calls resolved the test's identity for as long as the host lived, so a scenario's report carried an outbox processor's polls and other scenarios' messages, and its set of calls changed with the timing of a loop. 3.17.0 closes this from four sides, all of them automatic:

  1. Provenance. Every capture records how it got its scenario: attributionSource on each interaction in the data files (RequestHeader, TestContext, Scope, GlobalFallback, Detached, None, Expired, from 3.19.0 DocumentOwner, and from 3.20.0 DocumentFlow). A stated identity (a header, a Begin scope, a message) is distinguishable from an inherited one.
  2. A scenario's end. Every adapter records endedAt. At report time a call that inherited a scenario's identity and started after the scenario ended is re-attributed to no scenario: its attributionSource becomes Expired and expiredFrom names the scenario it was taken from. A pair follows its request. The report lists them under a root background block (calls, afterScenarioEnd, interactions), a Background calls section in the HTML, and a BackgroundCalls diagnostic per scenario. The history line, the digest and the component diagram read the same re-attributed snapshot.
  3. Detached hosted services. services.DetachHostedServicesFromTestIdentity() wraps every IHostedService so it starts, runs and stops in a detached flow (TestIdentityScope.Detach()): the loop inherits no identity, and neither does anything it starts. AddTestTrackingContextPropagation() calls it, so nothing changes for a consumer already propagating context. A message that names its scenario (SetFromMessage, Begin) still wins inside a detached flow. Call it after the hosted services are registered; ConfigureTestServices runs after the application's own registrations.
  4. Message identities end with the message. TestIdentityScope.ClearMessageIdentity() clears an identity SetFromMessage established and leaves one Begin established alone. The Kafka consumer calls it at every poll, whatever the poll returns (4.16.2+: before, an empty poll or a partition end kept the last message's identity), the Service Bus and Event Hubs wrappers before they establish the next message's identity, the MassTransit observer after the consumer ran, and the Pub/Sub subscriber scopes the identity to the handler it owns, so a message that carries no identity is nobody's rather than the previous message's.

RequestResponseLogger.CaptureBackground = true (default off) keeps a call the resolver could not attribute, under the unknown identity with its source, instead of dropping it; such calls appear in the same background block, under (no scenario) in the HTML table, and the section's heading counts them apart from the calls that expired (3.18.0). TestIdentityScope.Detach() is also usable directly around any long-lived infrastructure a test starts.

A document a scenario wrote is the scenario's (3.19.0)

Detaching a host's services leaves one kind of call with no scenario that has one: the work a scenario seeded and waited for. An outbox scenario writes a row through the SUT and waits for the host's processor to claim it, retry it and mark it failed; detached, the processor's replaces belong to nobody, and the scenario's diagram shows the seed and the polling reads alone. The Cosmos and Mongo trackers already record every attributed write in TestCorrelationStore (AutoCorrelateWrites, the store the change-feed and change-stream decorators read). From 3.19.0 they also read it: a document operation that resolved no scenario and names a document a scenario wrote is attributed to that scenario, the document's last attributed writer, for that one call, with attributionSource: "DocumentOwner".

  • The operation itself establishes nothing ambient: the poll before the claim stays the host's; the claim, the retry updates and the failed-status update land where the seed did. What the flow does between two operations on the document is the owner window's, below (3.20.0).
  • Only identity-less calls are candidates. A call that resolved by header, scope, context or fallback is never re-attributed, so the mechanism cannot take a call away from a flow that has an identity.
  • A query is never a candidate: it names no document.
  • A call attributed this way after its owner ended expires like an inherited one, into the background block with expiredFrom, so a scenario keeps what happened during its life and no more.

AttributeByDocumentOwner = false on CosmosTrackingMessageHandlerOptions or MongoDbTrackingOptions switches it off. DocumentOwnership.Resolve(resolved, key) and TestCorrelationStore.Lookup(key) are public for any other capturer that knows which document a call named. Spanner does not take part: its capture exposes no key.

The flow that wrote a scenario's document is doing its work (3.20.0)

Ownership by document leaves one call with no scenario: the one that names no document. An outbox processor claims the scenario's row, dispatches it through an HTTP client or a publisher, and updates the row's status; the claim and the update name the row, the dispatch names nothing. From 3.20.0 a detached flow that wrote a scenario's document keeps that scenario for what it does next, with the provenance attributionSource: "DocumentFlow", until its next operation on a document that is not the scenario's.

  • A write opens the window (Cosmos: create, upsert, replace, patch, delete; Mongo: insert, update, delete, findAndModify, bulk write) on a document with an attributed owner, made by a flow with no scenario of its own, once the write succeeded. The write itself is the owner's per call, as in 3.19.0.
  • While it is open, a call in the flow that resolves nothing else (a header, a Begin scope, a message identity all win) resolves to the owner: the dispatch, a publish, a call through any tracked client.
  • Any operation on the owner's document confirms what the flow did since the window opened, a read included; the window stays open.
  • A query ends it (a Cosmos query or listing; a Mongo find, aggregate, count, distinct, getMore or mapReduce naming no _id): it names no document, and it is the poll before the next claim. So does an operation on another scenario's document; a write on that one opens a window for its owner.
  • A document nobody owns, or one not yet known (a create names its document in the body; a multi-document write names none), leaves the window as it is, and is not the owner's either.
  • A read never opens a window: a poller reading a scenario's document every few hundred milliseconds must not make everything between its reads the scenario's.

Held until confirmed. What the window answers for is not recorded when it is captured: RequestResponseLogger holds a DocumentFlow entry on the flow, the next operation on the owner's document releases it into the store in its original place, with its original timestamp and under the step open when it was logged, and a foreign operation or a query drops it (kept under (no scenario) when CaptureBackground is on; whatever is still held at report time is settled as nobody's). A call is therefore attributed only when the same document closes it out with nothing else between: two workers that share one flow and interleave drop both dispatches, which is what 3.19.0 did with them, and a sequential loop over a batch attributes every one. There is no timeline in which one scenario's call lands in another scenario.

Only a detached flow holds a window. The tracker that learns a document's owner runs inside the database SDK's async call, and an AsyncLocal value set there is restored when the call returns; so TestIdentityScope.Detach() keeps the flow's state on a holder object its callees share, and TestIdentityScope.OwnerWindow reads it. Every hosted service already runs detached (DetachHostedServicesFromTestIdentity, called by AddTestTrackingContextPropagation). Outside a detached flow ownership stays per call.

Mongo's claim by filter. findAndModify({status: Pending} -> {status: Processing}) names no document until the server answers with the one it took: the subscriber defers the request half until the reply, attributes both halves by the document the reply names (keeping the request's time), and opens the window.

A call attributed by the window after its owner ended expires like an inherited one. AttributeByDocumentOwner = false switches the window off with the rest; DocumentOwnership.ForOperation(resolved, key, kind) and DocumentOwnership.AfterWrite(identity) are the two calls a capturer makes around a document operation. The store registers only attributed writers, and an owner whose id is the unknown one is no owner.

Solution 1: TestIdentityScope (v2.28.5+)

TestIdentityScope is an AsyncLocal-based ambient scope that propagates test identity across async boundaries. It is the recommended approach when you control the code that dispatches background work.

// Before dispatching background work, push the test identity into scope:
using (TestIdentityScope.Begin(testName, testId))
{
    await TriggerBackgroundProcessing();
    // All tracking within this scope (and any async continuations) will use testName/testId
}

Automatic Context Propagation Middleware (v3.0.19+)

For SUT code that dispatches work via Task.Run or similar fire-and-forget patterns within an HTTP request, you can automatically establish TestIdentityScope from the incoming request's tracking headers using AddTestTrackingContextPropagation():

builder.ConfigureTestServices(services =>
{
    services.AddTestTrackingContextPropagation();
});

This registers TestTrackingContextMiddleware (via IStartupFilter) which reads the test-tracking-current-test-name and test-tracking-current-test-id headers from each incoming request and calls TestIdentityScope.Begin(name, id) for the duration of the request. Because AsyncLocal propagates into Task.Run, any background work dispatched during that request will inherit the test identity automatically — no manual TestIdentityScope.Begin() calls needed.

Idempotent (v3.0.20+): AddTestTrackingContextPropagation() is safe to call more than once — for example, from both a base WebApplicationFactory and a derived one. Only one TestTrackingContextStartupFilter will be registered regardless of how many times the method is called.

When to use this: Your SUT receives an HTTP request from the test, then dispatches fire-and-forget work (e.g. Task.Run(() => httpClient.PostAsync(...))) within that request. Without this middleware, the background thread would have no HttpContext and no TestIdentityScope, causing the tracking handler to silently skip logging. With it, the AsyncLocal scope flows into the background thread.

Resolution order for all trackers (v2.28.16+):

Priority Source When Available
1 HTTP request headers (IHttpContextAccessor) Code running inside the SUT's request pipeline (e.g. WebApplicationFactory scenarios). Both test-tracking-current-test-name and test-tracking-current-test-id headers must be present; if only one is present the handler falls through to Priority 2.
2 CurrentTestInfoFetcher delegate Code running on the test thread where the framework's TestContext is accessible
3 TestIdentityScope.Current Code wrapped in TestIdentityScope.Begin(...), set via SetFromMessage(), or established by TestTrackingContextMiddleware (v3.0.19+) — propagates via AsyncLocal
4 Message headers (kronikol-test-name / kronikol-test-id) Consumer/subscriber receives a message produced during a tracked test (v2.34.0+)
5 TestIdentityScope.GlobalFallback Pre-existing background threads (Change Feed Processor, Hangfire, hosted services)
6 Returns null Operation is silently not logged, unless RequestResponseLogger.CaptureBackground is on (3.17.0), when it is kept under the unknown identity with its attributionSource and listed in the report's background block

v2.28.22+: Priority 2 (CurrentTestInfoFetcher) now throws InvalidOperationException when invoked outside a test context. The resolver catches this and falls through to priorities 3–6. See Tracking Custom Dependencies › Test Context Availability for details. xUnit v2's and NUnit's fetchers did not throw until 4.13.1: outside a test they answered with an id no scenario owns (xUnit v2 a new random id on every call, NUnit the fixture's id or, on a thread the test did not reach, an ad hoc one), so priorities 3–6 were never reached and the call appeared in no report.

Scenario Resolution Reference

Scenario Resolves At Why
HTTP request inside WebApplicationFactory Priority 1 TestTrackingMessageHandler propagates test identity as HTTP headers
gRPC call into a host Priority 1 or 3 GrpcTrackingInterceptor puts the same headers into the call's metadata (4.13.0+). A tracker in the host that holds its IHttpContextAccessor reads them (Priority 1); for one without, the scope the host's middleware begins from them answers (Priority 3)
Test method body (direct call) Priority 2 Framework TestContext is available on the test thread
Task.Run / background thread wrapped in TestIdentityScope.Begin Priority 3 AsyncLocal propagates to async continuations
Kafka/ServiceBus/EventHubs/PubSub consumer processing a tracked message Priority 4 Message headers carry test identity from producer to consumer
Change Feed Processor / hosted service with SetGlobalFallback Priority 5 Pre-existing thread reads the static fallback
Background thread with nothing configured Priority 6 Returns null — operation not tracked

See Tracking Custom Dependencies › Tracking Background Processing with TestIdentityScope for full details and examples.


Solution 1b: Automatic Message Header Propagation (v2.34.0+)

For event-driven architectures where an ASP.NET application listens to messages (Kafka, Service Bus, Event Hubs, Google Pub/Sub, MassTransit), test identity is automatically propagated through message headers — no manual TestIdentityScope.Begin() or GlobalFallback needed.

When a tracked producer sends a message during a test, the kronikol-test-name and kronikol-test-id headers are injected into the message metadata. When a tracked consumer receives that message, the headers are extracted and established as the ambient TestIdentityScope via SetFromMessage().

How It Works

Test Thread                    Producer                         Consumer (Background Thread)
    │                              │                                │
    │── Produce("order-created") ──│                                │
    │   [headers injected:         │                                │
    │    kronikol-test-name + id]       │                                │
    │                              │── Message on topic ───────────>│
    │                              │                                │── Headers extracted
    │                              │                                │── TestIdentityScope.SetFromMessage()
    │                              │                                │── All tracking resolves to test

Supported Extensions

Extension Producer Injects Consumer Extracts Header Location
Kafka TrackingKafkaProducer TrackingKafkaConsumer message.Headers (byte[])
Service Bus TrackingServiceBusSender TrackingServiceBusReceiver message.ApplicationProperties
Event Hubs TrackingEventHubProducerClient TrackingEventHubConsumerClient eventData.Properties
Google Pub/Sub TrackingPublisherClient TrackingSubscriberClient message.Attributes
MassTransit TrackingSendObserver / TrackingPublishObserver TrackingConsumeObserver Transport headers

Configuration

Propagation is enabled by default. To disable it (e.g. for performance-sensitive benchmarks):

var options = new KafkaTrackingOptions
{
    PropagateTestIdentity = false,  // Disable header injection/extraction
    // ...
};

All messaging extension option classes expose the same PropagateTestIdentity property.

When to Use This vs Other Solutions

Scenario Recommended Solution
Event-driven app consuming Kafka/ServiceBus/EventHubs/PubSub Automatic propagation (v2.34.0+) — zero config needed
Background processing triggered by HTTP request HTTP headers (Priority 1) — already works via WebApplicationFactory
Task.Run / manual thread dispatch TestIdentityScope.Begin() — wraps the dispatch point
Pre-existing hosted service threads GlobalFallback — for threads started before the test

Solution 2: Instance-Scoped Test Tracker

When you cannot use TestIdentityScope (e.g. you don't control the dispatch point), use an instance-scoped tracker that explicitly sets the active test identity on the fixture:

public class ActiveTestTracker
{
    private readonly object _syncLock = new();
    private (string Name, string Id)? _activeTest;

    public void Set(string name, string id)
    {
        lock (_syncLock) { _activeTest = (name, id); }
    }

    public void Clear()
    {
        lock (_syncLock) { _activeTest = null; }
    }

    /// <summary>
    /// Tries TestContext.Current first (works on the test thread),
    /// falls back to the explicitly set active test (works on background threads).
    /// </summary>
    public (string Name, string Id) Fetcher()
    {
        var (name, id) = CurrentTestInfo.Fetcher();
        if (name != "Unknown")
            return (name, id);

        lock (_syncLock)
        {
            return _activeTest ?? ("Unknown", "unknown");
        }
    }
}

Wiring It Up

1. Create the tracker on your fixture

public class MyCollectionFixture : IAsyncLifetime
{
    public ActiveTestTracker TestTracker { get; } = new();

    public async ValueTask InitializeAsync()
    {
        // Pass TestTracker.Fetcher to ALL tracking configuration:

        // Cosmos tracking
        var cosmosOptions = new CosmosTrackingMessageHandlerOptions
        {
            ServiceName = "CosmosDB",
            CallerName = "My Service",
            CurrentTestInfoFetcher = TestTracker.Fetcher,
        };

        // HTTP tracking
        services.TrackDependenciesForDiagrams(new TestTrackingMessageHandlerOptions
        {
            CallerName = "My Service",
            CurrentTestInfoFetcher = TestTracker.Fetcher,
        });

        // Message tracking
        services.TrackMessagesForDiagrams(new MessageTrackerOptions
        {
            CallerName = "My Service",
            ServiceName = "Service Bus",
            CurrentTestInfoFetcher = TestTracker.Fetcher,
        });
    }

    public ValueTask DisposeAsync() => ValueTask.CompletedTask;
}

2. Set/clear in the test lifecycle

public class MyTestBase : IAsyncLifetime
{
    protected MyCollectionFixture Fixture { get; }

    public ValueTask InitializeAsync()
    {
        // Set BEFORE test runs — background threads from here on will resolve correctly
        // ⚠️ Safe here because InitializeAsync runs on the test thread.
        // v2.28.22+: Fetcher() throws InvalidOperationException if called
        // outside a test context (e.g. background thread or fixture constructor).
        var (name, id) = CurrentTestInfo.Fetcher();
        Fixture.TestTracker.Set(name, id);
        return ValueTask.CompletedTask;
    }

    public ValueTask DisposeAsync()
    {
        // Clear BEFORE logging — prevents stale attribution
        Fixture.TestTracker.Clear();

        DiagrammedTestRun.TestContexts.Enqueue(TestContext.Current);
        return ValueTask.CompletedTask;
    }
}

Why Instance-Scoped (Not Static)

A static tracker breaks under parallel execution. Each fixture instance must own its tracker:

  • Per-test fixtures: New fixture per test → new tracker → no contention
  • Per-class fixtures: Tests within a class run sequentially (xUnit guarantee) → safe
  • Per-collection fixtures: Tests within a collection run sequentially → safe
  • Parallel collections: Each collection has its own fixture → own tracker → no cross-contamination

Solution 3: GlobalFallback (v2.28.16+)

When pre-existing background threads (Change Feed Processor polling threads, Hangfire workers, hosted service loops) were started before TestIdentityScope.Begin(), the AsyncLocal value never propagates to them. GlobalFallback is a process-wide static fallback that these threads can read.

public class MyTestBase : IAsyncLifetime
{
    protected MyCollectionFixture Fixture { get; }

    public ValueTask InitializeAsync()
    {
        var (name, id) = CurrentTestInfo.Fetcher();
        // AsyncLocal scope for new async continuations:
        _scope = TestIdentityScope.Begin(name, id);
        // Static fallback for pre-existing threads:
        TestIdentityScope.SetGlobalFallback(name, id);
        return ValueTask.CompletedTask;
    }

    public ValueTask DisposeAsync()
    {
        _scope?.Dispose();
        TestIdentityScope.ClearGlobalFallback();
        DiagrammedTestRun.TestContexts.Enqueue(TestContext.Current);
        return ValueTask.CompletedTask;
    }

    private IDisposable? _scope;
}

Resolution order (v2.28.16+): Same as the table above — HTTP headers → delegate → scope → GlobalFallback → null.

⚠ Parallel execution warning: GlobalFallback is a single process-wide value. It only works correctly when tests sharing the same background infrastructure run serially (e.g. within an xUnit collection fixture). This matches the existing ActiveTestTracker pattern's semantics — if your tests already run in parallel with shared infrastructure, use Solution 1 (TestIdentityScope.Begin) instead.

When to Use GlobalFallback vs ActiveTestTracker

Scenario Recommended
Pre-existing Change Feed Processor threads GlobalFallback — simplest, no custom tracker class needed
Hosted services started before test GlobalFallback — eliminates CurrentTestInfoFetcher boilerplate
Multiple independent message trackers ActiveTestTracker — each tracker can have its own fetcher
Need to distinguish between "no test" and "test active" ActiveTestTracker — Fetcher() can conditionally return Unknown

Understanding "Unknown" Entries

When a tracking handler captures an operation but cannot determine which test it belongs to, the operation is counted as Unknown in the diagnostic report.

Expected Unknown Entries

Some Unknown entries are normal and expected:

  • Background processing: Change feed processors, message handlers, hosted services — they have no HttpContext or test scope
  • Application startup/teardown: Operations during WebApplicationFactory initialisation or disposal happen outside test scope
  • Health checks and middleware: Periodic background operations from infrastructure

Unexpected Unknown Entries

If all entries for a component are Unknown, investigate:

Symptom Likely Cause Fix
All Cosmos operations are Unknown IHttpContextAccessor not wired to the tracking handler See [[Integration CosmosDB Extension › Using with CosmosDB.InMemoryEmulator
All outgoing HTTP calls are Unknown IHttpContextAccessor not passed to TestTrackingMessageHandler See [[HTTP Tracking Setup › Dual-Resolution Test Identity (v2.23.0+)
Unknown count matches total IHttpContextAccessor was null at handler construction time Use the LazyHttpContextAccessor pattern (see below)

Healthy Benchmarks

A typical diagnostic report shows:

  • 0% Unknown for inbound HTTP handlers (API controller calls)
  • 0% Unknown for synchronous outbound HTTP calls (typed HttpClients)
  • 30-80% Unknown for Cosmos/Service Bus when async processing is involved
  • 100% Unknown for components used only in background processing

LazyHttpContextAccessor Pattern

When CosmosTrackingMessageHandler (or any extension handler) is constructed before IHttpContextAccessor is available (common with WebApplicationFactory where the Cosmos client is a singleton built during DI), use a lazy wrapper:

internal sealed class LazyHttpContextAccessor : IHttpContextAccessor
{
    private IHttpContextAccessor? _inner;

    public HttpContext? HttpContext
    {
        get => _inner?.HttpContext;
        set { if (_inner is not null) _inner.HttpContext = value; }
    }

    public void SetInner(IHttpContextAccessor inner) => _inner = inner;
}

Usage:

// 1. Create wrapper and assign to options BEFORE building the Cosmos client
var lazyAccessor = new LazyHttpContextAccessor();
cosmosTrackingOptions.HttpContextAccessor = lazyAccessor;

// 2. Build client — handler captures the lazyAccessor reference
var cosmos = InMemoryCosmos.Builder()
    .WrapHandler(h => new CosmosTrackingMessageHandler(cosmosTrackingOptions, h))
    .Build();

// 3. Wire the real accessor after WebApplicationFactory is available
lazyAccessor.SetInner(factory.Services.GetRequiredService<IHttpContextAccessor>());

Important: The wrapper must be assigned to options before calling WrapHandler. Setting options.HttpContextAccessor afterwards has no effect — the handler has already captured the value at construction time.


Verification

After implementing either solution, check the Diagnostic Report with DiagnosticMode = true. The "Unknown" count should either disappear or be reduced to expected background operations (e.g. initial container setup before any test starts).

The generated report can also answer this directly: kronikol query steps s3 counts the calls that belong to no step, and kronikol query interactions --group-by step buckets unattributed calls under -, so the interactions that lost attribution are listed rather than guessed at; see Querying Reports.

Home · 🔍 Search this wiki Full-text search across every page. The “Pages” list above matches page and heading names, not page content.


Demo


Getting Started

Common Tasks

Integration Guides

Uninstrumentable / polyglot backends

Extensions

Configuration

Features

Reference

Clone this wiki locally