redb.Tsak.Core.Pro 4.7.2

Prefix Reserved

dotnet add package redb.Tsak.Core.Pro --version 4.7.2

NuGet\Install-Package redb.Tsak.Core.Pro -Version 4.7.2

This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.

<PackageReference Include="redb.Tsak.Core.Pro" Version="4.7.2" />

For projects that support PackageReference, copy this XML node into the project file to reference the package.

<PackageVersion Include="redb.Tsak.Core.Pro" Version="4.7.2" />
                    

                            Directory.Packages.props

<PackageReference Include="redb.Tsak.Core.Pro" />
                    

                            Project file

For projects that support Central Package Management (CPM), copy this XML node into the solution Directory.Packages.props file to version the package.

paket add redb.Tsak.Core.Pro --version 4.7.2

The NuGet Team does not provide support for this client. Please contact its maintainers for support.

#r "nuget: redb.Tsak.Core.Pro, 4.7.2"

#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.

#:package redb.Tsak.Core.Pro@4.7.2

#:package directive can be used in C# file-based apps starting in .NET 10 preview 4. Copy this into a .cs file before any lines of code to reference the package.

#addin nuget:?package=redb.Tsak.Core.Pro&version=4.7.2
                    

                            Install as a Cake Addin

#tool nuget:?package=redb.Tsak.Core.Pro&version=4.7.2
                    

                            Install as a Cake Tool

The NuGet Team does not provide support for this client. Please contact its maintainers for support.

redb.Tsak

The runtime container for redb.Route integration pipelines. Deploy DLLs into a folder. Start, stop, reload them at runtime. Scale across nodes with leader election. Watch every message, every error, every restart — from a built-in REST API, the tsak CLI, or a Blazor dashboard. Zero downtime. Production-tested.

LLMs / coding agents: canonical training contract for the whole redb ecosystem → https://redbase.app/llms.txt

What you get out of the box · Why redb.Tsak exists · For redb.Route users
Architecture · Project structure
Quick start: Docker images, standalone archive, build from source
Module workflow · 5-layer configuration
REST API · CLI · Web dashboard
Cluster mode · Watchdog · Lifecycle & graceful shutdown · Storage modes
Security · Observability (health probes, logs, tracing) · Quartz scheduler
Three deployment recipes · Docker · Kubernetes deployment
Testing · Frequently asked questions · Roadmap
Upgrading: UPGRADING.md, CHANGELOG.md

Guides: DEPLOYMENT.md · MODULE_DEPLOYMENT.md · CONFIG_GUIDE.md · PARAMETERS.md · API_GUIDE.md

What you get out of the box


Module-based deployment	Drop a package or a `.dll` into `modules/` — Tsak loads it, builds an `IRouteContext`, starts the routes. Update the DLL — Tsak hot-swaps it without dropping a single in-flight message in other contexts.
Three deployment modes	`Standalone` (in-memory, no DB) · `Single-node + redb` (durable RTTI-storage state) · `Cluster` (leader election + automatic context redistribution across nodes).
Three API modes	`Full` (REST API + management) · `Headless` (zero ports, embedded use) · `Readonly` (only `GET` endpoints — perfect for monitoring sidecars).
5-layer configuration	Module ships defaults. DevOps overrides. Config hot-reloads. No code changes, no restarts.
Built-in observability	Per-process metrics (CPU/RAM/threads/GC, 12h history), per-route metrics (count/error rate/latency), ring-buffer logs, OpenTelemetry traces, optional Prometheus scrape.
Watchdog	Detects suspected and hung routes. Optionally auto-restarts them.
Quartz scheduler	A scheduler per context, as Apache Camel has one per CamelContext — modules never collide on job names. `RAMJobStore` standalone, `AdoJobStore` cluster-safe — schema auto-created on first start.
Security	API Key + HMAC-SHA256 + roles + expiry + revocation. Constant-time comparison. redb-backed key store. Optional user binding (disabled user → key dies).
Module isolation	Per-module `AssemblyLoadContext` — dependencies don't conflict between modules.
Clients ready to ship	Typed C# `ITsakApiClient`, the `tsak` CLI with profiles and JSON output, Blazor Server dashboard.

"redb.Tsak — .NET analogue of Apache Camel K".

Why redb.Tsak exists

Most .NET teams either:

Bake their integration code into a Worker Service — every change means redeploy the entire process. Multiple unrelated pipelines fight in the same Program.cs.
Buy a heavyweight enterprise ESB — pay six figures for a UI, a runtime, a management API, and a deployment workflow they barely use.
Roll their own management plane — and re-invent metrics, hot-reload, leader election, and an admin API for the third time this decade.

Tsak is the missing piece between "a Worker Service" and "an enterprise ESB":

	Hand-rolled Worker	Tsak	Heavy ESB
Deploy a new pipeline	Redeploy whole process	`cp module.tpkg modules/` (hot-reload)	Vendor wizard, hours
Stop one pipeline without affecting others	Custom code	`tsak context stop orders`	Vendor UI
Distribute pipelines across nodes	Custom coordinator	Built-in leader election + auto-rebalance	Vendor cluster
REST API for ops/CI	Build it yourself	69 endpoints, typed client	Yes, vendor-locked
CLI for CI/CD	Build it yourself	`tsak` CLI, profiles, JSON output	Maybe
Web dashboard	Build it yourself	Blazor Server dashboard	Yes, vendor-locked
Cost	Engineering time	Apache 2.0, free	$$$$ + lock-in
Routing engine	DIY or MassTransit	redb.Route — 22 transports, 24 EIP processors, fluent DSL	Vendor's DSL

For redb.Route users

If you already write RouteBuilder classes for redb.Route, Tsak is what runs them in production. You do not change a single line of route code.

┌──────────────────────┐         ┌──────────────────────┐         ┌──────────────────────┐
│   Your routes        │  --->   │   class library      │  --->   │  Tsak picks it up    │
│   (RouteBuilder)     │ publish │   (Orders.dll)       │   cp    │  hot-reload + run    │
└──────────────────────┘         └──────────────────────┘         └──────────────────────┘

What Tsak adds on top of a plain redb.Route worker:

Pure `redb.Route` worker (`Host.CreateApplicationBuilder`)	`redb.Route` inside Tsak
One `Program.cs` wires every `RouteBuilder` at compile time.	Drop the package into `modules/`, Tsak loads it.
Add a route → redeploy the whole process.	Hot-reload one module without touching the others.
One process = one bag of routes.	Multiple named contexts, each with its own properties, lifecycle and assembly load context.
Stop one route = stop the process.	`tsak route stop <ctx> <id>` or `POST /api/routes/{id}/stop`.
Distribute across nodes = roll your own coordinator.	Built-in leader election + per-context assignment across cluster nodes.
Schedule a job = wire Quartz yourself in `Program.cs`.	Every context gets its own scheduler with its first `cron:` route, schema auto-bootstrapped.
Operate it = parse logs, build a dashboard, expose metrics yourself.	REST + CLI + Blazor + Prometheus + OTel out of the box.

A RouteBuilder you wrote against redb.Route's plain IHostedService runs unchanged inside a Tsak module — the same Configure() is called, the same IExchange flows through, the same OnException and .Transacted() semantics apply. Tsak only owns how the routes are loaded, scheduled, observed and torn down.

// MyRoutes/InitRoute.cs — the only Tsak-specific piece
public static class InitRoute
{
    public static IRouteContext main(IRouteContext context)
    {
        // Plain redb.Route — exactly what you would write outside Tsak
        ((RouteContext)context).AddRoutes(new OrderRoutes());
        ((RouteContext)context).AddRoutes(new ShipmentRoutes());
        return context;
    }
}

For the routing DSL itself (transports, EIP patterns, expressions, transactional pipelines, OnException, error handling), see the redb.Route README. Everything below in this document is about the container. redb.Route now includes an Amazon SQS + SNS connector (redb.Route.Sqs) and a Telegram Bot connector (redb.Route.Telegram) — both bundled into the Tsak distribution at release.

Architecture

Process layout

┌────────────────────────────────────────────────────────────────────┐
│                        redb.Tsak.Worker                            │
│                                                                    │
│  ┌──────────────────────────────────────────────────────────────┐ │
│  │  _system context (protected, cannot be stopped or removed)   │ │
│  │  ────────────────────────────────────────────────────────── │ │
│  │  HTTP listener  →  AuthMiddleware  →  ControllerDispatcher   │ │
│  │  /api/auth        Bearer / X-Api-Key   AuthController         │ │
│  │  /api/contexts    role check           ContextsController     │ │
│  │  /api/modules                          ModulesController      │ │
│  │  /api/cluster                          ClusterController      │ │
│  │  /api/system                           SystemController       │ │
│  │  /api/scheduler                        SchedulerController    │ │
│  │  /api/logs                             LogsController         │ │
│  │  /api/users                            UsersController        │ │
│  │  /api/watchdog  /api/diagnostics  /api/lifecycle  /api/routes │ │
│  └──────────────────────────────────────────────────────────────┘ │
│                                                                    │
│  ┌────────────────┐  ┌────────────────┐  ┌────────────────┐      │
│  │   Context A    │  │   Context B    │  │   Context C    │      │
│  │   "orders"     │  │   "payments"   │  │   "analytics"  │      │
│  │ ────────────── │  │ ────────────── │  │ ────────────── │      │
│  │ Module:        │  │ Module:        │  │ Modules:       │      │
│  │  Api.Orders    │  │  Pay.Stripe    │  │  Etl.Loader    │      │
│  │ Route 1: Kafka │  │ Route 1: HTTP  │  │  Etl.Reports   │      │
│  │ Route 2: Cron  │  │ Route 2: AMQP  │  │ Route 1..N     │      │
│  │ Route N: HTTP  │  │ Route N: Mail  │  │                │      │
│  │ ALC: isolated  │  │ ALC: isolated  │  │ ALC: isolated  │      │
│  └────────────────┘  └────────────────┘  └────────────────┘      │
│                                                                    │
│  ┌──────────────────────────────────────────────────────────────┐ │
│  │  Infrastructure services                                      │ │
│  │  ────────────────────────────────────────────────────────── │ │
│  │  HotReloadService   ClusterCoordinator    QuartzScheduler    │ │
│  │  ModuleRegistry     LeaderElection        WatchdogService    │ │
│  │  ContextManager     NodeRegistry          MetricsService     │ │
│  │  ConfigMerger       AssignmentManager     LogRingBuffer      │ │
│  │                                            HealthCheckService │ │
│  └──────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
                ┌─────────────────────────────────┐
                │  redb (Postgres or MSSQL)   │
                │  - API keys (RedbApiKeyStore)   │
                │  - Cluster topology (Tree)      │
                │  - Lifecycle events             │
                │  - Idempotent state             │
                │  - Quartz QRTZ_* tables         │
                └─────────────────────────────────┘

Anatomy of a request

External call:  curl -H "Authorization: Bearer $KEY" http://node:9090/api/contexts/orders/start
       │
       ▼
HTTP listener (Kestrel) inside _system context — port 9090
       │
       ▼
HeaderBridge processor      → normalizes route.path = "contexts/orders/start", route.method = "POST"
       │
       ▼
AuthorizeProcessor          → resolves API key (5min cache) → validates role → enriches Exchange with claims
       │
       ▼
ControllerDispatcher        → looks up [Route("contexts/{name}/start")] on ContextsController
       │
       ▼
ContextsController.Start    → ITsakContextManager.StartAsync("orders")
       │
       ▼
                            → graceful start: load assemblies → wire routes → connect transports → mark Running
       │
       ▼
JSON response               → { "name": "orders", "status": "Running", "startedAt": "..." }

The same ContextsController can also be invoked over RabbitMQ RPC, gRPC, SignalR, or any other redb.Route InOut transport — the dispatcher is transport-agnostic. This is how Tsak avoids tying its management API to HTTP-only.

Project structure

redb.Tsak/
├── src/
│   ├── redb.Tsak.Core/          Kernel: contracts, controllers, services,
│   │                            security, cluster, hot-reload, monitoring
│   ├── redb.Tsak.Core.Pro/      Pro extensions (cluster, hooks, monitoring)
│   ├── redb.Tsak.Worker/        Hosted process: DI wiring, Serilog,
│   │                            Quartz, appsettings.json, Dockerfile
│   ├── redb.Tsak.Contracts/     Wire DTOs (System.Text.Json only) shared
│   │                            between Core, Client, Web, CLI
│   ├── redb.Tsak.Client/        ITsakApiClient + TsakApiClient (HTTP)
│   ├── redb.Tsak.CLI/           tsak CLI, profiles, JSON output
│   ├── redb.Tsak.Web/           Blazor Server dashboard, custom design system
│   └── redb.Tsak.Web.Pro/       Pro web extensions (auth, node providers)
├── tests/
│   ├── redb.Tsak.Tests/         492 unit + HTTP integration tests
│   └── redb.Tsak.CLI.Tests/     83 CLI command tests
├── docs/
│   ├── PLAN.md                  Architecture reference
│   ├── CONFIG_GUIDE.md          5-layer configuration deep-dive
│   ├── LT_TSAK_ANALYSIS.md      Analysis of the original lt.tsak
│   ├── ENTERPRISE_AUDIT.md
│   └── phases/                  Per-phase implementation notes (0..8B)
├── README.md                    This file
├── API_GUIDE.md                 REST API: endpoint map, pipeline, auth, probes
├── MODULE_DEPLOYMENT.md         Module upload/rollback trust model & signing
├── CONFIG_GUIDE.md              5-layer configuration model
├── PARAMETERS.md                Every Tsak:* setting, defaults, and the traps
├── DEPLOYMENT.md                Running the published images
├── CHANGELOG.md                 Per-version changes
├── CONTRIBUTING.md              Contribution guide
├── SECURITY.md                  Security policy
└── LICENSE                      Apache 2.0

Quick start

Option A — Docker (fastest)

Pre-built images are published to GitHub Container Registry — no build step, no .NET SDK required:

# Worker only (REST API + cluster, no UI)
docker run -d --name tsak \
  -p 9090:9090 \
  -v $PWD/modules:/app/modules \
  ghcr.io/redbase-app/redb-tsak-worker:latest

# Stack (Worker + Web UI in one container, like rabbitmq:management)
docker run -d --name tsak \
  -p 9090:9090 -p 8085:8085 \
  -e Tsak__Modules__AssemblyPaths__0=/app/worker/modules \
  -v $PWD/modules:/app/worker/modules \
  ghcr.io/redbase-app/redb-tsak-stack:latest
# UI: http://localhost:8085  ·  REST: http://localhost:9090

Mount your modules into the dedicated modules/ directory, never over Libs/: Libs/shared holds the image's own redb and connector libraries, and a mount over it hides them.

Image	Contains	Size	Ports
`ghcr.io/redbase-app/redb-tsak-worker`	Worker (REST + cluster)	~360 MB	`9090`
`ghcr.io/redbase-app/redb-tsak-web`	Standalone Web UI	~250 MB	`8085`
`ghcr.io/redbase-app/redb-tsak-stack`	Worker + Web (supervisord)	~430 MB	`9090`, `8085`

Tags: latest, <version>, <version>-net10.

With PostgreSQL (durable RTTI-storage state, multi-node, cluster):

docker run -d --name tsak \
  -p 9090:9090 -p 8085:8085 \
  -e Tsak__Modules__AssemblyPaths__0=/app/worker/modules \
  -v $PWD/modules:/app/worker/modules \
  -e ConnectionStrings__Postgres="Host=pg;Port=5432;Database=redb;Username=postgres;Password=postgres" \
  ghcr.io/redbase-app/redb-tsak-stack:latest

A compose file for the stack image is in Docker → docker compose; the full deployment guide is DEPLOYMENT.md.

Option B — Standalone archive (no Docker)

Self-contained archives (no .NET runtime required on host) are attached to every GitHub Release:

File	Platform
`redb-tsak-<version>-linux-x64.tar.gz`	Linux x64
`redb-tsak-<version>-win-x64.zip`	Windows x64

Each archive bundles worker/, web/, cli/, the Route connectors and the redb.* framework in worker/Libs/shared/, and startup scripts (start-worker.sh/.bat/.ps1, same for web and stack).

Shared runtime layer — swappable redb.* libraries

The Tsak binary is deliberately thin: only redb.Tsak.* (+ redb.Licensing) live in the app bin. The redb.* framework and providers (redb.Core(.Pro), redb.Route.*, redb.Postgres/MSSql/SQLite (.Pro)) and every Route connector live in worker/Libs/shared/, byte-loaded at startup before any redb type is touched.

Patch a library without rebuilding Tsak. A binary-compatible fix of a redb leaf/provider/connector — or a new beta connector — is delivered by dropping its DLL into worker/Libs/shared/. No host rebuild, no archive re-spin. Use scripts/refresh-shared.ps1 -Lib <name> -SharedDir <archive>/worker/Libs/shared, then re-zip / re-cosign the archive.
Fail-fast + compat-gate. A missing/corrupt framework DLL aborts startup with a precise message rather than failing later under load; the shared redb.* minor must match the archive's Tsak minor (patches may differ) or startup is refused.
What's loaded, really. GET /api/system/assemblies (admin) reports every loaded assembly with its version and origin (shared / bin / runtime).

VERSION=<version>   # the release to install, as on the releases page
curl -LO https://github.com/redbase-app/redb-tsak/releases/download/v$VERSION/redb-tsak-$VERSION-linux-x64.tar.gz
tar xzf redb-tsak-$VERSION-linux-x64.tar.gz
cd redb-tsak-$VERSION-linux-x64
./scripts/start-stack.sh   # worker on :9090, web on :8085

Verifying signatures (recommended)

All images and archives are signed with cosign. The public key, cosign.pub, is attached to every release.

# Image
cosign verify --key cosign.pub ghcr.io/redbase-app/redb-tsak-worker:$VERSION

# Archive
cosign verify-blob --key cosign.pub \
  --bundle redb-tsak-$VERSION-linux-x64.tar.gz.bundle \
  redb-tsak-$VERSION-linux-x64.tar.gz

SHA256 sums for every artifact are in checksums.txt on the release page.

Option C — Build from source

git clone https://github.com/redbase-app/redb-tsak
cd redb-tsak/src/redb.Tsak.Worker
dotnet run

1. Run a Tsak worker (no database, in-memory)

Any option above starts a worker with in-memory storage and no cluster. Its REST API answers on port 9090, and it scans the modules/ directory for module packages (Tsak:Modules:AssemblyPaths):

curl http://localhost:9090/api/health/live
# {"status":"ok"}

2. Deploy your first module

A Tsak module is a plain .NET class library exposing one of two well-defined entry-point shapes: a public static class InitRoute with public static IRouteContext main(IRouteContext ctx) or its async form public static async Task<IRouteContext> main(IRouteContext ctx) (Apache Camel-style convention, shown below), or a concrete public type implementing ITsakModule. An async main is awaited, and the context starts only after it completes. An InitRoute whose main fits neither form is reported as an error when the assembly is scanned, not skipped. Inside the entry point you wire up your RouteBuilder subclasses against the supplied IRouteContext.

// MyRoutes/InitRoute.cs
public static class InitRoute
{
    public static IRouteContext main(IRouteContext context)
    {
        ((RouteContext)context).AddRoutes(new OrderRoutes());
        return context;
    }
}

// MyRoutes/OrderRoutes.cs
public class OrderRoutes : RouteBuilder
{
    protected override void Configure()
    {
        From("kafka://orders?brokers=localhost:9092&groupId=svc")
            .Filter(e => e.In.GetHeader<string>("type") == "new")
            .Process(async (e, ct) => await ProcessAsync(e, ct))
            .To("rabbitmq://processed?host=localhost");
    }
}

# Build and deploy
dotnet publish MyRoutes -c Release -o /opt/tsak/modules/   # straight into the module directory

# Tsak picks it up within HotReload:ScanIntervalSeconds (default 10s)

tsak context list
# Name      Status   Routes  Endpoints  Modules
# _system   Running       11          1   (system)
# MyRoutes  Running        1          2   MyRoutes

3. Manage from the CLI

tsak profile add --name local --url http://localhost:9090 --key $TSAK_KEY
tsak profile use local
tsak system health
tsak context restart MyRoutes
tsak log get --level Error --limit 50
tsak system metrics
tsak scheduler jobs
tsak cluster status

4. Open the dashboard

cd redb-tsak/src/redb.Tsak.Web
dotnet run
# Browse to http://localhost:5180 (the stack image serves it on 8085)

Module workflow

Tsak supports two equivalent deployment formats in its module directory (modules/, set by Tsak:Modules:AssemblyPaths). Both are scanned by HotReloadService, hot-reloaded on file change, gracefully unloaded on file removal — pick whichever fits your workflow.

Format	What it is	When to pick it
Bare directory	A folder of loose `*.dll` files (+ optional config)	Local dev, fast inner loop, atomic file replace via `cp -r`
`.tpkg` package	A single ZIP archive bundling manifest + DLLs + config	CI/CD, immutable artifacts, one-file deploys, atomic rollback

What Tsak actually loads

Tsak does not load arbitrary .NET DLLs. Each candidate assembly is scanned for one of two well-defined module shapes:

A concrete public type implementing ITsakModule.
A public static class named InitRoute exposing a static IRouteContext main(IRouteContext ctx) method, or its async form static Task<IRouteContext> main(IRouteContext ctx) (Apache Camel-style entry point convention). Any other signature of main, or both forms at once, fails the scan of that assembly with a message naming the type and the two supported forms.

If neither shape is found, the assembly is classified as a dependency, not a module:

It is registered in LoadedAssemblyTracker so that other modules in the process can resolve it as a transitive dependency (same Assembly instance everywhere — no type-identity drift across ALCs).
Its file timestamp is recorded in an internal "ignored" set; subsequent scans skip it unless the file is replaced with a newer version (which triggers a re-scan in case the new bits do contain a module).

So the rule is: random DLLs are not loaded as modules — but DLLs that other modules depend on are loaded and made available, just not started as routes. This keeps the runtime predictable and prevents random vendor SDKs sitting in modules/ from being treated as deployable units.

Format 1 — Bare directory layout

A directory of Tsak:Modules:AssemblyPaths is scanned at its top level, so a bare module's files sit right in it. To keep one folder per module, list each folder in AssemblyPaths.

modules/
├── Orders.dll                  ← module entry point (contains ITsakModule or InitRoute.main)
├── Orders.deps.json            ← dependency graph (from dotnet publish)
├── Orders.Domain.dll           ← companion dependency (no module → loaded as dep only)
├── Newtonsoft.Json.dll         ← any third-party dep that isn't already in the host
├── context.json                ← infrastructure defaults (Layer 3) — optional, for the modules of this directory
└── Orders.config.json          ← module business settings (Layer 4) — optional

How the loader treats each file:

File	Role
`Orders.dll`	Loaded into a per-module isolated `ModuleAssemblyLoadContext`, scanned for `ITsakModule`/`InitRoute.main`, registered as a module
`Orders.Domain.dll`	Scanned, no module shape found → kept as a shared dependency (resolvable by other ALCs)
`*.deps.json`	Used by `dotnet publish` to record the full dep graph (Tsak does not parse it directly, but it is what makes `CopyLocalLockFileAssemblies=true` work)
`context.json`	Layer 3 of the 5-layer config merge
`{Module}.config.json`	Layer 4 of the same merge

Format 2 — `.tpkg` package (atomic, immutable, isolated)

A .tpkg is a plain ZIP archive bundling everything a module needs into a single file. Drop one file into modules/, Tsak picks it up and loads it as one atomic unit. Replace or delete that one file, the whole bundle reloads or shuts down together — no half-state in the middle.

Orders.tpkg                          (ZIP archive, flat structure)
├── manifest.json                    ← REQUIRED — package metadata
├── Orders.dll                       ← entry point (declared in manifest.EntryPoints)
├── Orders.config.json               ← optional Layer 4 config
├── Orders.Domain.dll                ← companion dependency (loaded into shared tracker)
├── Orders.Models.dll                ← companion dependency
└── Newtonsoft.Json.dll              ← any other transitive dep not provided by the host

manifest.json schema (matches redb.Tsak.Core.Modules.ModuleManifest):

{
  "Name": "Orders",
  "Version": "1.0.0",
  "EntryPoints": [ "Orders.dll" ],
  "Dependencies": []
}

Field	Meaning
`Name`	Logical package name. Must be unique across all packages loaded in the process.
`Version`	Informational version string used in logs and diagnostics.
`EntryPoints`	List of DLL filenames inside the archive that contain `ITsakModule` or `InitRoute.main`. Each entry is loaded into the package's isolated ALC and scanned for modules.
`Dependencies`	Other packages this one depends on, by their `Name`. The contexts that run modules of those packages start first, both when the worker starts and on a hot reload. A name no package of the batch carries changes nothing: that package is already running or not deployed. Not a loading constraint.

A single .tpkg may declare multiple entry points — for example a Core module and an Api module that ship together and share a private dependency tree:

{
  "Name": "tsum",
  "Version": "1.0.0",
  "EntryPoints": [ "tsum.Core.dll", "tsum.Api.dll" ],
  "Dependencies": []
}

Both entry-point DLLs land in the same isolated ALC, so they share static state and can pass non-public types between each other without serialization.

Isolation model — what is shared, what is not

Tsak runs many modules in one .NET process without letting them collide. The isolation model has three layers:

Layer	What lives here	Sharing semantics
Default ALC (host)	Tsak Worker, redb.Core, redb.Route abstractions, all NuGet deps the host ships with	Singleton — all modules see the same `IRouteContext`, `IRedbService`, etc. type identity. This is what makes interfaces work across the boundary.
`LoadedAssemblyTracker` (shared dependency layer in Default ALC)	Companion DLLs from `.tpkg` files, bare-DLL non-module dependencies	One `Assembly` instance per assembly name across the whole process. If two packages each ship `MyCommon.dll` v1.5, the second one reuses the first load (last-loaded-wins on `forceReload=true` for hot updates).
Per-package `ModuleAssemblyLoadContext`	Entry-point DLLs declared in a package's `manifest.json`	Isolated. Each package gets its own ALC. Two packages can ship two different versions of the same entry-point assembly without conflict. Resolving a missing reference falls through: package ALC → tracker → host Default ALC.

Concretely, this means:

Host contracts (IRouteContext, ITsakModule, IRedbService, …) always resolve to the host's Default ALC. A module never accidentally rebinds them to a private copy.
Companion DLLs are de-duplicated. If your package ships Microsoft.Extensions.Logging.Abstractions.dll and the host already has it, the host wins. If the host doesn't have it, the first package to ship it wins for the whole process.
Entry-point code is fully isolated per package. Module A cannot reach into Module B's static fields, even if they both extend the same base class — different ALC, different Type instance.
One bad package does not poison the host. A type-load failure in Orders.tpkg is logged and Orders.tpkg is skipped; every other package keeps running.

How Tsak loads a module

HotReloadService scans every directory under Tsak:Modules:AssemblyPaths for *.dll and *.tpkg.
For each file the loader checks LastWriteTimeUtc against its known snapshot — unchanged files are skipped instantly.
For a .tpkg: the manifest is read, companion DLLs load first (so dependency resolution works), then entry-point DLLs load into a fresh per-package ALC.
For a bare .dll: the file loads into a fresh per-module ALC. If ITsakModule/InitRoute.main is found → it becomes a module; otherwise it is registered as a shared dependency and remembered as "not a module".
ConfigMerger deep-merges all 5 config layers into a single IDictionary<string, object?> and exposes it as the context's property bag.
TsakContextManager creates an IRouteContext, hands it the Quartz scheduler options and the tsak-dlq: endpoint, and awaits the module's InitializeAsync — which runs InitRoute.main(ctx) in either form, or the ITsakModule's own initialization — before the context starts.
The context starts (if AutoStart = true): transports connect, consumers begin reading, routes go live.

Hot-swap

Replace the artifact on disk — that is the whole API.

# Bare directory: atomic rsync over existing files
rsync -a ./publish/ /opt/tsak/modules/

# .tpkg: replace the single file
cp ./output/Orders.tpkg /opt/tsak/modules/

HotReloadService detects the timestamp change, performs a graceful swap (start new ALC → wait for it to settle → drain old context's in-flight exchanges → stop old context → optionally unload old ALC). In a cluster each node performs this swap independently on its own scan schedule — to stagger an update, deploy the package to nodes sequentially yourself. (HotReload:RollingUpdate is reserved and not implemented yet.)

Graceful shutdown when an artifact is removed

Deleting a file from modules/ is a first-class deployment operation, not an error condition. The runtime treats it as an explicit "stop and unload" command:

You do this	Tsak does this
`rm modules/Orders.tpkg`	After `RemovalDebounceScans` confirm the file is gone, all modules from that package are unloaded atomically: each context stops gracefully (drain in-flight, close transports, close DB connections), the package's isolated ALC is disposed. Other packages keep running.
`rm modules/Orders.dll`	Same flow scoped to one bare-DLL module: graceful context stop → unregister module → dispose its ALC.

The debounce (HotReload:RemovalDebounceScans, default 2 scans) protects against false positives during atomic file replacements where a deployment briefly removes the file before writing the new one.

Hot-reload knobs

Key	Default	Behavior
`Tsak:Modules:AssemblyPaths`	`["modules"]`	Directories to scan for `.dll` and `.tpkg` files, at their top level. Never point it at `Libs/`, which holds the shared layer.
`HotReload:ScanIntervalSeconds`	`10`	How often to scan the configured paths.
`HotReload:KeepVersions`	`2`	Old versions kept in memory for one-command rollback (bare-DLL flow).
`HotReload:StartupTimeoutSeconds`	`60`	Wait time for the new version to settle before retiring the old one.
`HotReload:RollingUpdate`	`false`	RESERVED — not implemented; nodes reload independently.
`HotReload:Collectible`	`false`	Enable `AssemblyLoadContext.Unload()` for full GC reclamation: an old module version is collected once nothing of the host refers to it (see "What does an unload release?" below). After each scan that let a context go, Tsak runs a few full collections in the background; a context still in memory after them is logged as a warning (`module context {Name} still alive after 10 collections`). With `false` (default) old contexts stay in memory until process restart. Either way the count is in `GET /api/system/info` (see "Module contexts in memory" below).
`HotReload:MaxOrphanedContexts`	`KeepVersions + 3`	How many module contexts may stay in memory after their modules were replaced or removed before the `hotreload` health check turns Degraded with "restart the node to reclaim memory". The versions rollback keeps on purpose (`KeepVersions`) are not counted. The default leaves room for a few releases on top of the rollback history: a planned release swaps a module once or twice, so more than `KeepVersions + 3` left-over versions on one node means a series of swaps without a restart (a dev loop, or a module whose old version does not unload), and the memory they hold only grows from there.
`HotReload:MaxWorkingSetMb`	none	A working set (MB) above which `GET /api/health/deploy` advises a cold deploy. Unset: the advice looks at the module contexts only.
`HotReload:RemovalDebounceScans`	`2`	Number of consecutive scan cycles a missing file must persist before its module is unloaded — protects against false positives during atomic file replacement.

Module contexts in memory

Each module version runs in its own load context. A hot swap, a package reload or a removal lets the old context go; whether it then leaves memory depends on HotReload:Collectible. Two numbers say where a node stands:

Number	Meaning
`ModuleContextsAlive`	Module contexts in memory: the live modules, the rollback history, the orphaned ones and the ones still unloading.
`OrphanedModuleContexts`	Contexts Tsak let go that are still in memory. With `Collectible: false` that is every context let go; with `true`, the ones still alive after the unload check. A context that leaves memory later stops being counted.

They are in GET /api/system/info (with MaxOrphanedModuleContexts and ModuleContextsNotice), in the system metrics of GET /api/diagnostics/dump and GET /api/dashboard, on the node page of the dashboard, and in tsak system info, tsak diagnostics dump and tsak diagnostics dashboard. HotReloadService.LeakedAlcCount is the same number as OrphanedModuleContexts. Above HotReload:MaxOrphanedContexts the hotreload health check is Degraded and says "restart the node to reclaim memory"; with Tsak:Health:DegradedNotReady that also takes the node out of readiness. What to do then: DEPLOYMENT.md, "After a series of hot swaps".

A pipeline asks before it deploys: GET /api/health/deploy answers 200 with hot-swap (copy the package into modules/ of the running node) or 409 with cold-deploy and the reason codes (cordon, stop, replace the packages, start, wait for ready, uncordon); GET /api/health/deploy/cluster asks every alive node in one call, and tsak system deploy-advice [--cluster] exits 0 or 3. Like the probes it needs no API key: the body holds the node id, the verdict, the reason codes and the counts behind them, nothing about configuration, connections, routes or modules. Details: API_GUIDE.md, "Deploy advice"; the Kubernetes and compose with TeamCity steps: DEPLOYMENT.md, "Deploy advice in a pipeline".

Building a `.tpkg`

Two production-tested approaches ship with the repo. Pick whichever matches your build pipeline.

Option A — MSBuild target inside the `.csproj`

Put packing inside the build itself. After every dotnet build the module is repackaged and dropped into the local Tsak modules/ for instant hot-reload. This is the pattern used by redb.Route.Demo:

<PropertyGroup>
  
  <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>

  <TsakModuleName>Orders</TsakModuleName>
  <TsakModulesDir>$(MSBuildThisFileDirectory)..\redb.Tsak\src\redb.Tsak.Worker\modules</TsakModulesDir>
</PropertyGroup>

<Target Name="PackTpkg" AfterTargets="Build">
  <PropertyGroup>
    <_TpkgStaging>$(IntermediateOutputPath)tpkg</_TpkgStaging>
    <_TpkgFile>$(MSBuildThisFileDirectory)output\$(TsakModuleName).tpkg</_TpkgFile>
  </PropertyGroup>

  <RemoveDir Directories="$(_TpkgStaging)" />
  <MakeDir Directories="$(_TpkgStaging);$(MSBuildThisFileDirectory)output" />

  <Copy SourceFiles="$(MSBuildThisFileDirectory)manifest.json"
        DestinationFolder="$(_TpkgStaging)" />
  <Copy SourceFiles="$(TargetPath)"
        DestinationFolder="$(_TpkgStaging)" />
  <Copy SourceFiles="$(MSBuildThisFileDirectory)$(TsakModuleName).config.json"
        DestinationFolder="$(_TpkgStaging)"
        Condition="Exists('$(MSBuildThisFileDirectory)$(TsakModuleName).config.json')" />

  <ZipDirectory SourceDirectory="$(_TpkgStaging)"
                DestinationFile="$(_TpkgFile)" Overwrite="true" />

  <MakeDir Directories="$(TsakModulesDir)" Condition="!Exists('$(TsakModulesDir)')" />
  <Copy SourceFiles="$(_TpkgFile)" DestinationFolder="$(TsakModulesDir)" SkipUnchangedFiles="false" />
  <Touch Files="$(TsakModulesDir)\$(TsakModuleName).tpkg" />

  <Message Importance="high" Text="Packed $(TsakModuleName) → $(_TpkgFile)" />
</Target>

The <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies> line is critical for class-library SDK projects — without it, NuGet PackageReference assemblies are not copied to bin/ and your .tpkg ships with missing transitive dependencies.

The final <Touch> updates LastWriteTimeUtc on the copy, which guarantees HotReloadService notices the change even when the new file's content hash matches a recently-loaded version.

Option B — Standalone PowerShell script

Best when you want to package multiple projects together as one .tpkg (e.g. a Core + Api pair sharing private dependencies), or when you want fine-grained control over which DLLs to include. Pattern from a real production module that bundles two entry-point projects plus three companions into one tsum.tpkg:

param(
    [ValidateSet("Debug","Release")] [string]$Configuration = "Release"
)
$ErrorActionPreference = "Stop"

$tfm     = "net10.0"
$pkgName = "tsum"
$staging = Join-Path $env:TEMP "${pkgName}_tpkg_staging"
$tpkg    = Join-Path $PSScriptRoot "output\$pkgName.tpkg"
$modules = "..\redb.Tsak\src\redb.Tsak.Worker\modules"

# 1. Build every project that contributes to the package
dotnet build .\tsum.Core\tsum.Core.csproj -c $Configuration --nologo
dotnet build .\tsum.Api\tsum.Api.csproj   -c $Configuration --nologo

# 2. Stage manifest, entry points, companion DLLs, configs
if (Test-Path $staging) { Remove-Item -Recurse -Force $staging }
New-Item -ItemType Directory -Path $staging | Out-Null

Copy-Item .\tsum.Core\manifest.json $staging
Copy-Item ".\tsum.Core\bin\$Configuration\$tfm\tsum.Core.dll" $staging   # entry point
Copy-Item ".\tsum.Api\bin\$Configuration\$tfm\tsum.Api.dll"   $staging   # entry point
Copy-Item ".\tsum.Core\bin\$Configuration\$tfm\tsum.Models.dll" $staging # companion
Copy-Item ".\tsum.Core\bin\$Configuration\$tfm\tsum.Domain.dll" $staging # companion
Copy-Item ".\tsum.Core\tsum.Core.config.json" $staging -ErrorAction SilentlyContinue
Copy-Item ".\tsum.Api\tsum.Api.config.json"   $staging -ErrorAction SilentlyContinue

# 3. Zip and deploy
if (Test-Path $tpkg) { Remove-Item $tpkg -Force }
Add-Type -AssemblyName System.IO.Compression.FileSystem
[IO.Compression.ZipFile]::CreateFromDirectory($staging, $tpkg)

Copy-Item $tpkg $modules -Force
(Get-Item (Join-Path $modules "$pkgName.tpkg")).LastWriteTime = Get-Date  # trigger hot-reload
Remove-Item -Recurse -Force $staging

The corresponding manifest.json declares both entry points so Tsak loads each one as a separately-managed module inside the shared package ALC:

{
  "Name": "tsum",
  "Version": "1.0.0",
  "EntryPoints": [ "tsum.Core.dll", "tsum.Api.dll" ],
  "Dependencies": []
}

Option C — Plain ZIP (any tool, any pipeline)

A .tpkg is just a ZIP. Any tool that can write a ZIP can produce one:

Compress-Archive -Path manifest.json,Orders.dll,Orders.Domain.dll,Orders.config.json `
                 -DestinationPath Orders.zip -Force
Move-Item Orders.zip Orders.tpkg -Force

Or zip -j Orders.tpkg manifest.json *.dll *.config.json on Linux.

Excluding host-provided DLLs (recommended for shipping)

Tsak Worker already loads dozens of assemblies (redb.Core, redb.Route.*, every transport it ships, Quartz, Serilog, ASP.NET Core). Repacking those into every .tpkg wastes disk, bloats the archive, and occasionally causes version skew if your module references a different version than the host.

The recommended pattern is: build an exclude set from redb.Tsak.Worker\bin\<tfm>\*.dll and skip any DLL in that set when staging — except for a small force-include list of packages you know are version-incompatible with the host's copy.

$hostBin   = "..\redb.Tsak\src\redb.Tsak.Worker\bin\$Configuration\$tfm"
$exclude   = New-Object 'System.Collections.Generic.HashSet[string]' (
                 [System.StringComparer]::OrdinalIgnoreCase)
Get-ChildItem "$hostBin\*.dll" | ForEach-Object { [void]$exclude.Add($_.Name) }

# Always include these even if the host ships an older copy (binary-incompatible)
$forceInclude = @( 'Microsoft.IdentityModel.*.dll' )

Get-ChildItem ".\bin\$Configuration\$tfm\*.dll" | ForEach-Object {
    $force = $false
    foreach ($pat in $forceInclude) { if ($_.Name -like $pat) { $force = $true; break } }
    if ($force -or -not $exclude.Contains($_.Name)) {
        Copy-Item $_.FullName $staging
    }
}

The module's per-package ALC then resolves a force-included DLL from the package itself before falling through to the host's older copy in the Default ALC.

5-layer configuration

The most powerful feature in Tsak's configuration model: deep merge across five layers, with predictable precedence.

Layer 1: Tsak:Contexts:default              ← base for all contexts (lowest priority)
Layer 2: Tsak:Contexts:{name}               ← named context overrides
Layer 3: modules/context.json              ← module infrastructure defaults, next to the package or DLL
Layer 4: modules/{Module}.config.json      ← module business settings (or inside the .tpkg)
Layer 5: Tsak:Contexts:{name}:Override      ← DevOps final word (highest priority)

Later layers deep-merge over earlier layers. Nested objects merge recursively — they do not replace each other wholesale.

Worked example

// appsettings.json
{
  "Tsak": {
    "Contexts": {
      "default": {
        "AutoStart": true,
        "RabbitMQ": { "Host": "localhost", "Port": 5672 }
      },
      "orders": {
        "Modules": ["Orders"],
        "RabbitMQ": { "Host": "rabbit-orders.local" },
        "Override": {
          "RabbitMQ": { "Password": "from-secret-manager" }
        }
      }
    }
  }
}

// modules/context.json
{ "RabbitMQ": { "Vhost": "/orders", "Username": "orders-svc" } }

// modules/Orders.config.json
{ "MaxRetries": 10 }

Effective configuration for the orders context:

{
  "AutoStart": true,
  "MaxRetries": 10,
  "RabbitMQ": {
    "Host": "rabbit-orders.local",       // from Layer 2 (overrode Layer 1 "localhost")
    "Port": 5672,                         // from Layer 1 (preserved through merge)
    "Vhost": "/orders",                   // from Layer 3 (module ships default)
    "Username": "orders-svc",             // from Layer 3
    "Password": "from-secret-manager"     // from Layer 5 (DevOps wins)
  }
}

Hot-reload of config

Edit context.json or {Module}.config.json while Tsak is running. The hot-reload scan re-merges all five layers and restarts the affected context with the new properties. No worker restart required.

Named vs anonymous contexts

Named — explicitly defined in Tsak:Contexts:{name} with a Modules array. Multiple modules share one IRouteContext.
Anonymous — for any module not assigned to a named context. One module = one context, named after the module.

{
  "Tsak": { "Contexts": {
    "api": { "Modules": ["Api.Orders", "Api.Catalog"], "AutoStart": true }
  }}
}

Api.Orders and Api.Catalog share the api context (and its property bag). Any other module gets its own anonymous context.

A module's database

A module reaches redb in two ways, and they are two different databases.

The worker's own redb. context.GetServiceProvider().GetRequiredService<IRedbService>() and an unnamed context.GetRedbService() return the database Tsak itself runs on. It is chosen by Tsak:Redb:Provider (sqlite by default) together with that provider's connection string: ConnectionStrings:Postgres, ConnectionStrings:MSSql or ConnectionStrings:Sqlite. Setting ConnectionStrings:MSSql alone does not move it to SQL Server: set Tsak:Redb:Provider to mssql as well, or SQL written for SQL Server runs against SQLite. The database must exist beforehand: redb creates its tables, not the database.

Tsak__Redb__Provider=mssql
ConnectionStrings__MSSql="Server=sql;Database=Orders;User Id=sa;Password=...;TrustServerCertificate=true"

A named instance is a database of the module's own, declared in the context configuration under Redb:{name}, in any of the five layers (usually the module's config file or Tsak:Contexts:{name}):

{
  "RedbInstanceName": "orders",
  "Redb": {
    "orders": { "Provider": "mssql", "ConnectionString": "Server=sql;Database=Orders;...", "UsePro": true, "EnsureCreated": true }
  }
}

Provider and ConnectionString are required. Only the instance named by RedbInstanceName creates its schema on start (EnsureCreated); the others are registered without it. In a route, resolve the instance per exchange, context.GetRedbService("orders", exchange), so that concurrent exchanges never share a connection. In main and other one-shot setup code, context.GetRedbService("orders") returns the shared instance.

Full reference: CONFIG_GUIDE.md.

Plain SQL through redb.Route.Sql. The worker registers the ADO providers it ships under their invariant names, so a module, including one written in XML only, names the provider and does not register it: Npgsql (PostgreSQL), Microsoft.Data.SqlClient (SQL Server), Microsoft.Data.Sqlite (SQLite), as ProviderName of a SqlConnectionFactory bean or provider= of a sql: endpoint.

A module's save interceptors

A module cannot reach the worker's DI, so it subscribes a redb IRedbSaveInterceptor through its context, in InitializeAsync or main:

public override async Task<IRouteContext> InitializeAsync(IRouteContext context)
{
    context.GetService<ITsakSaveInterceptors>()?.Add(new NotifyFrontInterceptor());
    // routes...
    return context;
}

The subscription lives as long as the context. When Tsak removes the context (the module is unloaded, or reloaded and its context built anew) it removes every interceptor the context added, before the module's load context unloads; the module needs no cleanup. Stopping the context keeps them, as it keeps the routes. Add returns a handle that removes one interceptor earlier. Without a worker database (Tsak:Redb:Provider unset) the service is absent and GetService returns null.

What such an interceptor sees and can do:

Every save and delete of the worker, not only the module's own: those of other modules, made through the worker's redb service or through the named instances of any context. Filter by scheme or object type inside the interceptor when only some objects matter.
Veto over all of them. An exception thrown from SavingAsync or DeletingAsync cancels the save or delete it was called for, whichever module made it. Throw only to stop a save that must not happen.
The semantics of IRedbSaveInterceptor: SavedAsync runs after the commit, or inside the transaction within ExecuteAtomicAsync; under PropsSaveStrategy=ChangeTracking RedbSavedContext.Changes lists the changed values.
A subscription reaches the next save or delete, scopes already open included; a save already running finishes with the interceptors it started with.

REST API

Every endpoint speaks JSON. Auth is opt-in (Tsak:Auth:Enabled) — when enabled, all endpoints require an API key except the auth-exempt health probes under /api/health/* (configurable via Tsak:Api:AuthExempt) and the echo probe.

The API is itself a route context (_system): one catch-all HTTP listener whose pipeline is header bridge → auth → controller dispatch, with controllers discovered by assembly scan. API_GUIDE.md documents the full endpoint map, the request pipeline, the auth model, and the extension points.

Controllers

Group	Endpoints	Purpose
`/api/health`	3	Kubernetes probes: startup / live / ready (auth-exempt)
`/api/system`	5	Health / metrics / metrics-history / info / config
`/api/contexts`	8	List / get / start / stop / restart / reset-route-states / endpoints / remove
`/api/contexts/{ctx}/routes`	8	List / get / start / stop / force-stop / inflight / context-wide inflight / metrics
`/api/modules`	6	List / get / remove / upload / validate / rollback (signed; upload off by default)
`/api/watchdog`	6	Status / alerts / enable / disable / alert-status / test-alert
`/api/diagnostics`	2	Dump (cluster-wide) / route-level diagnostics
`/api/lifecycle`	1	Recent lifecycle events (filtered)
`/api/audit`	1	Persisted admin-action trail (filtered, paged; `admin` role)
`/api/exchanges`	3	Dead-letter queue: list / replay / discard failed exchanges
`/api/logs`	3	Tail (incremental) / list-files / download-file
`/api/scheduler`	8	Status / scheduled / running / start / standby / pause-job / resume-job / fire-job
`/api/cluster`	6	Status / nodes / rebalance / remove-node / cordon / uncordon (Pro)
`/api/auth`	3	Create / list / revoke API keys
`/api/users`	5	User CRUD (redb-backed storage required)
`/api/dashboard`	1	Aggregated snapshot for the UI (one round-trip)

Sample calls

# Authenticate
KEY="$(tsak auth create --name ci --roles admin --output json | jq -r .rawKey)"

# Health
curl -s http://localhost:9090/api/system/health | jq
# {
#   "status": "Healthy",
#   "description": "All checks passed",
#   "checks": { "contexts": "Healthy", "metrics": "Healthy", "control-plane": "Healthy", "redb": "Healthy", "cluster": "Healthy" },
#   "details": {},
#   "timestamp": "2026-10-22T09:00:00Z"
# }

# Start a context
curl -s -X POST -H "Authorization: Bearer $KEY" \
  http://localhost:9090/api/contexts/orders/start | jq

# Live in-flight exchanges in a route (debugging stuck routes)
curl -s -H "Authorization: Bearer $KEY" \
  http://localhost:9090/api/contexts/orders/routes/route-1/inflight | jq

# Force-stop a hung route
curl -s -X POST -H "Authorization: Bearer $KEY" \
  http://localhost:9090/api/contexts/orders/routes/route-1/force-stop

# Recent lifecycle events
curl -s -H "Authorization: Bearer $KEY" \
  "http://localhost:9090/api/lifecycle?contextName=orders&limit=20" | jq

# Diagnostic dump (CPU profiles, GC info, thread states)
curl -s -H "Authorization: Bearer $KEY" \
  http://localhost:9090/api/diagnostics/dump > tsak-dump.json

Echo probe

/api/echo (path: Tsak:Api:Echo:Path) reflects the request back as JSON: a liveness probe that also shows what the host received. It needs no API key, but it ships stopped. Start the route system-echo of the _system context from the dashboard (Routes) or with the API:

curl -s -X POST -H "Authorization: Bearer $KEY" \
  http://localhost:9090/api/contexts/_system/routes/system-echo/start

While the route is stopped, /api/echo answers 503 with the name of the route to start.

Typed C# client

services.AddTsakClient(opts =>
{
    opts.BaseUrl = "http://tsak-prod:9090";
    opts.ApiKey = builder.Configuration["TsakKey"];
});

public class MyService(ITsakApiClient tsak)
{
    public async Task RestartAllAsync(CancellationToken ct)
    {
        var contexts = await tsak.ListContextsAsync(ct);
        foreach (var c in contexts.Where(c => c.Status == "Failed"))
            await tsak.RestartContextAsync(c.Name, ct);
    }
}

CLI

tsak is a single binary that talks to any Tsak worker. It supports multiple connection profiles, JSON output for CI, and rich tabular output for humans.

# Profile management
tsak profile add --name prod    --url http://prod-1:9090 --key $PROD_KEY
tsak profile add --name staging --url http://stg-1:9090  --key $STG_KEY
tsak profile use prod
tsak profile list

# Per-call override
tsak --url http://other:9090 --key $OTHER_KEY context list

# Output format
tsak context list                  # table (default)
tsak context list --output json    # JSON, for scripts and CI
tsak context list --output quiet   # nothing but the exit code

Command groups

Group	Commands
system	`system health`, `system metrics`, `system info`, `system config`
profile	`profile list`, `profile add`, `profile remove`, `profile use`
auth	`auth list`, `auth create`, `auth revoke`
context	`context list`, `context get`, `context start`, `context stop`, `context restart`, `context remove`
route	`route list`, `route get`, `route start`, `route stop`, `route inflight`, `route force-stop`
module	`module list`, `module get`, `module remove`, `module keygen`, `module sign`, `module deploy`, `module validate`, `module rollback`
scheduler	`scheduler status`, `scheduler jobs`, `scheduler running`, `scheduler start`, `scheduler standby`, `scheduler pause`, `scheduler resume`, `scheduler fire`
cluster	`cluster status`, `cluster nodes`, `cluster rebalance`, `cluster cordon`, `cluster uncordon`
watchdog	`watchdog status`, `watchdog alerts`, `watchdog enable`, `watchdog disable`
diagnostics	`diagnostics lifecycle`, `diagnostics dump`, `diagnostics route`, `diagnostics dashboard`
dlq	`dlq list`, `dlq replay`, `dlq discard`
log, audit	`log get`, `audit`

tsak <command> --help shows the arguments and options of a command. A required argument left out is named in the error (Required argument '<routeId>' is missing for command 'route get'.), followed by the command's help. module keygen writes a signing key pair: --out is required, an existing pair is never replaced without --force, and the full paths of both files are printed (MODULE_DEPLOYMENT.md).

Web dashboard

A separate Blazor Server process (redb.Tsak.Web) — works in two modes:

Mode	Storage	Node discovery
Standalone	None	Static node list in `appsettings.json`
Cluster	Required (Postgres / MSSQL)	Discovered dynamically from redb cluster topology

Pages

Page	Highlights
Dashboard	Cluster overview, status donut chart, sparkline metrics, node list grid (sortable, filterable, paginated).
Cluster	3-level topology tree (cluster → group → node), module assignment, per-node health, click-through to NodeDetail.
NodeDetail	5 tabs: Contexts · Scheduler · Modules · Monitoring · Logs. Live Chart.js graphs (CPU, GC, memory, threads), 10s auto-refresh.
Routes	All routes across all contexts. Status, message count, error rate, click-through to route detail.
RouteView	Per-route deep-dive: definition, current state, in-flight exchanges, recent diagnostics.
Endpoints	Consumer / producer endpoints per route.
Watchdog	Suspected and hung route alerts with manual stop/restart actions.
Logs	Searchable ring-buffer log viewer with level filter and tail mode.
Auth	API key management UI — create, revoke (with confirmation).
Login	Credential-based dashboard access (cluster mode = redb users; standalone = config).

Custom design system

The dashboard uses a custom CSS design system — no Bootstrap, no MUI, no Tailwind. Built on CSS variables, supports dark/light theme, system fonts, inline SVG icons.

Reusable components: TsakGrid<T>, TsakChart, TsakCard, TsakBadge, TsakIcon, TsakConfirmDialog, TsakToast, TsakPageHeader, TsakErrorBoundary.

Cluster mode

Enable with Tsak:Cluster:Enabled = true and a Postgres/MSSQL connection string. Tsak handles the rest.

What you get

Leader election — distributed lock in redb with TTL and epoch fencing. A new leader is elected automatically when the current one dies or loses its lock.
Node registry — each worker registers itself with periodic heartbeats. Dead nodes are evicted after DeadNodeTimeoutSeconds.
Automatic context assignment — the leader distributes contexts across live nodes (currently round-robin; weighted strategies on the roadmap). Contexts are reassigned automatically when nodes join or leave.
Rolling hot-reload — module updates roll across nodes in sequence, never updating multiple nodes concurrently.
Cluster-wide diagnostics — tsak cluster overview aggregates state from every node.

Topology in redb

Stored as a polymorphic 3-level tree using redb.Tree (so it shows up nicely in any redb-aware tool):

cluster:default                             ← scheme: _tsak_clusters
 └── group:default:default                  ← scheme: _tsak_groups
      ├── node:default:default:worker-1     ← scheme: _tsak_nodes
      ├── node:default:default:worker-2     ← scheme: _tsak_nodes
      └── node:default:default:worker-3     ← scheme: _tsak_nodes

Cluster operations (assignment, leader change, rebalance) mutate this tree atomically. Every operation is fenced by the leader's epoch token — a stale leader cannot corrupt state after losing election.

Coordination records (nodes, locks, module assignments) are keyed and looked up by cluster and group, so several clusters on one database elect their own leaders and keep their own nodes and assignments, even with equal group names or node ids.

Cluster configuration

{
  "Tsak": {
    "Cluster": {
      "Enabled": true,
      "ClusterName": "default",
      "GroupName": "default",
      "NodeId": "",
      "ApiEndpoint": "http://node-1.local:9090",
      "HeartbeatIntervalSeconds": 15,
      "DeadNodeTimeoutSeconds": 60,
      "LeaderLockTtlSeconds": 30,
      "Strategy": "round-robin"
    }
  }
}

Node identity and Quartz instance

NodeId names the node: its record, its locks, and, with Quartz:quartz.scheduler.instanceId = NODE (the shipped value), the Quartz instance of every context scheduler of the worker. Left empty it is the host name.

NODE: the schedulers take the node id. It stays across restarts, so a node that crashed takes its own fired triggers back at its first Quartz check-in instead of waiting until the cluster counts its old record as failed (about 15 s by default), and a preferred-node pin names a node that comes back. Every context's scheduler of the node carries the same id; each context is a Quartz cluster of its own (one scheduler name), so that is correct.
AUTO: Quartz draws a new id at every start. Nothing to configure; after a crash the old record waits out the check-in timeout.
An explicit id is used as written, for every context scheduler; give each worker its own.

Every worker needs a node id of its own. A worker that finds its node id held by a record that is still alive watches it until its heartbeat would expire (DeadNodeTimeoutSeconds): a previous process of the node, killed and restarted at once, goes quiet and the start goes on; a second process with the same node id keeps beating and the start is refused with the host and the last heartbeat of the other. A record left by a clean stop is no conflict.

Replicas:

Kubernetes StatefulSet: leave NodeId empty; the pod name (tsak-0, tsak-1) is the host name and stays across restarts.
Docker Compose deploy.replicas: leave NodeId empty; each container has a host name of its own, unique, but a re-created container gets a new one, which then behaves as AUTO. For a stable id run the workers as separate services with hostname: set, or set Tsak__Cluster__NodeId per service.
Never set one NodeId (or one explicit Quartz instanceId) in a configuration all replicas share.

Pluggable cluster backends

All cluster coordination is hidden behind interfaces in redb.Tsak.Core.Pro:

Interface	Default implementation (redb)	Drop-in alternative
`ILeaderElection`	`RedbLeaderElection` (epoch-fenced lock in redb)	`KubernetesLeaderElection` (Lease API), `EtcdLeaderElection`, ZK
`IDistributedLock`	`RedbDistributedLock` (TTL row in redb)	`RedisDistributedLock`, `KubernetesLeaseLock`
`INodeRegistry`	`RedbNodeRegistry` (heartbeat rows)	`KubernetesPodRegistry` (label selector), Consul
`IClusterCoordinator`	`ClusterCoordinator` (background loop)	implementation owns the loop
`IClusterBootstrap`	`RedbClusterBootstrap`	bootstrap from K8s ConfigMap
`IAssignmentManager`	`RoundRobinAssignmentManager`	`WeightedAssignmentManager`, custom

Replace any of them with a single DI registration before AddTsakCluster(). Nothing else in the codebase changes — the ClusterCoordinator only talks to these interfaces.

builder.Services
    .AddSingleton<ILeaderElection, KubernetesLeaderElection>()  // override default
    .AddTsakCluster(builder.Configuration);                     // everything else stays

This is the design path for native Kubernetes integration without ever touching redb for coordination — handy when the cluster runs against an external operational database that you do not want to use as a synchronization primitive.

Watchdog

WatchdogService continuously inspects every route and classifies it into one of three states:

                                 (no progress for SuspectedThresholdMinutes)
       ┌────────────┐                            │
       │  Healthy   │ ─────────────────────────────────────► ┌─────────────┐
       └────────────┘                                         │  Suspected  │
              ▲                                               └─────────────┘
              │                                                       │
              │ (progress resumes)            (no progress for HungThresholdMinutes)
              │                                                       │
              │                                                       ▼
              │                                                ┌─────────────┐
              └─────────────────────────────────────────────── │    Hung     │
                                                                └─────────────┘
                                                                       │
                                            (AutoRestartHungRoutes = true)
                                                                       │
                                                                       ▼
                                                       Force-stop + restart route

Configurable:

{
  "Tsak": {
    "Watchdog": {
      "Enabled": true,
      "CheckIntervalSeconds": 10,
      "SuspectedThresholdMinutes": 0.5,
      "HungThresholdMinutes": 1.5,
      "AutoRestartHungRoutes": false
    }
  }
}

Alerts surface in /api/watchdog/alerts, in the CLI (tsak watchdog alerts), and in the Web dashboard's Watchdog page.

Lifecycle & graceful shutdown

Tsak's startup and shutdown sequences are deterministic — LifecycleHookOrdering enforces the order so operators can rely on what is up before what depends on it, and what drains before what stops.

Startup order (Worker DI)

QuartzSchemaInitializer — applies Quartz DDL (Postgres / MSSQL) idempotently via raw ADO.NET. No redb dependency at this stage to avoid bootstrap deadlocks.
MetricsCollectionService — periodic process / GC sampling into the circular buffer.
TsakHostedService — main coordinator: shared assembly loader → module discovery → context start → cluster register.
RouteWatchdogService — hung-exchange detector (Pro).

Shutdown sequence (`TsakHostedService.StopAsync`)

Logs "Tsak graceful shutdown, stopping all contexts...".
Runs every ITsakLifecycleHook.OnStoppingAsync() in reverse registration order — last hook to start is first to stop. Cluster deregistration is a hook, so leadership is released before contexts begin draining.
For each context, calls ITsakContextManager.StopContextAsync(ctx, CancellationToken.None) — deliberately not the host shutdown token. One slow context cannot cancel the drain of the others. Each context owns its own context:graceful-stop-timeout for in-flight exchanges.
base.StopAsync() releases hosted-service resources.
Quartz scheduler stops, Serilog sinks flush.

Stop / restart REST responses include DrainTimeMs and InflightAfter — operators see exactly how long the drain took and whether anything was force-killed. SIGTERM handling is delegated to the standard .NET IHostApplicationLifetime.

Lifecycle hooks

public class MyHook : ITsakLifecycleHook
{
    public Task OnStartingAsync(CancellationToken ct)  => /* before module discovery */;
    public Task OnStartedAsync (CancellationToken ct)  => /* after all contexts running */;
    public Task OnStoppingAsync(CancellationToken ct)  => /* before context drain (reverse) */;
    public Task OnStoppedAsync (CancellationToken ct)  => /* after all contexts stopped */;
}

builder.Services.AddSingleton<ITsakLifecycleHook, MyHook>();

Every start / stop / error event is also persisted by LifecycleAuditService and queryable through /api/lifecycle.

Storage modes

Mode	API keys	Modules	Cluster	State	Use case
InMemory	`ConfigApiKeyStore` (read-only, from `appsettings`)	In-process registry	Not supported	Lost on restart	Dev, CI, embedded scenarios
Redb (Postgres)	`RedbApiKeyStore` (redb, runtime CRUD)	Persistent	Supported	Survives restart	Single-node production, lightweight clusters
Redb (MSSQL)	`RedbApiKeyStore` (redb, runtime CRUD)	Persistent	Supported	Survives restart	Single-node production, MSSQL shops

Switch modes with one config setting:

{ "Tsak": { "Storage": { "Type": "Redb" }, "Redb": { "Provider": "postgres" } } }

When UsePro = true, redb.Core.Pro is enabled — gives you redb change tracking (faster writes), distributed locking primitives, and the cluster topology features.

Security

Layer	Mechanism
Wire	API Key in `Authorization: Bearer` or `X-Api-Key` header
Storage	SHA-256 hash, raw key never persisted
Comparison	`CryptographicOperations.FixedTimeEquals` (timing-attack safe)
Authorization	Per-endpoint role checks (`admin`, `reader`, custom)
Lifecycle	Optional expiry per key, runtime revocation, immediate cache invalidation
User binding	Optional `UserId` link — when the user is disabled or deleted, the key dies (via `IUserProvider`)
Cache	5-minute TTL key lookup cache, invalidated on revoke
Cluster trust	Inter-node calls use the same API key auth — no implicit trust between nodes
Protected resources	The `_system` context cannot be stopped or removed by any caller, including admins

Full policy: SECURITY.md. Production secrets — which keys exist, what ships empty in the image, and how to inject connection strings, the dashboard login and the management-API secret through env vars — are covered in DEPLOYMENT.md.

Observability

Metrics

MetricsService — circular buffer with CPU, working set, managed memory, threads, GC pressure. Default window: 12 hours × 10s sample = 4320 points.
ContextMetricsCollector — per-context aggregation: messages/sec, error rate, in-flight count.
Per-route metrics from redb.Route's .Metered() step.
Cluster-wide periodic state report: ClusterReportIntervalSeconds.

OpenTelemetry & Prometheus

First-class OTel integration via the standard NuGet packages:

OpenTelemetry.Extensions.Hosting
OpenTelemetry.Exporter.Prometheus.HttpListener
OpenTelemetry.Instrumentation.Process
OpenTelemetry.Instrumentation.Runtime

When Tsak:Metrics:Prometheus:Enabled = true, Tsak exposes a Prometheus-format endpoint at http://<host>:9090/metrics — served straight from the facade's Kestrel server (the API port), auth-exempt. The OTel scrape listener itself binds loopback (localhost:<Port>, default 9464) and the /metrics route proxies it out. That means no separate exposed port and no Windows URL ACL — Kestrel binds sockets directly, so a wildcard HttpListener is never needed. Tsak:Metrics:Prometheus:Port only changes the internal loopback port.

The OTel pipeline registers the redb.Route ActivitySource (RouteActivitySource.SourceName) so any tracing emitted by route processors is collected automatically — point an OTLP collector / Jaeger / Tempo at the activity source (see Tracing), or scrape :9090/metrics from Prometheus.

"Tsak": {
  "Metrics": {
    "Prometheus": { "Enabled": true, "Port": 9464 }   // Port = internal loopback only
  },
  "Tracing": {
    "ServiceName": "redb-tsak-worker",   // OTel resource service.name (Jaeger/collector)
    "Otlp": { "Enabled": true, "Endpoint": "http://localhost:4317", "Protocol": "grpc" }
  }
}

The exporter is pre-flighted: if the loopback bind ever fails, Tsak logs a warning and runs without metrics rather than crashing the host — an optional exporter must never take the Worker down.

Ready-to-use Kubernetes manifests, a Grafana dashboard and a local Prometheus + Grafana + Jaeger docker compose stack live under deploy/.

Health probes — Kubernetes-ready

Distinct probes for the three K8s lifecycle phases. All three are auth-exempt by default (Tsak:Api:AuthExempt):

Endpoint	Probe type	Returns 200 when
`GET /api/health/startup`	startup	process is up
`GET /api/health/live`	liveness	process is not deadlocked
`GET /api/health/ready`	readiness	contexts are running and healthy (stricter than liveness)

The probe endpoints live under /api/health/* (the auth-exempt prefix). The rich aggregate at GET /api/system/health is a separate, auth-gated endpoint for CLI/Web.

HealthCheckService aggregates probes (worst status wins: Unhealthy > Degraded > Healthy) and never throws — exceptions inside a probe become Unhealthy, never a 500. Modules can contribute custom probes by implementing IModuleHealthContributor. Pro ships ClusterHealthContributor reporting leader / member health.

The redb check is Degraded while a redb database of the worker (the host's own or a named instance of a context) has schema problems that need a person, such as a root field of a scheme written twice by nodes of an older redb. The worker keeps serving and redb logs the problem as an error at every start; the check puts it where monitoring looks, with the problem and a link to redb's UPGRADING.md in details, the dashboard's node page and tsak system health. It reads what the last start found and queries no database on a poll. Readiness stays 200 on Degraded unless Tsak:Health:DegradedNotReady=true. Full breakdown of both health endpoints and the contributor SPI: API_GUIDE.md.

Logs

LogRingBuffer — Serilog in-memory sink, default 2000 entries, queryable via REST and Web UI.
File logs via Serilog (rolling files, configurable path).
MemoryUsageEnricher — every log event carries {MemoryUsage} for cheap memory pressure correlation.

Tracing

OpenTelemetry traces via redb.Route's .Traced() step — per-route, per-step spans (route/transport spans like route:identity-discovery are captured automatically).
OTLP exporter — opt-in via Tsak:Tracing:Otlp:Enabled (+ Endpoint, Protocol). Ships traces to any OTLP collector or directly to Jaeger (native OTLP ingest on :4317 gRPC / :4318 http). Off by default.
Service identity via Tsak:Tracing:ServiceName (default redb-tsak-worker) — the service.name shown in Jaeger / the collector.

Diagnostics

Per-route diagnostic dumps via /api/contexts/{ctx}/routes/{id}/diagnostics.
Cluster-wide dumps via /api/diagnostics/dump.
Lifecycle events feed (route start/stop/restart, hot-swap, cluster reassignment, watchdog alerts).
In-flight exchange tracking — see exactly which messages are sitting where right now.

Quartz scheduler

Every Tsak context runs its own Quartz scheduler — {instanceName}-{context}, as Apache Camel runs one per CamelContext — built with the context's first cron: / qtimer: route from the host's Quartz section. A job key is unique within a context, so two modules that both schedule cron://nightly never collide. On a clustered AdoJobStore a job fires on one of the nodes that run its context; don't add .Cluster(true) to such a route. Modules use it via redb.Route.Quartz:

From("cron:0 */5 * * * ?")           // every 5 minutes
    .Setbody(() => DateTime.UtcNow)
    .To("rabbitmq://heartbeats");

Mode	Storage	Cluster-safe	Use
`RAMJobStore`	In-memory	No	Standalone, dev
`AdoJobStore`	Postgres / MSSQL `QRTZ_*` tables	Yes	Production

Schema is auto-created on first start by QuartzSchemaInitializer — embedded SQL scripts for both Postgres and MSSQL, idempotent, runs before Quartz initializes its own connection pool. No DBA action required.

REST endpoints under /api/scheduler cover: status (per context and for the node), listing scheduled jobs, listing currently-running jobs, start/standby, pause/resume/fire-now per job key — each with an optional context.

Three deployment recipes

Recipe A — Standalone, single binary

For development, demos, embedded use.

{
  "Tsak": {
    "Storage": { "Type": "InMemory" },
    "Cluster": { "Enabled": false },
    "Auth":    { "Enabled": false }
  }
}

Just dotnet run. No DB, no auth, no cluster. Fastest path to running a Tsak module locally.

Recipe B — Single node + redb (Postgres)

For production single-node deployments where state must survive restarts.

{
  "ConnectionStrings": { "Postgres": "Host=db;Database=redb;Username=tsak;Password=$$" },
  "Tsak": {
    "Storage": { "Type": "Redb" },
    "Redb":    { "Provider": "postgres", "UsePro": true, "License": "$$" },
    "Cluster": { "Enabled": false },
    "Auth":    { "Enabled": true, "Secret": "$$" }
  }
}

API keys persist. Quartz jobs persist. Lifecycle events persist. Module assignments persist. One node, durable state.

Recipe C — Cluster

For HA and horizontal scaling. Same appsettings on every node, only NodeId and ApiEndpoint differ:

{
  "ConnectionStrings": { "Postgres": "Host=db.cluster;Database=redb;Username=tsak;Password=$$" },
  "Tsak": {
    "Storage": { "Type": "Redb" },
    "Redb":    { "Provider": "postgres", "UsePro": true },
    "Cluster": {
      "Enabled": true,
      "ClusterName": "production",
      "GroupName": "default",
      "ApiEndpoint": "http://node-1.local:9090",
      "Strategy": "round-robin"
    },
    "HotReload": { "RollingUpdate": false },
    "Auth":     { "Enabled": true, "Secret": "$$" }
  }
}

Start three workers. They'll discover each other through the shared redb store, elect a leader, distribute your modules, and roll updates without downtime.

Docker

Pre-built images are published to GitHub Container Registry for every release — no build step required. See Quick start → Option A for the basic flow. This section covers production deployment.

Image variants

Image	Best for	Default ports
`ghcr.io/redbase-app/redb-tsak-worker`	Headless workers (k8s `Deployment`/`StatefulSet`), one process per pod	`9090` (REST)
`ghcr.io/redbase-app/redb-tsak-web`	Separate management UI pod talking to a worker cluster	`8085`
`ghcr.io/redbase-app/redb-tsak-stack`	Single-host install (Worker + Web in one container, like `rabbitmq:management`)	`9090`, `8085`

Available tags:

latest, <version> — net10 build (default).
<version>-net10 — Worker tag with an explicit TFM; pick one that matches your shared connector ABI.
Web and Stack ship on net10 only.

Production run (Worker + external Postgres)

docker run -d --name tsak \
  -p 9090:9090 \
  -v /opt/tsak/modules:/app/modules \
  -e ConnectionStrings__Postgres="$PG_CONN" \
  -e Tsak__Auth__Secret="$TSAK_SECRET" \
  -e Tsak__Metrics__Prometheus__Enabled=true \
  ghcr.io/redbase-app/redb-tsak-worker:<version>

Mount modules/ from the host (or a shared volume) so module updates can be deployed without rebuilding the image; never mount over Libs/, which holds the image's Libs/shared layer. appsettings.json overrides flow through environment variables in the standard ASP.NET pattern (__ for nesting).

docker compose

services:
  tsak:
    image: ghcr.io/redbase-app/redb-tsak-stack:latest
    ports: ["9090:9090", "8085:8085"]
    environment:
      - Tsak__Modules__AssemblyPaths__0=/app/worker/modules
      - ConnectionStrings__Postgres=Host=pg;Database=redb;Username=redb;Password=${PG_PASSWORD}
    volumes:
      - ./modules:/app/worker/modules:ro   # your .tpkg packages (+ context.json)

Settings, secrets, a PostgreSQL service next to it and the cluster setup: DEPLOYMENT.md. The module templates (dotnet new redb-app, redb-worker, ...) ship a deploy/docker-compose.tsak.yml of their own.

Verifying images

cosign verify --key cosign.pub ghcr.io/redbase-app/redb-tsak-worker:<version>

The public key cosign.pub is attached to every release.

Building images yourself

If you need a custom build (e.g. proprietary connectors baked in), build from src/redb.Tsak.Worker/Dockerfile — a plain multi-stage dotnet publish, no private tooling involved. The template host ships its own Dockerfile for the same purpose.

Kubernetes deployment

Tsak is built from the ground up for container deployment — distinct K8s probes, OTel/Prometheus exporter, env-var configuration, and pluggable cluster backends are all in the box.

Probes

Map the three health endpoints to the matching K8s probe types:

startupProbe:
  httpGet: { path: /api/health/startup, port: 9090 }
  failureThreshold: 30
  periodSeconds: 4           # up to 120s to boot

livenessProbe:
  httpGet: { path: /api/health/live, port: 9090 }
  periodSeconds: 10
  failureThreshold: 3        # 30s before pod restart

readinessProbe:
  httpGet: { path: /api/health/ready, port: 9090 }
  periodSeconds: 5
  failureThreshold: 2        # 10s before removal from load balancer

Liveness intentionally does not check module health — that avoids restart loops during rolling updates. Readiness is stricter: any context in a non-running state → pod is removed from service endpoints (no restart), allowing the cluster to rebalance via IAssignmentManager.

Graceful termination

Kubernetes sends SIGTERM, then SIGKILL after terminationGracePeriodSeconds. Set Tsak:Shutdown:TimeoutSeconds to terminationGracePeriodSeconds − 5 so cluster deregistration has its buffer. The shutdown sequence then becomes:

SIGTERM → cluster deregister (lifecycle hook, reverse-order) →
context drain (isolated CT, up to TimeoutSeconds each) →
Quartz shutdown → log flush → SIGKILL never needed

spec:
  terminationGracePeriodSeconds: 60
  containers:
    - name: tsak
      env:
        - name: Tsak__Shutdown__TimeoutSeconds
          value: "55"

Pod identity in cluster mode

StableNode IDs across pod restarts matter for assignment continuity. Inject pod metadata via the downward API and bind NodeId to metadata.name:

env:
  - name: POD_NAME
    valueFrom: { fieldRef: { fieldPath: metadata.name } }
  - name: POD_NAMESPACE
    valueFrom: { fieldRef: { fieldPath: metadata.namespace } }
  - name: POD_IP
    valueFrom: { fieldRef: { fieldPath: status.podIP } }
  - name: Tsak__Cluster__NodeId
    valueFrom: { fieldRef: { fieldPath: metadata.name } }
  - name: Tsak__Cluster__ApiEndpoint
    value: http://$(POD_IP):9090

Use a StatefulSet (or a Deployment + headless service) for predictable pod names. Tsak treats Tsak:Cluster:NodeId as the cluster identity — mapping it to metadata.name keeps assignment stable across restarts.

Prometheus scraping

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port:   "9090"
    prometheus.io/path:   "/metrics"

Set Tsak:Metrics:Prometheus:Enabled = true to expose /metrics on the facade (port 9090). The OTel listener binds loopback only and is proxied out by the facade route — no separate metrics port and no Windows URL ACL. Ready-to-use manifests are under deploy/k8s/.

Native cluster integration

All coordination interfaces (ILeaderElection, IDistributedLock, INodeRegistry, IClusterCoordinator, IClusterBootstrap, IAssignmentManager) are pluggable — see Pluggable cluster backends. A Kubernetes-native Lease implementation (KubernetesLeaderElection) can be dropped in without touching redb for coordination, leaving redb only for module / lifecycle / API key state.

Testing

Every release runs these suites (they live in the development repository, not in this one):

Suite	What's covered
`redb.Tsak.Tests`	Module loading, context lifecycle, config merge, security, controllers, monitoring and health, scheduler, cluster (leader election, heartbeat, rebalance, epoch fencing), hot-reload (ALC isolation, rolling update, rollback, removal debounce), in-memory and redb stores, host startup, full API integration
`redb.Tsak.CLI.Tests`	CLI commands: output format (table/JSON/quiet), parse errors, error handling, auth failure, profile resolution, module signing keys
`redb.Tsak.Web.Tests`	Dashboard sign-in and throttling, page authorization, node views, cluster client selection
`redb.Tsak.Tests.Integration`	The worker against a real redb database (PostgreSQL)

Frequently asked questions

Is Tsak a routing engine? No. Routing is redb.Route. Tsak is the runtime container that hosts route contexts, manages their lifecycle, and exposes a management surface.

Can Tsak load any .NET DLL? Tsak loads class libraries that follow its module convention: a static InitRoute.main(IRouteContext) method, or one or more public RouteBuilder subclasses. Random DLLs are ignored.

Is module isolation a sandbox? No. It is a dependency isolation boundary (separate AssemblyLoadContext per module). A malicious module runs with the same OS privileges as the Tsak process. Restrict write access to modules/ and Libs/ accordingly. See SECURITY.md.

Why is Collectible: false the default for hot-reload? Because a collectible AssemblyLoadContext unloads only when nothing outside it refers to the module any more, and code the host does not control can: a library with a static cache of the types it handled, Emit-generated code kept outside the module's context. Default is the safe choice; with true Tsak checks after each scan that the old versions really go, and logs and counts the ones that stay (OrphanedModuleContexts, see "Module contexts in memory").

What does an unload release? (with Collectible: true) Everything the host and the framework hold of the module:

the context's routes, services, registry, the module's type resolver (bean:, <bean type>, outputClass), its ${prefix.} resolvers (context.RegisterPrefixResolver) and its save interceptors (ITsakSaveInterceptors): they are removed with the context;
the caches of redb.Route keyed by type (XML and SOAP serializers, Protobuf parsers, batch POCO properties, endpoint options): they hold the module's types weakly;
the caches the .NET libraries keep over the types they handled, System.Text.Json above all: redb and redb.Route empty them when a collectible context unloads (System.Text.Json does not let such a context go otherwise, dotnet/runtime#65323), at the cost of cold caches in those libraries once per unload;
XmlSerializer generates its serializer into the module's own context, so it goes with it; a compiled Regex emits a matcher that is collected on its own.
the services a bean or component of the module takes in its constructor: redb.Route builds them without asking the host's DI container about the module's own types (ILogger<TheBean> comes from the host's ILoggerFactory, IOptions<TheSettings> and the other options services hold the default instance, any other service over a module type is absent; redb.Route README, "Services of a collectible module's beans"), since that container keeps every type it was asked for as long as the process lives.

What stays is what a module puts beyond the host's reach: a static of a library it ships, a thread or timer it starts and does not stop, a handler it subscribes to a process-wide event, and a service its own code asks the host's container for when the type is the module's own (context.GetServiceProvider().GetService<ILogger<MyType>>(); take ILoggerFactory and call CreateLogger<MyType>() instead).

What happens to in-flight messages during hot-swap? The old context drains (existing exchanges complete). The new context starts in parallel. Once the new context is healthy and StartupTimeoutSeconds elapses, the old context is retired. Zero message loss for transports that support graceful shutdown.

Can I write my own assignment strategy? Yes — IAssignmentManager is the extensibility point. The round-robin strategy is the only one shipped today; weighted strategies are on the roadmap.

Does Tsak support multi-region clusters? Out of the box, no. The cluster coordination assumes low-latency access to the shared redb database. For multi-region, run one cluster per region and federate above the Tsak layer (e.g. via redb.Route.RabbitMQ shovels).

Why a custom design system instead of Bootstrap or MUI? The dashboard is small and focused. Custom CSS keeps the bundle tiny, eliminates a major source of UI churn (vendor breaking changes), and gives full control over theming. CSS variables enable dark/light themes with no JS.

Is there an OpenAPI spec? Not yet. The API is documented in API_GUIDE.md and exposed via the typed ITsakApiClient. OpenAPI / Swagger generation is on the roadmap.

Roadmap

OpenAPI / Swagger generation for the REST API
Weighted cluster assignment strategies (CPU + memory composite)
Batch operations (POST /api/contexts/batch/start)
Webhook subscriptions for lifecycle events (push instead of poll)
Live config editor in the dashboard
Multi-region federation primitives

Upgrading

What changes from one version to the next, and what to do about it: UPGRADING.md for the steps a breaking change needs, CHANGELOG.md for every release.

Part of

redb.Route — ESB & EIP framework for .NET (the routing engine Tsak hosts)
redb.Core / redb.Core.Pro — RTTI storage backend (the persistence layer Tsak uses)
RedBase — full ecosystem

License

Apache License 2.0 — see LICENSE and NOTICE.

Product	Compatible and additional computed target framework versions.
.NET	net8.0 is compatible. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed. net9.0 is compatible. net9.0-android was computed. net9.0-browser was computed. net9.0-ios was computed. net9.0-maccatalyst was computed. net9.0-macos was computed. net9.0-tvos was computed. net9.0-windows was computed. net10.0 is compatible. net10.0-android was computed. net10.0-browser was computed. net10.0-ios was computed. net10.0-maccatalyst was computed. net10.0-macos was computed. net10.0-tvos was computed. net10.0-windows was computed.

Product

.NET

Compatible target framework(s)

Included target framework(s) (in package)

Learn more about Target Frameworks and .NET Standard.

net10.0
- Microsoft.Extensions.Configuration.Json (>= 10.0.13)
- Microsoft.Extensions.DependencyInjection (>= 10.0.13)
- Microsoft.Extensions.Hosting (>= 10.0.13)
- Microsoft.Extensions.Logging.Console (>= 10.0.13)
- Microsoft.IdentityModel.Abstractions (>= 8.23.0)
- Microsoft.IdentityModel.JsonWebTokens (>= 8.23.0)
- Microsoft.IdentityModel.Logging (>= 8.23.0)
- Microsoft.IdentityModel.Protocols (>= 8.23.0)
- Microsoft.IdentityModel.Protocols.OpenIdConnect (>= 8.23.0)
- Microsoft.IdentityModel.Tokens (>= 8.23.0)
- OpenTelemetry.Exporter.OpenTelemetryProtocol (>= 1.20.0)
- OpenTelemetry.Exporter.Prometheus.HttpListener (>= 1.20.0-beta.1)
- OpenTelemetry.Extensions.Hosting (>= 1.20.0)
- OpenTelemetry.Instrumentation.Process (>= 1.20.0-rc.1)
- OpenTelemetry.Instrumentation.Runtime (>= 1.20.0)
- Quartz.Extensions.DependencyInjection (>= 3.22.6)
- Quartz.Extensions.Hosting (>= 3.22.6)
- Quartz.Serialization.Json (>= 3.22.6)
- redb.Core.Pro (>= 4.7.2)
- redb.Licensing (>= 4.7.2)
- redb.Tsak.Core (>= 4.7.2)
- Serilog (>= 4.4.0)
- System.IdentityModel.Tokens.Jwt (>= 8.23.0)
- System.Security.Cryptography.Pkcs (>= 10.0.13)
net8.0
- Microsoft.Extensions.Configuration.Json (>= 10.0.13)
- Microsoft.Extensions.DependencyInjection (>= 10.0.13)
- Microsoft.Extensions.Hosting (>= 10.0.13)
- Microsoft.Extensions.Logging.Console (>= 10.0.13)
- Microsoft.IdentityModel.Abstractions (>= 8.23.0)
- Microsoft.IdentityModel.JsonWebTokens (>= 8.23.0)
- Microsoft.IdentityModel.Logging (>= 8.23.0)
- Microsoft.IdentityModel.Protocols (>= 8.23.0)
- Microsoft.IdentityModel.Protocols.OpenIdConnect (>= 8.23.0)
- Microsoft.IdentityModel.Tokens (>= 8.23.0)
- OpenTelemetry.Exporter.OpenTelemetryProtocol (>= 1.20.0)
- OpenTelemetry.Exporter.Prometheus.HttpListener (>= 1.20.0-beta.1)
- OpenTelemetry.Extensions.Hosting (>= 1.20.0)
- OpenTelemetry.Instrumentation.Process (>= 1.20.0-rc.1)
- OpenTelemetry.Instrumentation.Runtime (>= 1.20.0)
- Quartz.Extensions.DependencyInjection (>= 3.22.6)
- Quartz.Extensions.Hosting (>= 3.22.6)
- Quartz.Serialization.Json (>= 3.22.6)
- redb.Core.Pro (>= 4.7.2)
- redb.Licensing (>= 4.7.2)
- redb.Tsak.Core (>= 4.7.2)
- Serilog (>= 4.4.0)
- System.IdentityModel.Tokens.Jwt (>= 8.23.0)
- System.Security.Cryptography.Pkcs (>= 10.0.13)
net9.0
- Microsoft.Extensions.Configuration.Json (>= 10.0.13)
- Microsoft.Extensions.DependencyInjection (>= 10.0.13)
- Microsoft.Extensions.Hosting (>= 10.0.13)
- Microsoft.Extensions.Logging.Console (>= 10.0.13)
- Microsoft.IdentityModel.Abstractions (>= 8.23.0)
- Microsoft.IdentityModel.JsonWebTokens (>= 8.23.0)
- Microsoft.IdentityModel.Logging (>= 8.23.0)
- Microsoft.IdentityModel.Protocols (>= 8.23.0)
- Microsoft.IdentityModel.Protocols.OpenIdConnect (>= 8.23.0)
- Microsoft.IdentityModel.Tokens (>= 8.23.0)
- OpenTelemetry.Exporter.OpenTelemetryProtocol (>= 1.20.0)
- OpenTelemetry.Exporter.Prometheus.HttpListener (>= 1.20.0-beta.1)
- OpenTelemetry.Extensions.Hosting (>= 1.20.0)
- OpenTelemetry.Instrumentation.Process (>= 1.20.0-rc.1)
- OpenTelemetry.Instrumentation.Runtime (>= 1.20.0)
- Quartz.Extensions.DependencyInjection (>= 3.22.6)
- Quartz.Extensions.Hosting (>= 3.22.6)
- Quartz.Serialization.Json (>= 3.22.6)
- redb.Core.Pro (>= 4.7.2)
- redb.Licensing (>= 4.7.2)
- redb.Tsak.Core (>= 4.7.2)
- Serilog (>= 4.4.0)
- System.IdentityModel.Tokens.Jwt (>= 8.23.0)
- System.Security.Cryptography.Pkcs (>= 10.0.13)

NuGet packages (1)

Showing the top 1 NuGet packages that depend on redb.Tsak.Core.Pro:

Package	Downloads
redb.Tsak.Web.Pro Runtime container for redb.Route contexts with management API	2.9K

GitHub repositories

This package is not used by any popular GitHub repositories.

Version	Downloads	Last Updated
4.7.2	38	11/13/2026
4.7.1	56	11/11/2026
4.7.0	82	11/7/2026
4.6.2	84	11/5/2026
4.6.1	79	11/4/2026
4.6.0	80	11/3/2026
4.5.1	96	10/30/2026
4.5.0	76	10/29/2026
4.4.2	87	10/23/2026
4.4.1	78	10/21/2026
4.3.0	125	10/14/2026
4.2.2	133	10/5/2026
4.2.0	120	9/30/2026
4.1.1	130	9/25/2026
4.1.0	136	9/21/2026
4.0.1	141	9/18/2026
4.0.0	133	9/11/2026
3.7.2	147	8/26/2026
3.7.1	159	8/26/2026