Meshmakers.Octo.ConstructionKit.Models.System.Bot 3.5.74

Octo Bot Services

The bot services of OctoMesh: an ASP.NET Core background-processing service that runs the platform's long-running and recurring jobs (model import/export, repository backup/restore, fixups and attribute-value aggregation) on top of Hangfire. The repository also publishes two NuGet packages: the System.Bot construction kit model and the reusable Jobs library.

Overview

The service (Meshmakers.Octo.Backend.BotServices) is a Docker-deployed web host that schedules and executes jobs through Hangfire (backed by MongoDB). It consumes commands from the OctoMesh distribution event hub - importing CK and runtime models, exporting runtime data by query or deep graph - and exposes a versioned system API, a versioned tenant API ({tenantId}/v1/jobs/..., AB#5060) plus the Hangfire dashboard under /ui/jobs. Resumable uploads are handled via the tus.io protocol, and authentication is wired through OpenID Connect / JWT bearer against the OctoMesh identity service.

The jobs themselves live in the reusable Meshmakers.Octo.Backend.Jobs library and include:

• Model import/export (ImportModelJob, ExportModelJob)
• Repository dump and restore (DumpRepositoryJob, RestoreRepositoryJob)
• Construction-kit fixups (RunFixupJob)
• Attribute-value aggregation for autocomplete (AttributeValueAggregatorJob)
• Hourly cleanup of stale backup/upload files and of expired artifacts in the artifact store (CleanupStaleFilesJob)
• Secret sweep over the Secret attribute values of every tenant (SecretSweepJob, AB#5539) and the secrets admin API (status, run history, dump deletion; AB#5544; pre-sweep dump restore, AB#5559)

Runtime-model exports automatically embed the CK model dependencies required by the exported entities into the transport container. The deep-graph export resolves the full transitive dependency closure (a model's dependencies, their dependencies, and so on) based on the models installed in the tenant, so the exported file lists every model version range the import target must satisfy. The System model is omitted because it is always available.

The export file is deterministic (AB#7471): the deep-graph export orders entities by CK type id and RtId, associations by role and target, and model dependencies by name; the query export orders by RtId unless the query defines its own sorting. Two exports of the same data are byte-identical and can be diffed.

A failed export (for example a stored query that is refused, or a file that cannot be written) aborts its database transaction at once (AB#8080); it no longer holds the transaction open until MongoDB expires it after about 60 seconds, so a tenant reset or index build right after it does not wait.

Tenant-routed job operations (AB#5060)

The five job operations whose subject is one tenant — repository dump, restore from a tus upload, archive data export, archive data import and the fixup script run — are served on tenant routes:

The System variants were removed in stage 3 of AB#5060. The checkout was searched for callers first and none was left: the SDK stopped addressing them when its five job verbs moved to per-call tenant routes, octo-cli and octo-mcp-service inherit that through the package, and the frontend builds the tenant route itself.

An external caller still on an old path is refused — but with 403, not 404, and the reason is worth knowing. With no System action left to match, system/v1/jobs/dump-repository now matches the tenant route {tenantId:tenantId}/v1/jobs/dump-repository with tenantId = "system", so the request reaches the tenant gate and is refused there because the caller's token names a different tenant. That is the stricter of the two possible outcomes, and it is what the tests pin. The one consequence to keep in mind: a tenant named system would make the old URLs live again as that tenant's routes.

Why the route shape is the whole point. TenantAuthorizationMiddleware reads the tenant from the route value. As long as these operations carried their tenant in ?tenantId=… the gate never saw them, so a token issued for one tenant could dump, restore or export any other tenant's repository. No new tenant-addressed operation may be added to the System controller — it holds job-instance operations only.

Both surfaces share their bodies through Controllers/JobsControllerBase.cs, so the tenant route enqueues the identical Hangfire job with identical arguments; TenantJobRouteAuthorizationTests checks that rather than asserting it.

The five tenant routes carry [AllowParentTenantAdministration] (octo-common-services, AB#5068). An administrator of the parent tenant may back up, restore, export and fix up a child tenant, so a user token of the parent passes the gate on the child's route. Service tokens are never widened by that rule — a client-credentials tenant_id proves nothing, because mirrored clients share the parent's secret. The marker means administration, not access: none of these endpoints returns tenant content, and system/v1/jobs/download (which does hand out a job artifact) is deliberately not marked. Do not put the marker on anything that reads or writes tenant data.

The tus upload sink is tenant-routed too. POST {tenantId}/v1/tus-upload — moved off /system/v1/tus-upload in stage 3. The old shape required a tenantId upload-metadata field that nothing read: the file was stored flat under its tus file id, both consuming jobs took the tenant from the request that started them, and the gate never saw the upload at all because it reads the route value. So any caller holding octo_api could stage bytes for any tenant, and the metadata field looked like an ownership binding while being decoration.

Three parts, because a route rename alone would have promised an ownership the storage did not have:

• Route — {tenantId}/v1/tus-upload, registered through endpoint routing (only that overload produces route values) and carrying [AllowParentTenantAdministration], the same marker the consuming restore / import routes carry. Without it the upload would refuse the exact parent administrator the restore then accepts.
• Storage — uploads live under a per-tenant key prefix of the artifact store (restore-staging/<tenant>/…, see "Shared tus uploads" below; before AB#8736 a per-tenant directory, BackupFileStorageService.GetTusUploadDirectory). This is the actual binding: a restore for one tenant cannot resolve a file staged by another even when handed its id, and there is no separate ownership check to remember or forget. "No such upload" and "belongs to another tenant" both answer 404, so the endpoint is not an oracle for which tus ids exist elsewhere.
• Metadata — tenantId is still accepted for older clients, but the route decides, and a disagreement is refused rather than silently resolved.

🔴 The tenant id becomes a directory name, so ResolveTenantDirectory validates it as a path segment and then verifies the resolved path still sits under the storage root — both checks, because either alone is a single point of failure. The shared TenantIdRouteConstraint already rejects such a value on the route, but the same method serves dumps, which arrive as a Hangfire job argument on no route at all.

The tenantId route constraint arrived with these routes; without it the {tenantId:tenantId} templates never match. It now lives once in octo-common-services (Infrastructure/Routing/TenantIdRouteConstraint.cs) and is registered with builder.Services.AddOctoTenantIdRouteConstraint() — until AB#5060 each host carried its own internal copy, and the seven copies had drifted. While both surfaces existed, /system/... won over /{tenantId}/... because literal route segments outrank parameter segments. That precedence is also why removing the five System actions turns their old URLs into tenant routes for a tenant named system rather than into 404s — see the section above.

Coverage: tests/Jobs.Tests/Api/TenantJobRouteAuthorizationTests.cs — an in-process TestHost running the two real controllers behind the real gate: own tenant allowed, parent user token allowed on the child route, unrelated tenant 403, parent service token 403 under Enforce while its own tenant passes, identical enqueued job on both surfaces, and the marker present on the tenant controller and nowhere else in the assembly.

Required roles of the job operations and the Hangfire dashboard (AB#4497)

Scope alone (octo_api) is not enough for the tenant-wide data jobs. They need the tenant role TenantManagement in the caller's token (policy TenantJobManagementPolicy = full-access scope + role; 403 without it, no job is enqueued):

• The role is read from the caller's own token, i.e. the caller's own tenant. A user of a parent tenant keeps [AllowParentTenantAdministration] on these routes, but needs TenantManagement in the parent tenant; the role is not looked up in the target tenant.
• A dump is a whole-tenant backup and bypasses row-level data permissions by design; the role gate is what protects it. The runtime-data exports (ExportRtByQuery, ExportRtByDeepGraph) are the opposite: they run with the identity and roles of the user who requested them, so the result contains only what that user may read, and only the initiating subject can download it (see "Export jobs run as the caller" in CLAUDE.md).
• The Hangfire dashboard /ui/jobs lists the jobs of all tenants, so on top of the scope gate it is reachable only for callers of the system tenant (tenant_id claim = OctoSystemConfiguration.SystemTenantId) that also hold the tenant role TenantManagement (HangfireDashboardAccessMiddleware + HangfireDashboardAuthorizationFilter); every other tenant, and a system-tenant caller without the role, gets 403.
• Tests: TenantJobRoleAuthorizationTests (per entry point: no role = 403 and nothing enqueued, with role = allowed, parent user with / without role), TenantJobRoleWiringTests (policy registration as Program.cs calls it, production gate defaults, policy on every entry point, tus rule, dashboard rule, Program.cs call sites).

Import jobs: cache lookup and retry policy (AB#7256)

CK and RT imports read the uploaded model from the distributed cache (a temporary GridFS file of the tenant database that the asset repository wrote moments before the job was enqueued).

• Cache lookup: ImportModelJob looks the file up up to three times (waits of 0.5 s and 2 s). A file that is really missing fails the job with JobFailedException.CacheStreamNotFound, whose message names tenant and cache key. The read itself asks the primary when a secondary has not received the file yet (engine-mongodb, "GridFS read preference and read-your-writes").
• Hangfire retry (IImportModelJob): a failed import job is never deleted (OnAttemptsExceeded = Fail), it stays Failed in the dashboard for diagnosis. ImportCkAsync and ImportCkBatchAsync are retried three times (after 10, 30 and 60 s): importing a CK model version that is already installed is a no-op, and the cache streams are cleared only after a successful import (the batch clears all of its keys at the end), so a retry finds the files again. ImportRtAsync is not retried by Hangfire because the RT import strategies are not idempotent.
• Other job types (export, dump, restore, sweeps) keep Attempts = 0 and Delete; they do not read the cache.

Secret sweep and secrets admin API (AB#5539, AB#5544)

Implements phases 4 and 5 of the SECRET attribute value type (concept octo-construction-kit-engine/docs/concept-secret-attribute-type.md, §5.2–§6) on top of the engine's ISecretMaintenanceService; the REST contract is §9/§10 of octo-construction-kit-engine/docs/secret-frontend-handover.md. SecretSweepJob sweeps one tenant or every tenant of the instance (the system tenant first, then every registered tenant, child tenants included):

Every other mode keeps values with an unknown key id: they read as "key missing", are listed in the report's unreadable[] / secretsToReEnter[] and become readable again once their key is added to the ring (decisions 2026-10-06, item 2). Decrypt is never offered. Every writing mode first takes a fresh mongodump of the tenant, encrypts it with the key ring and stores it in the artifact store as presweep/<tenant>/<tenant>-<utc>-<guid>.presweep.octoenc (AB#5559/AB#5561, see "Artifact storage" below; never downloadable, deleted after BackupRetentionDays = 7 by the hourly cleanup, or earlier through the API); if that dump fails the tenant is skipped unless RequirePreSweepBackup=false. A pre-sweep dump is never stored unencrypted - without a key ring no writing sweep runs at all (a restore then only runs the key-free Verify). A writing run is followed by a Verify, so the report and the engine gauge octo.secrets.values describe the state after the sweep. After every repository restore the job runs Verify → Encrypt → Verify on the restored tenant and returns the secrets to re-enter in the restore job result - nothing is deleted after a restore. Sweeps of the same tenant exclude each other (Hangfire distributed lock): a sweep that finds its tenant busy is skipped, the post-restore sweep waits up to 30 minutes.

Every run (manual, recurring, restore) is kept in the tenant's run history (Hangfire storage, last 50 per tenant): run id = Hangfire job id, mode, trigger, outcome (Running while in progress), starter, counts and the dump state (createdAt, expiresAt, deletedAt, deletedBy; the hourly cleanup records deletedAt on expiry). skippedLegacyV1KeyMissing (report, step, run) counts the enc:v1 values a CleanupUnreadable kept because only the legacy key is missing (configuration gap, not key loss).

A run left in Running by a crashed or killed process is marked Failed with reason Interrupted (service restart) once, Bot:SecretSweep:InterruptedRunCheckDelaySeconds (default 120, 0 disables) after the service started: when its Hangfire job is no longer processing, or - if Hangfire still says processing - when the processing server is gone or sent no heartbeat since this start and the run started before it. Runs processing on a live server (another replica) are left alone.

Roles are tenant roles carried in the token (role claims); missing role → 403. The policies live in Configuration/BotAuthorizationPolicies.cs. The secrets routes match the token's tenant exactly (no parent-tenant administration marker, unlike the job routes).

Restoring a pre-sweep dump (restore-dump, job IRestorePreSweepDumpJob): the run is looked up in the route tenant's own run history, so a dump can only go back into the tenant it was taken from. The job decrypts the dump from the store into the scratch directory (never streamed into mongorestore - a tampered or truncated dump is detected before anything is restored), drops and replaces the tenant database with mongorestore (the same mechanism as the repository restore), deletes the scratch file and runs a Verify (trigger Restore, visible in the run history). 🔴 A dump taken before the environment's first Encrypt holds plaintext secrets: restoring it brings them back (secretSweep.remainingLegacyValues in the job result) - run Encrypt again afterwards. Legacy plaintext dumps on the local disk (before AB#5561) are restored from there.

Key-id retention: a key id must stay in the key ring until the newest dump encrypted with it has expired (7 days for pre-sweep dumps, 1 day for tenant dumps and staged uploads). GET …/secrets/status lists the key ids still needed (requiredKeyIds, read from the clear-text header of each .octoenc file, no decryption) and warns DumpKeyMissing when one is no longer in the ring - such a dump can neither be restored nor downloaded.

The Hangfire recurring job secret-sweep-encrypt is never scheduled and exists so the one-time Encrypt can be triggered from the dashboard.

Strict mode (Bot:SecretSweep:StrictModeSince, a UTC timestamp per environment, decision 10: 14 days after the sweep reported zero plaintext) means legacy clear text and enc:v1 are no longer acceptable. From that moment every sweep that still finds such values logs an error and reports the tenant in the gauge octo.secrets.strict_mode.violations{tenant} (meter Meshmakers.Octo.Secrets, registered by AddObservability()); alert on > 0. The engine itself does not refuse legacy reads yet.

Artifact storage (AB#5561, AB#5559)

At start-up the bot logs one info line (logger Meshmakers.Octo.Backend.Jobs.Services.ArtifactStoreStartupLog): provider, bucket or container, instance prefix, whether a CrateDB endpoint is set and whether copy staging is possible (yes/no with the reason). It never contains a key, secret, SAS token or a URI with credentials.

Pre-sweep dumps, tenant dumps (DumpRepositoryJob) and staged restore uploads live in the platform artifact store (Meshmakers.Octo.Services.ArtifactStorage, IArtifactStore; providers FileSystem, S3, AzureBlob), so they survive pod restarts and work with any number of replicas. Keys: <instancePrefix>/<category>/<tenant>/<file>.

copy-staging keys are nested: <instancePrefix>/copy-staging/<tenant>/<runId>/<archiveRtId>/data/<slice>/<file>.json.gz (and …/genmap/all/… for a rollup). A run deletes its own folder at the end (IBotArtifactStorage.DeleteRunAsync, returns the number of objects); the hourly cleanup removes the objects of a crashed run after the retention and logs each run it found (tenant, run id, number of objects). Its objects are never part of the key-id retention check (DumpKeyMissing looks at presweep and tenant-dumps only). The category needs its own lifecycle rule (expiry after one day) on each store.

• Tenant dumps contain secret ciphertext and every piece of business data, so they are encrypted too when a key ring exists. The job result is a store reference (octo-artifact:tenant-dumps/<tenant>/<file>); GET {tenantId}/v1/jobs/download?id=… (and the System variant) streams it and decrypts on the fly - the client receives the same plain .tar.gz / .octobak.zip under the same name as before (content type application/gzip, Content-Length = plaintext length). A decryption failure in the middle aborts the response, so a client never gets a short file that looks complete. Only the job's own tenant and only the tenant-dumps category are served; an expired dump answers 404.
• Restore uploads: restore-from-upload checks that the tus upload is complete (without copying it) and RestoreRepositoryJob assembles it into restore-staging/<tenant>/<tusFileId>[.octoenc], deletes its parts (see "Shared tus uploads") and reads the staged file (plain or .octoenc, and an uploaded .octoenc file of this environment is unwrapped as well), decrypting into the scratch directory; the staged artifact and the upload are consumed by the restore. A file staged by an earlier version, or received on the local disk before AB#8736, is still used from there.
• Archive data import uploads (archiveRtId, AB#8736): import-archive-data-from-upload checks that the upload is complete and enqueues the job with the tus file id; ImportArchiveDataJob copies the plaintext of the parts into the scratch directory (the ZIP reader needs a seekable file) on whichever replica runs it, and deletes the upload, a staged file of an earlier version and the scratch copy at the end, after a success as after a failure. A job queued before AB#8736 carries the local path of the upload instead; it is imported when that file is on the replica that runs it (inside the tenant's tus directory), and fails with "upload again" otherwise.
• Shared tus uploads (AB#8736): the bot runs with more than one replica and the ingress has no affinity, so the creation, the offset HEAD, every PATCH and the job start of one upload may each reach another replica. The tus endpoint therefore stores into the artifact store (ArtifactTusStore over ISharedUploadStore), not into the pod's disk (TusDiskStore before): restore-staging/<tenant>/<id>.tus-info (upload length and tus metadata) and one object per received PATCH body, <id>.tus-part.<offset>.<length>.<attempt>[.octoenc], written from a scratch copy of the body (bounded by the client part size; the bytes of a dropped connection are kept, so the client resumes from the offset). The offset is the end of the contiguous run of parts from byte 0, the same on every replica without a lock. A part is stored at the Upload-Offset its request was validated against, never at an offset read later, and under a key of its own (<attempt> is new per request, so a failed write never deletes another request's part); parts of two concurrent requests for the same offset hold the same bytes and count once. The artifact store has no separate temporary area: in-flight parts live in restore-staging and are removed by the consuming job, or after Bot:ArtifactRetentionHours by the hourly cleanup (lifecycle backstop one day), so an upload must complete and be consumed within that time. Nothing is copied inside a request: the completion leaves the upload as it is (a retried HEAD after a lost final response still answers the full offset), and the job start only checks it. tus extensions offered: creation and termination (no deferred length, checksum, concatenation or expiration; neither Studio nor the SDK uses them).
• Scratch: mongodump writes to and decryption reads from Bot:ScratchPath (default <temp>/octo-bot/scratch, owner-only; the chart mounts a scratch volume at /tmp). Scratch files are deleted right after use and by the hourly cleanup after Bot:FileRetentionHours.
• Health: /health includes the check artifact-storage (reachability; status Degraded, not part of readiness - an unreachable store breaks dumps, restores and writing sweeps, not the other jobs).

Configuration (section ArtifactStorage, environment OCTO_ARTIFACTSTORAGE__*; credentials only from a Kubernetes Secret, never in values or appsettings):

File system root precedence (AddOctoBotArtifactStorage): (1) ArtifactStorage:FileSystem:RootPath when set, for every category; (2) otherwise, if Bot:SecretSweep:BackupStoragePath (OCTO_BOT__SECRETSWEEP__BACKUPSTORAGEPATH) is set explicitly - the chart's PVC stop-gap - pre-sweep dumps are stored below that path (backwards-compatible alias) and the other categories below <temp>/octo-bot/artifacts; (3) otherwise <temp>/octo-bot/artifacts for everything. With S3 / AzureBlob the store holds every category. A file system root inside the tus or dump directory is refused at start-up (the hourly cleanup empties those).

Legacy local dumps (Bot:SecretSweep:BackupStoragePath/<tenant>/*.presweep.tar.gz, written before AB#5561) are not migrated: the run history keeps showing them, early deletion and restore still work on them, and the hourly cleanup deletes them when their 7 days are over.

Tenant backup formats (AB#4231, AB#5967)

A dump with archive data (Dump -iad in the CLI, the Studio checkbox, MCP dump_tenant) produces one <tenant>-<utc>-<guid>.octobak.zip. Download, upload and the commands are the same for both formats; which path ran is in the job log and in manifest.json (formatVersion).

Dump, format 2. Order: mongodump first, then every archive with a table (IStreamDataRepository.DumpArchiveToStagingAsync: data slices, and for a rollup the generation map read before and after, repeated when it moved), then the container: mongo.tar.gz, the generation maps, the data slices (byte copies of the staged files, stored without compression), manifest.json last. The manifest records per archive the tables with their columns, the slices (counted, copied and loaded row counts, staged bytes), the attempts and the moments of the generation map read and of the last data slice, and at the top the storage version of the source and the moment mongodump finished. The finished container is encrypted like any dump (including its archive entries) when a key ring exists. The run's copy-staging files are deleted at the end, also after a failure or a cancellation; a failure names the archive and how many archives and slices were done, and messages of the data store pass through the redaction of the staging location (no key, secret or SAS token in the job log, the result or an exception).

Restore, format 1 and 2 (Restore -rad, same upload as before; the format is detected). A mongodump, a ZIP without manifest and a format 1 container take the unchanged path (the upload is decrypted to the scratch directory and read with a ZIP reader). A format 2 container is read as a stream (ForwardZipReader, no seek, no copy of the container): the mongo.tar.gz entry goes to the scratch directory (the scratch usage is the mongodump blob plus a small constant), the archive entries are copied byte for byte into copy-staging/<tenant>/<runId>/…, the manifest is read last; an encrypted upload is read to its last byte, so a tampered or truncated file is detected before anything is restored. Then, per archive of the manifest that exists after the Mongo restore and passes the schema match: disable, drop the table, create it fresh (platform DDL), LoadArchiveFromStagingAsync (slices loaded by CrateDB COPY FROM, counted against the manifest; a slice counts as loaded when the loaded rows equal its count and none was rejected), restore the backed-up status. One archive's failure (including a row count that does not match) is reported and the loop goes on; before anything is restored the job asks the stream data factory (cluster level, no tenant context) whether the native copy is ready, and again the repository before the first archive table is touched (CheckStagingCopyReadinessAsync); a missing job log filter or a missing native copy fails the restore there, naming the reason, and nothing is restored when the factory answers not ready (the repository check after the Mongo restore is a second line of defence). Should the filter disappear while the loop runs, the loop stops after that archive and the rest are reported as not attempted. The run's staging files are deleted at the end. The job result lists per archive the outcome, rows and seconds, and backupFormatVersion. The restore works under the same or another tenant id (the files hold no tenant id). It fails before the Mongo restore with a message that names the missing piece when the backup is format 2 with archive data and the installation has no staging area (no factory, a FileSystem store, Azure without account key) or no native copy; an unknown formatVersion is refused listing the supported versions 1 and 2. The storage version recorded in the manifest and the target's are logged side by side; a difference is not refused.

Rollup reconcile (after the archives of every restore with archive data, format 1 and 2; AB#5967 slice 2). A recompute flag restored from the backup (RecomputeInProgress) is reset first, because it would suppress the orphan check. The restore-reconcile marker is set (reclaim, forward aggregation and recompute stand still for the tenant), the rollups are validated (ValidateRollupsAsync). A consistent restore needs no repair: the marker is cleared and the result reports ok. Otherwise RepairRollupsAsync (validate, repair, validate, synchronous inside the job) runs; it clears the marker only when the tenant validates ok afterwards. The result (rollupReconcile) carries per rollup the findings before, the repair steps with their status and the findings after. A failing step leaves the marker set and is named in failedStep; the restore itself has succeeded and the job does not fail.

Fallback to format 1 (the job log names the reason; nothing else changes): no artifact store or a FileSystem store, an Azure store without an account key (managed identity is phase 2), a stream data repository without native copy (SupportsStagingCopy false), or a CrateDB whose job log filter is missing (the readiness check, or ArchiveCopyRefusedException with reason JobLogNotFiltered).

The tenant gate was a no-op until AB#5054

TenantAuthorizationMiddleware inspects only principals whose Identity.AuthenticationType reads Bearer — its guard against false 403s on the cookie principal this service also issues. That label comes from TokenValidationParameters.AuthenticationType, which the JWT handler leaves at the framework default AuthenticationTypes.Federation unless the host sets it. This service did not, so the gate returned early on every bearer request. AB#5054 sets it in Configuration/ConfigureJwtBearerOptions.cs.

🔴 It also had to remove a second configurator: Program.cs passed AddJwtBearer(jwt => { jwt.TokenValidationParameters = new TokenValidationParameters { … }; }), which — because the options factory runs configurators in registration order — ran after ConfigureOptions<ConfigureJwtBearerOptions>() and replaced the whole instance, discarding both the explicit ValidIssuer and the label. It compiles, and an isolated unit test of the configurator stays green: octo-ai-services shipped a release in exactly that state (AB#5051 → AB#5056). The rule is now one configurator owning Authority, Audience, claim types, issuer and the label, with AddJwtBearer() taking no argument. (The OpenID Connect block in the same file legitimately assigns its own TokenValidationParameters — different options type.)

At the time of AB#5054 the gate still changed nothing here, by construction: every controller was routed system/v{version}/[controller], there was no {tenantId} route segment anywhere, and a job's target tenant travelled as a query argument (?tenantId=…) or as TUS upload metadata — the middleware reads the route value only, so it returned early on every request. The label was set anyway so that the first tenant-scoped route added here would arrive gated instead of silently unguarded, which is exactly the failure AB#5054 exists to remove. AB#5060 added those routes (see above) and the gate is now live on them.

This service keeps the platform default UserTokenEnforcement = Enforce and does not opt down to the LogOnly migration mode that asset-repo and the communication controller use: the tenant routes are new, so there is no installed caller base to stage for, and the deprecated System variants are untouched by the gate because they carry no route tenant.

Coverage: tests/Jobs.Tests/Configuration/TenantAuthorizationWiringTests.cs. This repo had no test project for the service host at all, so Jobs.Tests — its only unit-test assembly — gained a project reference to BotServices plus an InternalsVisibleTo entry; the alternative was leaving the security-relevant wiring of this host permanently untested.

Tenant authorization for service tokens (AB#5032 / AB#5047)

The request pipeline runs the shared TenantAuthorizationMiddleware from octo-common-services (app.UseOctoTenantAuthorization() in Program.cs, after UseAuthorization()): on every {tenantId}/... route it matches the route tenant against the caller's tenant_id claim. How client-credentials (service) tokens are treated is operator-settable, and Program.cs binds that setting with builder.Services.AddOctoTenantAuthorization(builder.Configuration) — configuration section TenantAuthorization, i.e. the environment variables

LogOnly changes no request outcome but logs every service token that addresses a tenant it was not issued for; Enforce answers those with 403, including a service token carrying no tenant_id at all. The Add… call is what makes the variables take effect — UseOctoTenantAuthorization() alone runs on the built-in defaults and the environment is ignored (AB#5047). Every OctoMesh service hosting the middleware (Identity, Communication Controller, Asset-Repo, Bot, MCP) binds the same section through the same helper, so one fleet-wide value reaches all of them. Semantics and the Enforce rollout rules are documented in octo-common-services/CLAUDE.md.

Running more than one replica: command consumers (AB#6910)

The six bus commands of the service (ImportCk, ImportCkBatch, ImportRt, ExportRtByQuery, ExportRtByDeepGraph, RemoveRecurringJobsByScheduleGroup) are consumed by every replica as competing consumers (queues …::shared, registered in src/BotServices/Consumers/BotCommandConsumers.cs). With two or more replicas each command is processed exactly once by one of them, and the caller gets the answer as before. Senders need no change: the consumers serve the classic command address (MigratesLegacyCommand = true) once no old exclusive replica is alive, so a rolling update from an older build loses or duplicates nothing. Requires Meshmakers.Octo.Common.DistributionEventHub 3.5.65 or newer (AB#8368). Details: CLAUDE.md, "Command consumers run on every replica".

Repository dump as an asynchronous CK method (AB#8445)

Besides POST {tenantId}/v1/jobs/dump-repository a tenant administrator can start the dump through the CK method System.Bot/TenantRepository.Dump-1 (static, execution.mode: async, role TenantManagement), e.g. with the GraphQL mutation of the asset repository. The call returns an operation handle at once; state, progress and the result are read with runtime.operations(operationId: …):

• The handler (src/BotServices/Methods/TenantRepositoryDumpHandler.cs) enqueues the same Hangfire job type (IDumpRepositoryJob.RunForOperation, AutomaticRetry(Attempts = 0)), so system/v1/jobs/{jobId}, the delete and both download routes answer for it exactly as for a REST dump (the tenant binding reads the tenantId argument). Old CLI versions (Dump -w) keep using the REST route and Run, which is unchanged.
• At most once. The job is enqueued inside the operation's work, after the operation record was written. Hangfire requeues a job whose server stopped or died; RunForOperation therefore marks its first start (job parameter OctoDumpStarted) and refuses a second one, and it refuses to start when its operation is no longer open (IDumpOperationGate on the operation store).
• Progress is written by the job as the job parameter OctoDumpProgress and relayed by the handler (poll interval Methods:RepositoryDump:PollInterval, default 2 s). mongodump runs as one process, so the progress is per stage and per archive, not per collection.
• Authorization. The method host validates the forwarded caller token (authority of this service, Methods:Token:*) and the declared role TenantManagement before the handler runs; the token's tenant must equal the tenant of the call. Unlike the REST route there is no parent-tenant administration: a parent administrator uses the REST route for a child tenant.
• Operation store. MethodOperations in the system database (AddCkMethodMongoOperationStore), shared with the asset repository that serves runtime.operations; records expire 7 days after completion.
• Rollout. System.Bot 3.7.0 (and 3.8.0) has minEngineVersion 3.5.67 (execution.mode): every service of an environment must run an engine that reads execution.mode before the model is published there.
• Tests: tests/Jobs.Tests/Methods/TenantRepositoryDumpHandlerTests and tests/Jobs.Tests/Jobs/Operations/DumpRepositoryJobForOperationTests.

Restore and archive data export/import as asynchronous CK methods (AB#8447)

System.Bot 3.8.0 adds three methods to System.Bot/TenantRepository, with the same handle, rules and operation store as Dump-1 (static, execution.mode: async, role TenantManagement):

• Synchronous checks. UPLOAD_NOT_FOUND and METHOD_VALIDATION_FAILED (empty tusFileId / databaseName, malformed archiveRtId, toUtc not after fromUtc) are answered at once, no operation is recorded. The upload is checked like the REST routes do (Services/StagedUploadCheck, shared): a staged file, a complete tus upload, or a file received before AB#8736; an upload of another tenant is "not found". Upload first through {tenantId}/v1/tus-upload.
• Same jobs, at most once. The work enqueues the job type the REST route enqueues (IRestoreRepositoryJob, IExportArchiveDataJob, IImportArchiveDataJob, method RunForOperation, AutomaticRetry(Attempts = 0)). OperationJobGuard (src/Jobs/Jobs/Operations) marks the first start (OctoOperationStarted), refuses a requeued job (*_INTERRUPTED) and a job whose operation was ended (IDumpOperationGate). system/v1/jobs/{jobId} and the downloads answer for these jobs exactly as for the REST jobs.
• Progress (job parameter OctoOperationProgress, relayed by Methods/JobOperationRelay, poll interval Methods:BotJobs:PollInterval, default 2 s): restore QUEUED → PREPARING_UPLOAD 5 % → RESTORING_DATABASE 20 % → RESTORING_ARCHIVES 50–85 % (archive, archiveCount) → SWEEPING_SECRETS 90 %; export EXPORTING_ROWS and import VALIDATING / IMPORTING_ROWS / FREEZING_ROLLUP_WINDOW report rows (every 10,000) without a percentage (the total is known only at the end). Every report carries jobId. The job writes a count summary (OctoOperationResult) that becomes the result.
• Failures carry jobId; reason another-job-running = the job waited more than 10 min for its DisableConcurrentExecution lock and never started; reason job-missing = the job vanished from the job storage. The job's error (schema mismatch, archive not disabled, unknown archive, mongorestore failure) is read through system/v1/jobs/{jobId}.
• Restore and the operation record. The restore replaces the tenant database; the record is in the system database and stays readable during and after it (Jobs.IntegrationTests/RestoreOperationRecordTests). A REST restore and a method restore of the same tenant use different job methods and are not serialized against each other (the same holds for the dump).
• Deleted tenant. TenantManagementConsumer (PosDeleteTenant) removes the tenant's operation records through ICkMethodOperationTenantPurge (octo-common-services); a failure is logged and does not block the delete. A restore (it drops and re-attaches the tenant) and a detach send the same event; the purge therefore runs only when the tenant's lifecycle record carries the Deleting tombstone the tenant delete writes first. The event is not a proof that the delete succeeded: the engine sends it from a finally after the database and archive drop, so a delete whose drop fails purges the records too (the tenant's registry row is already removed at that point). A tenant without a lifecycle record, or a failed purge, leaves the records to expire with the retention.
• Upload ids must name one file: BackupFileStorageService.GetTusUploadFilePath refuses separators, ./.. and paths that leave the tenant's upload directory, so the upload check answers UPLOAD_NOT_FOUND for them.
• system/v1/jobs is deprecated ([OctoRestDeprecated(Wave = "BotJobs1")] on the System API JobsController, link to the backup/restore documentation): calls are counted in octo.rest.deprecated.calls and OpenAPI marks the endpoints deprecated; the Deprecation / Sunset headers appear only when RestDeprecation:Waves:BotJobs1:Since|Sunset are configured. Do not set a sunset while other job types (ImportRt, ImportCk, RT exports, fixup run, secret sweep) still report only through it.
• Tests: tests/Jobs.Tests/Methods/TenantRepositoryJobMethodHandlerTests, tests/Jobs.Tests/Jobs/Operations/JobsForOperationTests, tests/Jobs.Tests/Consumers/TenantDeleteOperationPurgeTests, tests/Jobs.Tests/CkModels/SystemBotCkLanguage2Tests, tests/Jobs.IntegrationTests/RestoreOperationRecordTests, SystemBotCkLanguage2PersistenceTests.

Published packages

The repository produces two NuGet packages (the service host project itself is IsPackable=false and ships as a container image):

• Meshmakers.Octo.ConstructionKit.Models.System.Bot - the construction kit model System.Bot-3.7.0 (ckLanguage: 2 since 3.6.0, depending on System-[2.5,3.0); changes in src/SystemBotCkModel/ConstructionKit/CHANGELOG.md). It defines the entity types the bot service operates on:
  • Fixup - a named, ordered, scriptable migration entity tracking application state (IsApplied, AppliedAt, Output, Error, IsSuccess).
  • AttributeAggregateConfiguration - configures autocomplete value aggregation for a target entity attribute (filter regex, result limit) via the Configures association.
  • TenantRepository (3.7.0) - hosts the static async method Dump-1 (repository dump with an operation handle, see "Repository dump as an asynchronous CK method"); result record RepositoryDumpResult (JobId, DownloadPath).
• Meshmakers.Octo.Backend.Jobs - the reusable Hangfire job and command implementations library consumed by the service.

Project structure

Build

dotnet build Octo.Bots.sln

Test

dotnet test Octo.Bots.sln

Documentation

The complete OctoMesh documentation is available at https://docs.meshmakers.cloud.

License

Released under the MIT License.

Hangfire job storage: schema migration (AB#7196)

Hangfire.Mongo keeps its schema version in <job database>.hangfire.schema ({_id: 25} for Hangfire.Mongo 1.16.x). The storage is configured in HangfireMongoStorageSetup and never drops job data (recurring jobs = the cron triggers of all tenant pipelines):

• Older schema version: migrated in place (MigrateMongoMigrationStrategy); every Hangfire collection is copied first to <collection>.<fromVersion>.migrationbackup-<utc timestamp> (CollectionMongoBackupStrategy). The backups are not removed automatically - drop them once the migrated data has been verified.
• Schema document missing but Hangfire data present (e.g. after a database restore): the document is restored as version 24 first (the library would otherwise replay all legacy migration steps on current data and back up the wrong collections), then migrated to 25 with a backup. A database without Hangfire data is initialised normally.
• Schema newer than the library supports (downgrade), an unknown marker or more than one schema document: the service stops with a HangfireSchemaException that names the database and the found/supported version. Nothing is changed.

Covered by tests/Jobs.IntegrationTests/HangfireSchemaMigrationTests (real MongoDB via Testcontainers, needs Docker).