Skip to content

Content Download

github-actions[bot] edited this page Oct 28, 2026 · 5 revisions

Content Download and Caching System

Overview

MakaMek uses a content downloading and caching system to handle unit data files that are not included in the application binary. This system ensures efficient loading while providing persistent storage across sessions.

Architecture

Core Components

  1. MainMenuViewModel - Entry point for content loading
  2. UnitCachingService - Orchestrates unit loading and caching
  3. GitHubResourceStreamProvider - Downloads content from GitHub
  4. IFileCachingService - Platform-specific caching implementations

Content Flow

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│ MainMenuViewModel│───▶│UnitCachingService│───▶│GitHubResource   │
│                 │    │                  │    │StreamProvider   │
└─────────────────┘    └──────────────────┘    └─────────────────┘
                                │                        │
                                ▼                        ▼
                       ┌──────────────────┐    ┌─────────────────┐
                       │  Caches unit     │    │IFileCaching     │
                       │  data/images     │    │Service          │
                       └──────────────────┘    └─────────────────┘

Unit Data Format

MMUX Packages

Unit data is distributed as MMUX packages (MakaMek Unit eXchange format):

unit.mmux
├── manifest.json        # Format version (v1 or v2)
├── unit.json            # Unit specifications (JSON)
├── unit.png             # Static unit image, also the reduced-motion fallback
├── unit-sprites.png     # Optional (v2): one animation row per clip
└── unit-sprites.json    # Optional (v2): cell geometry, rows/fps/loop, distancePerCycle

unit.json contains:

  • Model name and specifications
  • Component definitions
  • Movement and combat stats
  • Weight class and type information

manifest.json is optional for older packages: a missing manifest or a 1.x version is read as v1. A 2.x version adds the sprite pair; a package without it still loads as v1. Other major versions are rejected with an unsupported-version error, and a broken sprite pair falls back to the static image instead of dropping the unit. MmuxPackager writes v2 only when --sprites-source supplies <Model>/unit-sprites.png and <Model>/unit-sprites.json for a unit.

Platform-Specific Implementations

Desktop/Mobile (FileSystemCachingService)

WebAssembly Browser (BrowserCachingService)

// IndexedDB-based persistent caching
// Same SHA256 hash logic for consistency
// Data stored as Uint8Array in IndexedDB

[JSImport("getFromCache", "cacheStorage")]
[return: JSMarshalAs<JSType.Promise<JSType.Object>>()]
private static partial Task<JSObject> GetFromCacheAsObjectJs(string cacheKey);

Caching Strategy

Cache Key Generation

Both implementations use identical SHA256 hashing:

private static string GetHashedCacheKey(string originalKey)
{
    var hashBytes = SHA256.HashData(Encoding.UTF8.GetBytes(originalKey));
    return Convert.ToHexString(hashBytes).ToLowerInvariant();
}

This ensures cache compatibility across platforms.

Initialization Process

App Startup Sequence

  1. MainMenuViewModel loads on startup
  2. PreloadUnits() runs in background
  3. UnitCachingService.EnsureInitialized() triggers
  4. LoadUnitsFromStreamProviders() processes each provider
  5. GitHubResourceStreamProvider downloads unit files
  6. Files cached for future sessions

Loading States

Loading Content... → Loading Items... → {X} Items Loaded

Error Handling

Graceful Degradation

  • Network failures: Continue with cached content
  • Corrupted files: Skip and continue with other units
  • Provider failures: Log errors but continue with other providers

Storage Characteristics

Desktop/Mobile

  • Location: User application data directory
  • Format: Binary files with .cache extension
  • Persistence: Until manually cleared
  • Size: Limited by available disk space

Browser (WASM)

  • Location: IndexedDB (MakaMekCache database)
  • Format: Uint8Array objects in fileCache store
  • Persistence: Until browser data cleared
  • Size: Limited by browser storage quotas (~60% of disk space)

Browser DevTools Inspection

IndexedDB Inspection

  1. Open DevTools (F12)
  2. Go to Application tab
  3. Navigate to Storage → IndexedDB
  4. Select MakaMekCache database
  5. Inspect fileCache object store

Console Debugging

// Check database contents
const request = indexedDB.open('MakaMekCache');
request.onsuccess = (event) => {
    const db = event.target.result;
    const transaction = db.transaction(['fileCache'], 'readonly');
    const store = transaction.objectStore('fileCache');
    const getAllRequest = store.getAllKeys();
    getAllRequest.onsuccess = () => {
        console.log('Cached files:', getAllRequest.result);
    };
};

Performance Considerations

Parallel Loading

// Process units in parallel batches
var batches = unitIdList.Chunk(MaxDegreeOfParallelism);
foreach (var batch in batches)
{
    var batchTasks = batch.Select(unitId => ProcessUnitAsync(provider, unitId));
    await Task.WhenAll(batchTasks); // Parallel execution
}

Testing and Debugging

Cache Verification

  1. Clear cache before testing
  2. Monitor network tab for initial downloads
  3. Check cache storage after loading
  4. Refresh page - should load from cache (no network requests)

Common Issues

Issue Symptom Solution
Cache miss Files re-downloaded Check cache key consistency
Empty cache No IndexedDB entries Verify save operations
Wrong format Uint8Array issues Check marshalling code
Network errors Download failures Check GitHub API access

R2 Release Pipeline

Unit and terrain assets under data/ are also published to a Cloudflare R2 bucket by a dedicated release workflow (.github/workflows/deploy-data-release.yml). Git remains the source of truth; the bucket is a flat mirror of the data/ folder at the last released tag.

The bucket is always flat — there are no versioned subfolders and no per-release copies. Re-releases simply sync the single copy in place. Versioning happens at the bucket level: a breaking change (one the app's manifest/format can't read) is published to a new bucket, and the app's base URL is switched to point at it. Within any given bucket the layout stays flat.

Pipeline Flow

  1. Bucket provisioning: the R2 bucket is created upfront via Pulumi (src/MakaMek.Infra/MakaMek.Infra.Data, run by .github/workflows/infra-data.yml, Cloudflare provider). See hub-deployment.md for the same ad-hoc workflow pattern. A new bucket (and corresponding base URL) is provisioned whenever a breaking change ships; non-breaking releases reuse the existing flat bucket.
  2. Trigger: push of a v* tag.
  3. Manifest generation: .github/scripts/generate-data-manifest.cs (a .NET 10 file-based app run via dotnet run --file) scans each asset type's subtree of data/ (one manifest per immediate subdirectory: units, hexes, …) and writes a manifest.json at the root of each asset type (e.g. data/units/manifest.json, data/hexes/manifest.json). Each asset type's resource provider fetches only its own manifest, so no single shared listing is downloaded by every provider.
  4. Upload: aws s3 sync (S3-compatible endpoint, region auto) mirrors the whole data/ folder into the bucket with --delete, so removed files disappear from the bucket. Because the manifests are generated before the sync and live inside the data/ tree, this single sync publishes them together with the assets.

Configuration uses repository secrets (R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, CLOUDFLARE_ACCOUNT_ID, CLOUDFLARE_R2_BUCKET) and the public base URL variable vars.DATA_R2_BASE_URL.

manifest.json Schema

One manifest per asset type, e.g. units/manifest.json:

{
  "version": "0.63.6",
  "generatedAtUtc": "2026-08-27T00:00:00.000Z",
  "fileCount": 200,
  "files": [
    {
      "path": "units/mechs/Atlas.mmux",
      "name": "Atlas.mmux",
      "hash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
      "url": "https://<data-r2-domain>/units/mechs/Atlas.mmux"
    }
  ]
}

Fields

Top level:

  • version: App release version (<VersionPrefix> from Directory.Build.props, picked up by the deploy pipeline and passed to the manifest generator). Lets clients detect when a new app release ships updated downloadable content.
  • generatedAtUtc: ISO-8601 timestamp of generation
  • fileCount: Number of entries in files
  • files: Array of file entries

Per-file:

  • path: Path relative to data/ (forward slashes, includes subfolders)
  • name: File name (last segment of path)
  • hash: SHA-256 of the raw file bytes (used as an opaque cache-version marker)
  • url: Public download URL (DATA_R2_BASE_URL + / + path)

These per-asset-type manifests are unrelated to the per-package manifest.json inside each .mmux/.mmtx archive — those schemas are unchanged.

Current Status

The existing GitHub-based runtime path (GitHubResourceStreamProvider) stays active for local development. In release builds the app consumes the per-asset-type manifest (e.g. units/manifest.json) from the bucket via BucketResourceStreamProvider and downloads files via their url field.

Possible Future Enhancements

  • Delta Updates: Only download changed files
  • Delta Updates: Only download changed files
  • Compression: Reduce storage size
  • Preloading: Background content preparation

Clone this wiki locally