CasCap.Common.AI.Evaluation 4.15.0

CasCap.Common.AI.Evaluation

Evaluation harness for Microsoft Agent Framework agents built with CasCap.Common.AI. It asks an application's real agents realistic questions across a matrix of models, instruction variants and MCP tool-surface variants, and reports accuracy, latency, token usage, throttling and visible thinking with the statistics a probabilistic system needs.

Purpose

Hosted evaluation services cannot target a model on your own hardware, and they do not exercise your agent factory, tool filters or delegation. This library runs the production agent pipeline instead, so an edge small language model and a frontier cloud model are measured on identical terms:

• Real surface, synthetic state. Tools keep their production names, descriptions and JSON schemas. The scenario supplies their responses through the real return types, so the ground truth is known and runs are repeatable.
• Nothing reaches a real system. Query tools return fixtures; side-effect tools are recorded and answered with a sandbox acknowledgement, and calling one fails the run unless the scenario allows it.
• Delegation is real. Every agent in the tree is rebuilt against the model under test, so an orchestrator's hand-off and the sub-agent's own tool calls are both measured.
• Every request is recorded: duration, message count, instruction and tool-schema size, prompt, cached, output and reasoning tokens, visible thinking, and Azure SDK throttling.
• Variants are first-class. Rewrite instructions across the tree, rewrite tool or parameter descriptions, hide tools, allowlist an agent's tools, or add a candidate tool before implementing it.
• Results are statistics. Pass counts carry 95% Wilson intervals, and a session summary compares every model's speed with a reference model.

The library has no test-framework dependency; the consuming test project decides what to assert.

Data Flow

flowchart LR
    Scenario([AgentEvaluationScenario]) --> Harness
    Matrix([Models x variants x repetitions]) --> Harness
    Catalog[McpToolCatalog] --> Harness
    Harness[AgentEvaluationHarness] -->|AgentExtensions.CreateAgent| Agent[Agent tree]
    subgraph Run[One run]
        Agent -->|tool calls| Fixtures[FixtureToolFunction]
        Agent -->|every model request| Recorder[RecordingChatClient]
        Recorder --> Model[(Model under test)]
    end
    Fixtures --> Grader[AgentEvaluationGrader]
    Recorder --> Report([AgentEvaluationReport])
    Grader --> Report

Types

Usage

var catalog = McpToolCatalog.FromAssemblies(typeof(MyOrderMcpQueryService).Assembly);
using var throttling = new AzureThrottlingMonitor();
var harness = new AgentEvaluationHarness(loggerFactory, aiConfig, catalog,
    instructionsAssembly: typeof(MyOrderMcpQueryService).Assembly,
    tokenCredential: credential, config: evaluationConfig, throttlingMonitor: throttling);

var scenario = new AgentEvaluationScenario
{
    Id = "order-status",
    AgentKey = "OrderAgent",
    Question = "Has order 42 shipped?",
    Answer = AnswerExpectation.Matches(@"order 42 (?:has|is) shipped", "says order 42 shipped"),
    RequiredToolGroups = [["get_order_status"]],
    ToolResponses = new Dictionary<string, Func<AIFunctionArguments, object?>>
    {
        ["get_order_status"] = _ => new OrderStatus { OrderId = "42", Status = "Shipped" },
    },
};

var runs = new List<AgentEvaluationRun>();
foreach (var providerKey in evaluationConfig.ProviderKeys)
{
    await harness.WarmUpAsync(providerKey, TimeSpan.FromMinutes(30), cancellationToken);
    for (var repetition = 1; repetition <= evaluationConfig.Repetitions; repetition++)
        runs.Add(await harness.RunAsync(scenario, providerKey, InstructionVariant.Baseline,
            ToolSurfaceVariant.Baseline, repetition, TimeSpan.FromMinutes(5), cancellationToken));
}

var directory = await AgentEvaluationReport.WriteAsync(evaluationConfig.ResultsDirectory, scenario.Id, runs, cancellationToken);
var comparison = await AgentEvaluationReport.WriteSessionSummaryAsync(directory,
    evaluationConfig.ReferenceProviderKey ?? evaluationConfig.ProviderKeys[0], cancellationToken);

Run scenarios sequentially: a shared model server and the process-wide throttling monitor both assume one run at a time.

Configuration

Bound from CasCap:AgentEvaluationConfig. Keep provider endpoints in gitignored local configuration.

Configuration Examples

Minimal — one model, defaults for everything else:

{
  "CasCap": {
    "AgentEvaluationConfig": {
      "ProviderKeys": [ "Edge" ]
    }
  }
}

Full — an edge model compared with two cloud deployments, with every switch set:

{
  "CasCap": {
    "AIConfig": {
      "Providers": {
        "Edge": { "Type": "OpenAI", "Endpoint": "https://llm.example.com", "ModelName": "example-edge-model", "ApiKey": "none" },
        "CloudMini": { "Type": "AzureOpenAI", "Endpoint": "https://example.openai.azure.com/", "ModelName": "example-mini-deployment" },
        "CloudFlagship": { "Type": "AzureOpenAI", "Endpoint": "https://example.openai.azure.com/", "ModelName": "example-flagship-deployment" }
      }
    },
    "AgentEvaluationConfig": {
      "ProviderKeys": [ "Edge", "CloudMini", "CloudFlagship" ],
      "ReferenceProviderKey": "Edge",
      "Repetitions": 5,
      "InstructionVariants": [ "answer-directly" ],
      "ToolSurfaceVariants": [],
      "MinimumPassRate": 0.6,
      "RunTimeoutSeconds": 300,
      "WarmUpTimeoutSeconds": 1800,
      "ResultsDirectory": "TestResults/AgentEvaluation",
      "ProviderReasoningEffortApplied": false,
      "EndpointToolsMapped": false
    }
  }
}

Results

Each session writes a timestamped folder under ResultsDirectory:

Dependencies

License

This project is released under The Unlicense.