Lyo.Query 3.0.0

Lyo.Query

Execution engine for Lyo.Query.Models filter trees. IWhereClauseService / BaseWhereClauseService turns polymorphic WhereClause ASTs (condition / group, optional SubClause) into LINQ expression trees on IQueryable<T>, applies multi-key SortBy, and can evaluate / explain the same AST against a loaded entity in memory.

What this package is. AST → expression / matcher. What it is not. HTTP endpoints, QueryConcreteReq / ProjectionQueryReq / QueryReq builders, or query- result caching. Those live in Models + Lyo.Api.

EF-agnostic at the boundary (works on any IQueryable, including EF DbSet and in-memory). Needs ICacheService + CacheOptions for compiled predicate / matcher / metadata caches.

Targets net10.0.

Features

• AST → IQueryable. ApplyWhereClause builds expression trees from WhereClause that EF (or the provider) can translate.
• 18 comparators. Equals / NotEquals / Contains* / StartsWith* / EndsWith* / Greater* / Less* / In / NotIn / Regex / NotRegex / Fuzzy / NotFuzzy.
• Collection paths. Lines.Quantity becomes Any(...) on collection elements. Bare collection / .Count style paths compare against count.
• Two-phase SubClause. includeSubClauses: false for the SQL root pass. Load includes then match the sub-tree in memory.
• Multi-key sort. SortByProperty + ApplyOrdering using Priority and a stable default tie-break.
• In-memory match + explain. MatchesWhereClause / ExplainMatch with blocking path and OR-branch detail. Use ExplainMatch(...).ToErrors() for structured validation errors.
• ICache hot path. Compiled EF predicates (filter_ef_predicate:*), matchers (filter_matcher:*), sort keys, and property-path metadata.
• Path validation. Bad dotted paths raise InvalidQueryException via TryValidatePropertyPath.
• Contains→Regex coalesce. Adjacent Contains leaves on the same string field can merge into a single Regex alternation.
• Property diffing. IPropertyComparisonService for patch and update pipelines.
• Value coercion. IValueConversionService maps JSON literals → CLR property types (suppressible when Lyo.Api owns conversion).
• QueryProject aggregates. AggregateQuery groups an IQueryable by Select and projects Count, CountDistinct, Sum, Average, Min, and Max. Empty Select is one summary row. Count with no field counts rows in the group. A collection field, or a path ending in .count, totals that collection. Select and aggregate paths accept [n], [start:], [:end], [start:end], and [:] on a string or a collection.

Examples

Add query services

using Lyo.Query;

services.AddLocalCache(/* … */); // or AddFusionCache(...)
services.AddLyoQueryServices();
// registers IValueConversionService, IPropertyComparisonService, IWhereClauseService (singletons)

// Hosting Lyo.Api (richer ITypeConversionService already registered):
// services.AddLyoQueryServices(registerValueConversion: false);

Filter an IQueryable

using Lyo.Query;
using Lyo.Query.Models;
using Lyo.Query.Models.Builders;

var where = WhereClauseBuilder.And()
    .Equals("Status", "Active")
    .GreaterThan("Age", 18)
    .Build();

IQueryable<Person> people = db.People.AsQueryable();
var filtered = whereClauseService.ApplyWhereClause(people, where);
var page = await filtered.Skip(20).Take(20).ToListAsync(ct);

SQL first, then in-memory SubClause

// Root runs as SQL; SubClause refined after Include load
var where = WhereClauseBuilder.And()
    .Equals("Status", "Open")
    .AddSubClause(sub => sub.Contains("Tags", "vip"))
    .Build();

var sqlPhase = whereClauseService.ApplyWhereClause(db.Orders, where, includeSubClauses: false);
var includePaths = whereClauseService.GetCollectionIncludePathsForWhereClause<Order>(where);
// apply Include(includePaths…) then materialize
var rows = await sqlPhase /* .Include(...) */ .ToListAsync(ct);
var matched = rows.Where(o => whereClauseService.MatchesWhereClause(o, where)).ToList();

Order by several keys

using Lyo.Query.Models;
using Lyo.Common.Core.Enums;

var sorted = whereClauseService.ApplyOrdering(
    queryable: people,
    sortByProps:
    [
        new SortBy { PropertyName = "LastName", Direction = SortDirection.Ascending, Priority = 0 },
        new SortBy { PropertyName = "FirstName", Direction = SortDirection.Ascending, Priority = 1 },
    ],
    defaultOrder: p => p.Id,
    defaultSortDirection: SortDirection.Ascending);

Match plus ExplainMatch

var entity = await db.People.AsNoTracking().FirstAsync(ct);

if (whereClauseService.MatchesWhereClause(entity, where))
{
    var explain = whereClauseService.ExplainMatch(entity, where);
    // explain.Passed, BlockingPath, FailureSummary, Nodes (per-condition ActualValueSummary)
    // In-memory only — not for SQL-only pipelines
    IReadOnlyList<Error> errors = explain.ToErrors(); // structured validation errors; empty when Passed
}

Validate paths and collection includes

if (!whereClauseService.TryValidatePropertyPath<Person>("Addresses.City", out var err))
    throw new InvalidOperationException(err);

foreach (var path in whereClauseService.GetCollectionIncludePathsForWhereClause<Person>(where))
    query = query.Include(path); // e.g. "Addresses" for "Addresses.City"

Fingerprints from WhereClauseHelpers

using Lyo.Query.Services.WhereClause;

var hash = WhereClauseHelpers.GetWhereClauseTreeHash(where); // stable structural fingerprint
var hasSub = WhereClauseHelpers.HasAnySubClause(where);
if (WhereClauseHelpers.TryExtractConditions(where, out var conditions, out var op))
{
    // flat And/Or leaves — false when SubClause or unsupported nesting present
}

Diff properties for a patch

var diffs = propertyComparisonService.GetPropertyDifferences(entity, patchDto);
// Dictionary<propertyName, newValue> — only changed writable props

Benchmarks

• Portfolio suite: query

Package map

Builders and endpoint payloads are documented on Models / Api. This doc covers the runtime translator.

QueryProject aggregates

AggregateQuery groups an IQueryable<T> when ProjectionQueryReq.Aggregates is non-empty. Select is the group key. An empty Select is one summary row over the filtered set. WhereClause filters rows before grouping. SortBy may name a select path or an aggregate output name. Start / Amount page groups, and an exact total is the number of groups.

Count with no field is COUNT(*) (every row in the group). Count of a collection, or of a path ending in .count, totals that collection's elements in the group. Count of a scalar skips nulls and keeps duplicates. CountDistinct counts distinct values of a scalar field and skips nulls. Sum and Average take a numeric field; Average of an integral field is double, and decimal stays decimal. Min and Max take a scalar field. An empty ungrouped source returns one row: Count and CountDistinct are 0, a non-nullable Sum is 0, and a nullable Sum / Average / Min / Max is null. A grouped query over no rows returns no items.

A path segment may end with [n], [start:], [:end], [start:end], or [:]. start is inclusive and end is exclusive. On a string the window is still a string, so it can be a group key. On a collection, entity elements order by a single [Key] or Id, and scalar elements order by their value. A one-element window is a scalar. A longer collection window is a select list and is rejected as a group key.

Lyo.Api runs this before the entity projection. If the provider cannot translate the grouping, the call fails instead of grouping a loaded page.

IWhereClauseService

BaseWhereClauseService is the default implementation in DI.

Invalid paths / operators → InvalidQueryException (Lyo.Query.Models.Exceptions).

Comparator and path semantics

Operators come from ComparisonOperatorEnum on each ConditionClause:

Dotted paths

• Scalar: Status, Address.City
• Into collection: Lines.Quantity → lines.Any(e => e.Quantity …)
• Collection count: path metadata IsCountPath builds Enumerable.Count then compares

Group optimization: within an AND/OR group, multiple Contains leaves on the same string field may be coalesced into a single case-insensitive Regex alternation before expression build.

Two-phase SubClause

WhereClause.SubClause (on a condition or group) supports split execution. Lyo.Api query pipelines use this often:

1. Phase 1 (DB). ApplyWhereClause(..., includeSubClauses: false) so only the primary predicate becomes SQL.
2. Load. GetCollectionIncludePathsForWhereClause → EF Include for collection segments referenced by the sub-tree.
3. Phase 2 (memory). MatchesWhereClause / ExplainMatch on materialized entities with the complete tree (includeSubClauses default true).

Helpers:

• WhereClauseHelpers.HasAnySubClause(node). Detect whether a two-phase path is needed.
• WhereClauseHelpers.TryExtractConditions. Flatten simple And/Or trees for projection-level filtering (returns false if any SubClause).
• WhereClauseHelpers.GetWhereClauseTreeHash. Structural fingerprint for cache keys / logging (not cryptographic).

var where = WhereClauseBuilder.And()
    .Equals("Age", 10)
    .AddSubClause(sub => sub.AddAnd(s => s.Equals("Name", "Alice")))
    .Build();

How ICache holds predicates

AddLyoQueryServices requires ICacheService + CacheOptions (e.g. AddLocalCache / AddFusionCache). That is not the same as Lyo.Api query- result caching (QueryOptions.CacheQueryResultsAsUtf8Payload).

Tags typically include the entity CLR type so hosts can invalidate by type when schemas change.

Api result cache (optional UTF-8 payload entries for /QueryConcrete + /QueryProject) is documented under Lyo.Api query result caching.

IPropertyComparisonService

GetPropertyDifferences<TEntity, TOther>(entity, newData) walks public writable properties on TEntity that have a same-named readable property on TOther, compares with an inferred strategy (direct equality, enum/string coercion, or Convert.ChangeType), and returns Dictionary<string, object?> of changed property → proposed new value.

Patch/update pipelines (e.g. Lyo.Api) use this to build minimal diffs. Strategies are cached via ICacheService.

IValueConversionService

• ConvertToTargetType(value, targetType). JSON literals → CLR (primitives, nullables, Guid, enums, date/time, lists)
• GetUnderlyingType(type). Strip Nullable<T>
• IsObjectEnumerable(value). Non-string / non-byte[] IEnumerable

When hosting Lyo.Api, register Api's ITypeConversionService (extends this interface) and call:

services.AddLyoQueryServices(registerValueConversion: false);

Otherwise two IValueConversionService registrations collide.

Registration

services.AddLocalCache(/* … */); // prerequisite
services.AddLyoQueryServices(registerValueConversion: true);

Registers as singletons:

• IValueConversionService → ValueConversionService (unless registerValueConversion: false)
• IPropertyComparisonService → PropertyComparisonService
• IWhereClauseService → BaseWhereClauseService

Optional: IMetrics is taken from DI when present and forwarded only if CacheOptions.EnableMetrics is set.

Metrics

Recording lives on WhereClauseEvaluator in Lyo.Query.Evaluation, so these fire whichever implementation you use. BaseWhereClauseService forwards the container's IMetrics only when CacheOptions.EnableMetrics is true; otherwise nothing is recorded. Names come from Lyo.Query.Constants.Metrics:

Each is tagged entity_type (Constants.Metrics.Tags.EntityType).

Related docs

• DTOs / builders / request shapes → Lyo.Query.Models
• HTTP endpoints, projection SQL, result caching → Lyo.Api Query & Request Builders
• Blazor workbench → Lyo.Query.Web.Components
• Portfolio benchmarks suite → query

Links

• Lyo.Query.Models
• Lyo.Api query builders
• Lyo.Api query result caching

Dependencies

Generated from ProjectReference / PackageReference (same model as docs/Lyo.ProjectGraph.html).

• Lyo.Cache (direct, lyo)
• Lyo.Common.Core (direct, lyo)
• Lyo.Exceptions (direct, lyo)
• Lyo.Metrics (direct, lyo)
• Lyo.Query.Evaluation (direct, lyo)
• Lyo.Query.Models (direct, lyo)
• Microsoft.Extensions.DependencyInjection.Abstractions 10.0.5 (direct, microsoft)
• Lyo.Common.Metadata (transitive, lyo)
• Lyo.Compression (transitive, lyo)
• Lyo.Encryption (transitive, lyo)
• Lyo.Hashing (transitive, lyo)
• Lyo.Health (transitive, lyo)
• Lyo.Json (transitive, lyo)
• Lyo.KeyStore (transitive, lyo)
• Lyo.Parameters (transitive, lyo)
• Lyo.Result (transitive, lyo)
• Lyo.Streams (transitive, lyo)
• BouncyCastle.Cryptography 2.6.2 (transitive, third-party, netstandard2.0)
• EasyCompressor 2.1.0 (transitive, third-party)
• Konscious.Security.Cryptography.Argon2 1.3.1 (transitive, third-party)
• Microsoft.Bcl.AsyncInterfaces 10.0.5 (transitive, microsoft, netstandard2.0)
• Microsoft.Extensions.Caching.Memory 10.0.5 (transitive, microsoft)
• Microsoft.Extensions.Configuration.Binder 10.0.5 (transitive, microsoft)
• Microsoft.Extensions.DependencyInjection 10.0.5 (transitive, microsoft)
• Microsoft.Extensions.Logging.Abstractions 10.0.5 (transitive, microsoft)
• Microsoft.Extensions.Options.ConfigurationExtensions 10.0.5 (transitive, microsoft)
• System.Buffers 4.6.1 (transitive, microsoft, netstandard2.0)
• System.IO.Hashing 10.0.5 (transitive, microsoft, net10.0)
• System.Memory 4.6.3 (transitive, microsoft, netstandard2.0)
• System.Text.Encoding.CodePages 10.0.5 (transitive, microsoft)
• System.Text.Json 10.0.5 (transitive, microsoft, netstandard2.0)
• System.Threading.Tasks.Extensions 4.6.3 (transitive, microsoft, netstandard2.0)