PollySdk.Documents
0.2.8
dotnet add package PollySdk.Documents --version 0.2.8
NuGet\Install-Package PollySdk.Documents -Version 0.2.8
<PackageReference Include="PollySdk.Documents" Version="0.2.8" />
<PackageVersion Include="PollySdk.Documents" Version="0.2.8" />
<PackageReference Include="PollySdk.Documents" />
paket add PollySdk.Documents --version 0.2.8
#r "nuget: PollySdk.Documents, 0.2.8"
#:package PollySdk.Documents@0.2.8
#addin nuget:?package=PollySdk.Documents&version=0.2.8
#tool nuget:?package=PollySdk.Documents&version=0.2.8
Knowledge as a standalone library
General document processing (independent of Knowledge)
src\Documents\Polly.Documents.csproj / local package PollySdk.Documents targets .NET 10
and has no Knowledge, Storage, Agent, embedding, database, or application dependency.
Polly.Extensions.Knowledge.IDocumentProcessor still means chunking and is unchanged.
using Polly.Documents;
services.AddNativeDocumentExtraction(); // Also registers IDocumentProcessingService.
var processing = provider.GetRequiredService<IDocumentProcessingService>();
var result = await processing.ProcessAsync(new DocumentProcessingRequest(stream, mediaType)
{
FileName = displayName, // Descriptive only; never used to guess format or write a path.
ExpectedSha256 = knownSourceHash,
Profile = new DocumentProcessingProfile { Id = "native-v1", ExtractorId = "native" },
Limits = new DocumentLimits { MaxInputBytes = 32 * 1024 * 1024, MaxUnits = 1000 },
}, ct);
The caller owns the input stream. It is read from its current position and never disposed or
modified. Processing takes a bounded private snapshot and validates the optional SHA-256.
Hosts retain the original file; Content.RawText is an extracted text projection, not the file.
Content.NormalizedText only normalizes CRLF/CR to LF (line-endings-lf-v1). TXT and MD
retain source syntax, not a rendered Markdown interpretation. No placeholders pretend to be OCR.
DocumentProcessingResult schema 1.0 is System.Text.Json round-trippable, including
discriminated block and locator subclasses. It includes the actual profile/limits, source
hash, provider operations/versions/library versions/configuration fingerprints, issues, and
derived renditions. No credentials are recorded. Block IDs are deterministic within the exact
source hash and profile, not stable across edits. XML element paths use local names and
one-based sibling positions. Paragraph/table/row/cell indexes are zero-based; source PDF pages,
slides and sheet order are one-based. Native IDs (slide ID, shape ID, sheet ID) remain distinct.
PostgreSQL jsonb and other key-reordering stores can move the kind discriminator away
from the first property. Use .NET 10 JsonSerializerOptions.AllowOutOfOrderMetadataProperties=true
when reading persisted results from such stores. A default in-memory JSON roundtrip does not
prove a JSONB roundtrip. Store schema version with the artifact and retain compatible readers;
enums use numeric JSON values with the default serializer.
| Native format | Structure and locator | Deliberate limits |
|---|---|---|
| Real source page number and dimensions, text-layer blocks | No invented text bounding boxes; geometry crop/rotation fields stay unknown when not read | |
| DOCX | Main-body XML part/path, paragraphs, tables/cells, explicit outline headings, OfficeMath source, embedded-image references | No native pagination, layout reconstruction, headers/footers, or lossless rendering promise |
| PPTX | Presentation relationship order; native slide/shape IDs and paragraph paths; tables and pictures | XML order, not visual reading order; notes only with IncludeSlideNotes; chart/SmartArt opaque warnings |
| XLSX | Sheet identities/order, cells, merge ranges, source data type, raw cached value, resolved shared text, formula text/shared metadata and style | No recalculation or display formatting; WasRecalculated=false; 1900/1904 date system kept separately (including serial 60, without inventing a date) |
| TXT/MD/other declared text | Decoded source character offsets and encoding | Not byte offsets or fabricated pages |
Empty PDF text is NoTextLayer with no_text_layer warning: a page may be truly blank or
scanned. Native extraction still succeeds; it does not assert OCR completion. Recognition is
opt-in (RecognizerId plus Recognition=MissingText or Always). MissingText keeps native
text units and replaces only no-text units with the same native kind/index. A missing requested
OCR unit is an error issue and makes the result Partial. Recognition confidence is a model
score with provider/model/meaning, never reviewed evidence or an accuracy guarantee.
Capabilities declares MIME patterns, operation, precision, limitations, and configuration
availability separately. Selection uses explicit provider IDs or first registered extraction
provider (recorded in executions), never a silent fallback. Optional recognition/conversion is
never selected automatically. RenditionMediaType requires an explicit ConverterId.
Renditions contain a bounded byte snapshot, OpenRead() (caller disposes), output/source hashes,
provider and rendition page count; those page numbers do not address source DOCX/PPTX/XLSX.
Embedded image assets are internal part references, not external URLs. Use
DocumentAssets.OpenPackagePartAsync(source, sourceHash, asset, limits, ct) to obtain a bounded,
caller-owned read-only stream; verify/serve its media type safely rather than rendering active XML.
IDocumentContentExtractor, IDocumentRecognizer, and IDocumentConverter are small
replacement seams. They receive DocumentInput.OpenRead() snapshots and a processing context;
providers dispose the streams they open. No plugin discovery, persistent task store, hidden
cloud upload, or database writes are involved. Hosts own durable versions, task leases and
progress persistence. Call ProcessAsync once with the complete profile. The SDK owns
native/OCR/conversion selection and the explicitly configured partial-failure policy; the
host must not re-run the whole entry point once per stage to reconstruct the same pipeline.
OnProgressAsync is an awaited Func<DocumentProgress, CancellationToken, ValueTask>:
var result = await processing.ProcessAsync(new DocumentProcessingRequest(stream, mediaType)
{
Profile = profile,
OnProgressAsync = async (stage, token) =>
{
await hostStageStore.RecordAsync(stage.ExecutionId, stage.Operation, stage.ProviderId,
stage.State, stage.Reason, stage.ErrorCode, token);
},
}, ct);
It emits ordered Started, Completed, Skipped, and Failed events with actual provider
identity and the final Execution record where available. The next provider does not run
until the callback finishes. Callback exceptions always abort; even a typed callback error
cannot be swallowed as an optional provider failure. Callbacks must honor cancellation and
must not start unawaited work. Host persistence still needs its own lease/generation fences.
Legacy IProgress<DocumentProgress> remains start-only for synchronous/UI compatibility;
do not use an async void progress handler for database writes.
Executions.State, WasSkipped, and SkipReason distinguish decisions from real execution.
MissingText with no no-text-layer units records a skipped recognition stage with reason
no-units-requiring-recognition, without resolving or calling an unused OCR provider.
This also accurately covers empty non-raster documents; it does not assert they contain text.
No compatible native extractor (for example, image input) records no-compatible-provider;
recognition can then supply the content. Operations disabled by the profile are not requested
and have no stage record. Skips have Succeeded=false, no error and no configuration fingerprint;
they never claim successful inference. A returned partial provider result can be Completed
while the overall result remains Partial with its error issues. Cancellation/deadline throws,
does not emit a false completion, and leaves final job-cancellation persistence to the host.
Pipeline version 1.1 identifies these reporting semantics; native provider versioning is separate.
Input bytes, expanded ZIP bytes/entry count, units, blocks, cells, text, outputs and deadlines are
bounded. Exceeding a limit is an explicit DocumentProcessingException with Code/ProviderId;
unsupported media/provider, wrong MIME signature/container, hash mismatch and unavailable
configuration also fail explicitly. Caller cancellation propagates OperationCanceledException.
Deadlines produce timeout. Synchronous native library parse calls cannot be interrupted
mid-call; cancellation is checked between reads/units and before returning/publishing a result.
Partial results contain error-severity issues; they are never silently reported as full success.
The default aborts on a failed optional stage. Set Profile.ContinueOnOptionalFailure=true
explicitly to keep already-extracted content on a typed recognition/conversion error. The result
is Partial, includes the error issue and a failed execution (Succeeded=false, ErrorCode;
configuration fingerprint is empty because no successful provider result was received).
Caller cancellation/deadline still throws and never publishes a partial success. If no content
exists, optional-stage errors still throw rather than manufacturing an empty document.
PDF page caps are strict by default. With Profile.AllowPartialPdfExtraction=true,
the native provider retains only the first Limits.MaxUnits real source pages, keeps the
full Metadata.NativePageCount, and emits page_limit_exceeded as an error issue / Partial.
This opt-in is for hosts preserving older bounded-PDF behavior; it never fabricates page
numbers or hides truncation. It does not relax input-byte/text/ZIP limits, and does not
silently truncate OCR or converted renditions. Native provider version 1.2 records this policy.
For legacy Knowledge callers, AddDocumentProcessingExtractor() or
new DocumentProcessingExtractor(processing) adapts to the existing IDocumentExtractor /
first-match ExtractorRegistry. Register it before older extractors to prefer it.
The legacy extracted-page slots for DOCX/slides/sheets are compatibility indexes, not source
PDF locators; use the new result for anchors. This opt-in bounded path rejects over-limit or
partial results because the old return type cannot express them. Existing PdfExtractor,
DocxExtractor, PlainTextExtractor, ImageExtractor, registration APIs and old defaults
remain available unchanged, including old truncation/image-placeholder behavior; new consumers
should use Documents rather than treating those old placeholders as recognized text.
Each built-in capability also exposes a non-sensitive ConfigurationFingerprint owned by
that provider. Hosts can combine the selected descriptors' IDs/versions/fingerprints with
the generic profile and limits for reuse/version keys; they must not duplicate per-provider
configuration switches in business code. The optional property remains null for custom
providers that do not declare one. This is declared configuration, not a live health/model
probe; execution records still carry actual returned model/runtime hashes. ONLYOFFICE
secrets and credential rotation are excluded from its descriptor fingerprint.
Optional local OCR and conversion
src\Documents.Services\Polly.Documents.Services.csproj / local package
PollySdk.Documents.Services contains HTTP adapters only. Neither Python, Paddle models nor
ONLYOFFICE binaries are NuGet dependencies. Register only the services a host deliberately
provisions; lack of either service cannot prevent native parsing from starting.
using Polly.Documents.Services;
services.AddNativeDocumentExtraction();
services.AddPaddleOcr(new PaddleOcrOptions
{
Endpoint = new Uri("http://127.0.0.1:8765/"),
});
services.AddSingleton<IConversionSourcePublisher, HostOwnedSourcePublisher>();
services.AddOnlyOfficeConversion(new OnlyOfficeOptions
{
Endpoint = new Uri("http://127.0.0.1:8080/"),
SourceOrigin = new Uri("https://controlled-source.example/"),
JwtSecret = configuration["DocumentServices:OnlyOfficeJwtSecret"],
ServerVersion = "9.1.0-168",
});
// Enable only the operations desired for this version; no automatic cloud fallback.
var profile = new DocumentProcessingProfile
{
Id = "native-ocr-preview-v1",
ExtractorId = "native",
RecognizerId = "paddleocr",
Recognition = RecognitionMode.MissingText,
ConverterId = "onlyoffice",
RenditionMediaType = DocumentMediaTypes.Pdf,
};
The combined example is configuration, not a promise that every operation supports every
MIME: Paddle handles PNG/JPEG/BMP/single-frame TIFF and PDF; ONLYOFFICE handles DOCX/PPTX/XLSX
to PDF. Choose the profile according to the declared capabilities (e.g. OCR for scanned PDF,
preview conversion for Office documents). The core fails unsupported requested operations.
Remote adapter endpoints require HTTPS and explicit AllowRemoteEndpoint=true; this is host
consent to that endpoint, not permission to send arbitrary private files. HTTP redirects and
proxy/environment credential forwarding are disabled. Endpoint configuration is not a health probe.
For a heterogeneous profile, leave ExtractorId unset to select only an actually compatible
native provider; explicitly naming native for an unsupported image MIME is still an error,
not an implicit fallback. Use recognition for raster/PDF inputs and conversion for Office
inputs according to the capability catalog, but never duplicate the SDK's stage execution.
The Paddle worker's declared Capability.Constraints hard ceilings are input 32 MiB,
1000 pages, 100000 blocks, 4000000 text characters, output 64 MiB, 16000000 raster pixels,
and a 600-second request deadline. PaddleOcrOptions.RequestTimeout defaults to 120 seconds
and must be positive and at most 600 seconds. Its actual stage deadline is the smaller of
that value and the overall DocumentLimits.Timeout; a separate linked token enforces it.
Host job/lease timeouts may be longer without sending an invalid 1800-second timeout to the
worker. Page/block/text upper bounds are intersected with the declared worker ceilings; an
input or output actually exceeding the effective limit fails explicitly. Render scale is
positive and at most 4. These effective options/profile limits are fingerprinted; missing
constraint fields mean undeclared, not unlimited.
External response-body I/O/transport failures are normalized to service_read_failed, including
failures after successful HTTP headers. A PDF header alone is not accepted as a valid rendition:
known PdfPig parse/stream-format errors become invalid_rendition. Both follow the explicit
strict/optional policy, record failures, and dispose source leases. Cancellation is not reclassified
as an external read error or silently turned into a partial result.
Paddle worker. The implementation in tools\documents\paddle-worker\worker.py invokes
the official PaddleOCR 3.3 Python pipeline
PaddleOCR(...).predict(ndarray). The validated profile is deliberately text OCR only:
PP-OCRv5 mobile detection/recognition; CPU, two threads, no orientation, unwarp, or line-rotation
models; no implicit model downloads. PP-StructureV3
is a separate, larger layout/table/formula pipeline. Its capabilities are not advertised by
this adapter and no mathematical meaning or handwriting accuracy is guaranteed.
Provision the versions in requirements.txt in a dedicated environment, and explicitly obtain
the two official inference model archives before starting the worker:
mobile detector
and mobile recognizer.
Each model directory must contain inference.json, inference.pdiparams, and inference.yml.
--model-root must contain directories named PP-OCRv5_mobile_det and PP-OCRv5_mobile_rec.
The worker checks package versions, fingerprints all three model files, and returns runtime/
model hashes with every result. models.lock.json pins official archive and extracted-file-set
SHA-256 values. Altered local model files fail model_mismatch. The actual Windows CPython 3.11
dependency cohort is frozen in requirements.windows-cp311.lock.txt; use this lock when reproducing
that environment, not a fresh unconstrained transitive resolution. It fails with model_missing or runtime_mismatch, rather
than trying a mirror or returning an empty document.
# Run only after installing the explicitly approved runtime/packages/models.
python tools\documents\paddle-worker\worker.py `
--model-root .artifacts\ocr-models --temp-root .artifacts\ocr-temp --port 8765
python tools\documents\paddle-worker\synthetic_samples.py `
--output .artifacts\synthetic --font C:\Windows\Fonts\msyh.ttc
$env:POLLY_DOCUMENT_PADDLE_ENDPOINT = "http://127.0.0.1:8765/"
$env:POLLY_DOCUMENT_SYNTHETIC_SAMPLES = (Resolve-Path .artifacts\synthetic).Path
dotnet test tests\Polly.Documents.Tests --filter FullyQualifiedName~RealPaddle
PDFium (pypdfium2), not PdfPig, renders PDF source pages in their actual order, using the
page crop and native rotation. Returned regions are in rendered-crop-points-top-left-after-source-rotation;
media/crop boxes retain PDF source coordinate values, and rotation is explicit. Raster polygons
refer to input pixels. No EXIF transpose or perspective unwarp quietly changes the coordinate basis.
The worker enforces input/page/pixel/block/text/output/log/time bounds and permits one inference
at a time. Each request owns a child process and a unique temporary directory. Deadline or client
disconnect terminates only that child PID; cancellation never publishes a completed result.
MissingText conditionally starts recognition and merges only no-text-layer units, but the
current Paddle worker still renders/OCRs the whole original PDF, not a selected page subset.
AllowPartialPdfExtraction truncates native output only; an over-limit original PDF still
fails OCR with the worker's explicit page-limit error. Only an explicitly allowed optional
failure policy can retain that native partial result, with failed—not completed—inference.
Python-level network calls are blocked during inference; production hosts should also sandbox
native runtime egress. /health verifies local versions/model files without inference.
ONLYOFFICE. This adapter uses the official Docs conversion API,
not the Document Builder commercial SDK and not an editor. It sends JSON to /converter,
requests asynchronous conversion, polls the same signed request, and accepts only same-origin
/cache/files/ PDF responses. It checks signature, parseability, positive page dimensions/count,
source/output hashes and size limits. ServerVersion is the deployment-pinned version,
not an unverified runtime version probe; pin the container digest operationally.
The host supplies IConversionSourcePublisher.PublishAsync(DocumentInput, CancellationToken)
returning an IConversionSourceLease with DownloadUri and DisposeAsync. Publish only that
snapshot under a short-lived unguessable capability at the configured SourceOrigin; revoke it
on disposal, including cancellation/failure. Do not accept a caller-controlled URL, publish a
shared directory, or let it expose arbitrary local files. Server-side conversion may continue
after client cancellation; the adapter stops polling/download/publication and revokes the source.
The host must isolate Docs egress to its own source endpoint, disable macros/external resources,
and restrict inbound access. Package checks refuse external relationships (including hyperlinks
on this conservative conversion path), active objects, macro parts and external fields/formulas
before publishing. They are defense in depth, not a substitute for service sandboxing.
No caller JavaScript/command, DOM script, edit callback or collaboration operation is exposed.
OpenXML native locators provide the required source structure access independently.
Real conversion tests use tools\documents\source_bridge.py inside an isolated test Docs
container only. It holds bounded synthetic byte buffers with 180-second expiry, authenticates
PUT/DELETE with an ephemeral POLLY_SOURCE_BRIDGE_TOKEN, and uses random unguessable GET keys.
Publish its container port 8899 only to a random host loopback port. Configure tests using
POLLY_DOCUMENT_ONLYOFFICE_ENDPOINT, POLLY_DOCUMENT_SOURCE_BRIDGE,
POLLY_DOCUMENT_SOURCE_TOKEN, and optional POLLY_DOCUMENT_ONLYOFFICE_JWT. Run
dotnet test tests\Polly.Documents.Tests --filter FullyQualifiedName~RealOnlyOffice.
The test creates DOCX/PPTX/XLSX locally, checks resulting PDF text/pages, then revokes sources.
Without explicitly configured services, external tests are inconclusive, never mock successes.
Deployment and licenses. Native OpenXML is MIT, PdfPig is Apache-2.0, and Microsoft DI is MIT. PaddleOCR/PaddleX/PaddlePaddle are Apache-2.0; obtain models from the official project and retain model provenance. Python is PSF licensed; Pillow uses HPND. pypdfium2 is Apache-2.0/BSD-3-Clause, with PDFium and its bundled third-party license notices. Audit the full resolved Python dependency set for the deployment platform. ONLYOFFICE Docs Community is AGPL-3.0 with additional legal/ branding provisions; assess obligations for your deployment/distribution. No commercial license purchase, user registration, terms acceptance, GPU or system-wide install is performed by Polly. Current Document Builder downloads describe a watermarked free mode/commercial licensing and are intentionally not this deployment path. Models/container images are never pulled on SDK startup.
Validated locally with synthetic inputs: Windows CPython 3.11.9 / PaddleOCR 3.3.0 CPU on
English/Chinese PNG, image-only PDF, and cropped/90-degree-rotated PDF; and the Linux
ONLYOFFICE Docs 9.1.0-168 package from the pinned 9.1.0 image digest in models.lock.json.
DOCX, complete PPTX and XLSX converted to parseable PDFs with expected text and unchanged input
hashes. The complete synthetic PPTX also passes OpenXML validation. A deliberately incomplete
PPTX fails real conversion and is retained as a negative/explicit-partial-policy case, not
"fixed" by ignoring converter errors. Native provider version 1.1 preserves repeated DOCX
image occurrences and explicit PPTX line breaks; version 1.0 results remain distinct.
These checks do not establish fidelity for all Office layouts/fonts, all PDFs, model accuracy
on arbitrary documents, GPU execution, or other operating systems.
Knowledge owns ingestion and retrieval over Polly's mapped document and segment collections. It does not require Polly Server, a gateway, a business application, or its build infrastructure. The optional native search index is a rebuildable projection, not a second document store. All mapped writes and projection writes use the same Storage connection and transaction.
SQLite composition
Reference src\Extensions\Knowledge.Sqlite\Polly.Extensions.Knowledge.Sqlite.csproj, or the
locally built package PollySdk.AgentFramework.Extensions.Knowledge.Sqlite. It brings in
PollySdk.AgentFramework.Extensions.Knowledge and PollySdk.Storage.Sqlite.
The core-only project remains src\Extensions\Knowledge\Polly.Extensions.Knowledge.csproj.
Every project targets .NET 10. Local package definitions do not imply publication.
For local NuGet consumption, build the entire required dependency cohort from this same checkout;
do not mix these adapters with older binaries carrying the same 0.1.3 version. The two adapter
packages alone are not a complete offline feed. Project references are the simplest local handoff.
using Microsoft.Extensions.DependencyInjection;
using Polly.AI.Embeddings;
using Polly.Extensions.Knowledge;
using Polly.Extensions.Knowledge.Sqlite;
using Polly.Storage.Sqlite;
static ServiceProvider OpenKnowledge(string databasePath, IEmbeddingService embedding)
{
var services = new ServiceCollection();
services.AddSqliteStorage(databasePath);
services.AddKnowledge(options =>
{
options.EmbeddingModel = "your-model-id";
options.EmbeddingDimensions = 1536; // Must match the selected model.
options.EmbeddingBatchSize = 32;
options.ChunkingStrategy = ChunkingStrategy.Markdown;
});
services.AddSqliteKnowledgeIndex();
services.AddSingleton<IEmbeddingService>(embedding);
return services.BuildServiceProvider();
}
Use the existing Polly AI provider/registry to obtain IEmbeddingService; do not duplicate its
transport, credentials or model-aware embedding cache. An application with a pre-ingested corpus
can omit IEmbeddingService when it only uses Keyword or FullText. For first-time ingestion
without a model, explicitly disable embeddings as described below; omission alone does not disable
the existing vector-enabled ingestion default.
var knowledge = provider.GetRequiredService<IKnowledgeService>();
await knowledge.SetupAsync(ct);
await knowledge.IngestAsync(new RawDocument
{
SourceId = "handbook:chapter-1",
Content = markdown,
Domains = new[] { "handbook" },
Metadata = new Dictionary<string, string> { ["edition"] = "2026" },
}, ct);
var hits = await knowledge.SearchAsync("retry policy", new SearchOptions
{
Strategy = SearchStrategy.FullText,
TopK = 10,
Filter = new SearchFilter
{
Domains = new[] { "handbook" },
MetadataFilters = new Dictionary<string, string> { ["edition"] = "2026" },
},
}, ct);
The facade signatures remain IngestAsync(RawDocument, CancellationToken),
RemoveAsync(string sourceId, CancellationToken) and
SearchAsync(string query, SearchOptions options, CancellationToken).
In the default Automatic mode, SetupAsync creates mapped resources and initializes the optional projection. A newly enabled
projection backfills existing stored segments transactionally, without re-embedding.
IKnowledgeIndexMaintenance.RebuildIndexAsync(CancellationToken) clears and rebuilds the
projection atomically. Failures and cancellation propagate; they do not return successful ingestion.
Direct writes to the underlying mapped collections are unsupported while a projection is enabled;
use the facade or rebuild before serving queries. All writers sharing a store must register the
same projection; a core-only writer cannot maintain an optional projection it does not know about.
Optional PostgreSQL composition
Reference src\Extensions\Knowledge.PostgreSql\Polly.Extensions.Knowledge.PostgreSql.csproj,
or PollySdk.AgentFramework.Extensions.Knowledge.PostgreSql. This adds the existing
PollySdk.Storage.PostgreSql provider, not another database connection owner or gateway.
using Polly.Extensions.Knowledge.PostgreSql;
using Polly.Storage.PostgreSql;
services.AddPostgreSqlStorage(connectionString);
services.AddKnowledge(options =>
{
options.EmbeddingModel = "your-model-id";
options.EmbeddingDimensions = 1536;
});
services.AddPostgreSqlKnowledgeIndex(options =>
{
options.Dimensions = 1536;
options.EnableHnsw = true;
});
services.AddSingleton<IEmbeddingService>(embedding);
Vector-enabled mode requires pgvector 0.7+ installed (and PostgreSQL supporting stored generated columns
and materialized CTEs). CreateVectorExtension is off by default; enable it only when the role
is explicitly authorized to install the extension. If pgvector is installed in a different schema
and its type/similarity operators are not visible on the configured search path, setup reports a
configuration error. The adapter never expands the path to public to make them visible. Provision
compatible extension visibility deliberately before enabling vectors. No default Azure credential or managed identity
is introduced by this adapter. Host-supplied NpgsqlDataSource/Storage configuration remains available.
One transactional initializer owns a UUID document/segment projection and vector(Dimensions)
schema. An existing projection with different dimensions fails setup rather than executing a conflicting
second schema script. Use an isolated database/schema or an explicit migration when dimensions change.
All indexing SQL is compiled into the Polly adapter assembly; no external SQL path is needed.
Native-backed facade operations hold the projection's transaction-scoped advisory lock through
document/segment/result materialization, coordinating PostgreSQL reads and writes across instances.
This prioritizes consistency; concurrent throughput and ANN performance have not been benchmarked.
By default PostgreSQL full text uses english tsvector, plainto_tsquery (AND semantics), and ts_rank_cd;
it is not strict BM25 and differs from SQLite's tokenization and ranking.
By default vector queries are exact. Filtered candidates are materialized before ranking and
TopK, avoiding HNSW post-filter underfill. EnableHnsw creates cosine, L2 and inner-product indexes;
creation failures propagate and abort initialization. PostgreSqlKnowledgeIndex.GetStatusAsync(ct)
reports actual projection existence, vector column type and count of valid HNSW indexes, not an
assumed ANN success. SearchOptions.UseApproximateSearch = true opts into approximate retrieval
only for an unfiltered, entirely public corpus; filtered/Memory queries remain exact. Approximate
retrieval is not a guarantee of exact top-K recall. HnswEfSearch is applied with transaction-local
configuration and cannot leak to pooled connections. Requesting approximation without configured
HNSW fails explicitly on the native approximate path.
First-time text-only ingestion on PostgreSQL
Both defaults remain vector-enabled for compatibility. Opt out independently at ingestion and at the PostgreSQL projection. This uses the same facade, processors, mapped storage, transactions, full-text index, backfill and rebuild; no direct table writes or placeholder vectors are needed.
using Microsoft.Extensions.DependencyInjection;
using Npgsql;
using Polly.Extensions.Knowledge;
using Polly.Extensions.Knowledge.PostgreSql;
using Polly.Storage.PostgreSql;
var connection = new NpgsqlConnectionStringBuilder(existingPostgreSqlConnectionString)
{
SearchPath = "quizmind_knowledge",
};
var services = new ServiceCollection(); // Dedicated Knowledge composition.
services.AddPostgreSqlStorage(connection.ConnectionString);
services.AddKnowledge(options =>
{
options.EnableEmbeddings = false;
options.DefaultStrategy = SearchStrategy.FullText;
options.ChunkingStrategy = ChunkingStrategy.Markdown;
});
services.AddPostgreSqlKnowledgeIndex(options =>
{
options.EnableVectorSearch = false;
});
using var provider = services.BuildServiceProvider();
var knowledge = provider.GetRequiredService<IKnowledgeService>();
await knowledge.SetupAsync(ct);
await knowledge.IngestAsync(new RawDocument
{
SourceId = "handbook:chapter-1",
Content = markdown,
Domains = new[] { "handbook" },
}, ct);
var results = await knowledge.SearchAsync("retry policy", ct: ct);
Do not register an embedding service for this configuration. No embedding model ID, API key,
dimensions, pgvector extension or HNSW index is required. The text-only DI path does not even resolve
an already-registered embedding factory. Segments store Embedding = null, not fake or zero vectors.
Keyword remains available explicitly. Vector/hybrid queries fail clearly in text-only mode.
EnableHnsw and CreateVectorExtension are invalid when EnableVectorSearch is false.
External requirements remain a reachable PostgreSQL instance (12+ for this backend's SQL), a
pre-created dedicated quizmind_knowledge schema, and a role with USAGE/CREATE and subsequent
read/write privileges in that schema. Schema creation/role grants are deployment actions, not
performed by these registration calls. A search path containing only that schema prevents silently
falling back to public if it is absent; setup fails instead.
The domain database/provider does not need to change. With a dedicated schema, Knowledge's
polly_resources, polly_entity_tables, mapped polly_* resources and
polly_knowledge_segments_v1/its indexes are in a different namespace from domain tables.
Do not rely on unverified domain-table names to assume same-schema compatibility.
If a host already registers a different Polly IStorage, keep Knowledge in a separate owned
composition as above; AddPostgreSqlStorage uses singleton registrations, not named stores.
An EF/Npgsql domain store can remain independently registered. Dispose the Knowledge composition
with the owning host, not after each request.
Text-only documents carry a distinct index profile; toggling embedding mode and re-ingesting rebuilds segments even for unchanged text. Vector-enabled fingerprints from earlier versions remain unchanged by adding this option. For a PostgreSQL projection initially created without a vector column, later enabling vector search requires an explicit schema migration or separate Knowledge schema and re-ingestion; setup does not silently alter dimensions or manufacture vectors. Do not mutate runtime options while operations are active; recompose the host when changing modes.
Operator initialization and strict PostgreSQL runtime
Initialization policy is independent of embedding mode:
| Knowledge mode | PostgreSqlStorage AutoCreate |
Setup and runtime preparation | Explicit rebuild |
|---|---|---|---|
Automatic (default) |
true |
May create/repair mapped resources and projection; retains legacy auto-setup behavior | Allowed |
RequireInitialized |
false |
Validate existing storage, catalog registrations, tables and required indexes; no DDL | Rejected |
Mismatched combinations fail configuration validation. The mode is captured when services are constructed; changing the options object cannot escalate a runtime instance into an operator. Use separate operator/runtime compositions with the same schema, mapping and ingestion profile.
The operator registers Automatic + AutoCreate=true using a role permitted to initialize this
dedicated schema, then explicitly calls IKnowledgeService.SetupAsync(ct). First setup backfills
a newly created projection. Repeated setup is idempotent but can execute DDL and acquire locks.
IKnowledgeIndexMaintenance.RebuildIndexAsync(ct) is a separate, explicit operator operation;
there is no reason to rebuild immediately after normal first setup.
The runtime uses a restricted role with schema USAGE and SELECT/INSERT/UPDATE/DELETE on the Knowledge tables (including the Storage catalog tables), without CREATE:
services.AddPostgreSqlStorage(new PostgreSqlStorageOptions
{
ConnectionString = runtimeConnectionString, // search_path contains only the dedicated schema.
AutoCreate = false,
});
services.AddKnowledge(options =>
{
options.InitializationMode = KnowledgeInitializationMode.RequireInitialized;
options.EnableEmbeddings = false;
options.DefaultStrategy = SearchStrategy.FullText;
});
services.AddPostgreSqlKnowledgeIndex(options => options.EnableVectorSearch = false);
SetupAsync now means validate-only in that composition. Ingest/search/remove/list operations and
their repeated transaction preparation validate rather than executing CREATE/ALTER/INDEX/extension
DDL. Missing required resources throw KnowledgeInitializationRequiredException, with a
MissingResources list. Permission, connection and incompatible configuration errors remain
errors rather than becoming successful empty results. Strict runtime still acquires transaction
coordination locks and executes application DML; it is not a read-only data service.
Read-only readiness and status
var readiness = await provider.GetRequiredService<IKnowledgeReadiness>()
.GetReadinessAsync(ct);
// readiness.IsReady and readiness.MissingResources; the interface also exposes InitializationMode.
This inspection does not resolve the facade/embedding service, call Storage EnsureSchema, register
resources or initialize a projection. It uses fixed catalog/SELECT probes through a trusted internal
connection path. Standalone inspection opens a read-only transaction; within an existing transaction
it borrows that connection without committing/closing it or running initialization. This also works
with a one-connection runtime pool. SQL callbacks are internal to the Storage/provider assemblies,
not a consumer-supplied SQL surface.
PostgreSqlKnowledgeIndex.GetStatusAsync(ct) uses the same genuinely read-only path. Its existing
ProjectionExists, VectorType and ValidHnswIndexes fields remain; Readiness reports the full
required-state check. A missing schema/table/index yields not-ready. Missing SELECT/USAGE privileges
or failed authentication/connection throws explicitly; inspection never catches these as
"not initialized." A status-only role needs USAGE and SELECT on the required tables, not DML or DDL,
and status stays read-only even if composed with Automatic + AutoCreate=true.
Readiness describes initialized storage/projection structure, not corpus/profile compatibility,
content consistency after unsupported direct writes, or application authorization.
Strict mode requires search_path to contain one explicit lowercase schema identifier. The schema
must already exist; missing schema is reported, never redirected to public. Vector type/operator
visibility errors remain explicit and never expand that path.
SQLite currently does not support RequireInitialized or the read-only readiness capability.
Its normal DI path rejects strict mode before resolving Storage, opening a connection or creating a
file/directory. Do not treat SQLite's AutoCreate=false alone as this Knowledge lifecycle contract.
Its existing automatic mode remains supported.
The restricted-role PostgreSQL lifecycle tests cover cold startup and reopened instances, full text-only CRUD, read-only status, missing base/projection/index resources, operator repair/backfill, permission/authentication errors and no public fallback. They install a test-only DDL rejection event trigger for runtime/status roles: unchanged table counts alone are not the no-DDL proof.
Retrieval semantics and compatibility
| Strategy | Lexical score | Embedding required for query |
|---|---|---|
Keyword |
Existing term frequency / token count | No |
Vector |
Cosine, L2 similarity, or inner product | Yes |
Hybrid |
Existing keyword score + weighted RRF | Yes |
FullText |
Native backend ranking; SQLite uses positive -bm25(FTS5) |
No |
HybridFullText |
Native ranking + vector ranks, fused by weighted RRF | Yes |
Keyword and default Hybrid retain their scoring choices; full text is opt-in. Without an index,
requesting full text fails explicitly rather than falling back to embeddings. SQLite FTS5 uses
porter unicode61 and an escaped OR of Unicode word tokens, not raw user-supplied FTS syntax.
The legacy tokenizer remains unchanged.
SearchOptions exposes MatchAlgorithm, CandidateMultiplier, RrfK, VectorWeight,
KeywordWeight, MinVectorScore, MinKeywordScore and final MinScore.
Thresholds apply to their own score scales before fusion; final MinScore applies after fusion.
The default candidate multiplier is 3, preserving Polly's previous pool size.
Setting it to 4 reproduces the source implementation's pool choice.
Scores from different strategies/backends are not interchangeable.
Domains and SegmentTypes use exact OR matching, SourceId exact matching, and
MetadataFilters exact AND matching. Missing metadata does not satisfy any requested key.
Document eligibility and segment type are enforced before native TopK. Current document
metadata enriches results, so metadata-only updates cannot leave copied segment metadata stale.
Metadata sets and grouped PostgreSQL full-text paging
IKnowledgeService.SearchAsync remains the top-K API with its existing response and scoring.
SearchFilter.MetadataInFilters is an additive
IReadOnlyDictionary<string, IReadOnlyList<string>>: values in one field are ORed; fields and
MetadataFilters (Eq) are ANDed. Matching is ordinal and literal, including %, _, *, case
and whitespace. Missing keys never match. An empty In set matches nothing; null sets/values
are invalid. Eligibility is applied before ranking/TopK and before grouped counts.
Existing core/SQLite and PG search strategies support this filter without changing Keyword scores.
Do not express allowed (sourceVersion, processingRevision) pairs using two independent In
lists: that admits their Cartesian product. Instead store an application-issued, opaque projection
ID and filter that single field. For large corpora an opaque processing-projection-generation ID
shared by that generation's documents can avoid enumerating every application segment; an
application segment's own stable ID and locator remain separate metadata. The application must
derive the current allowed set from its authoritative lifecycle and IAM state on each request.
Domains and these filters are not ACLs, and cursor contents must never supply that allowed set.
AddKnowledge also registers the separate optional capability:
public interface IKnowledgePagedSearch
{
Task<KnowledgeGroupPage> SearchGroupsAsync(
string query, GroupedSearchOptions options, CancellationToken ct = default);
Task<KnowledgeSearchGroup> SearchGroupHitsAsync(
string query, GroupedSearchOptions options, string groupKey,
string cursor = null, CancellationToken ct = default);
}
The PostgreSQL implementation is FullText only, using the configured lexical profile, not
vector/hybrid candidate approximation. Core-only and SQLite compositions throw
NotSupportedException for this capability; they do not emulate it with client-side top-K loops.
It performs server-side filtered aggregation and ordered bounded SQL pages; it does not enumerate
all documents/results into application memory. Exact counts require scanning matching candidates.
Each selected group has its own bounded hit query; large offsets and many groups increase database
work. Throughput/large-corpus latency are not benchmarked. This is not an ANN or count-estimation API.
// These are trusted server-side values after current IAM and processing-state checks.
IReadOnlyList<string> allowedProjectionIds = currentAllowedProjectionIds;
var options = new GroupedSearchOptions
{
GroupByMetadataKey = "qm.source.version",
AccessScopeStamp = currentPrincipalScopeAndAuthorizationRevision,
GroupPageSize = 1,
HitsPerGroup = 5,
Filter = new SearchFilter
{
MetadataFilters = new Dictionary<string, string> { ["qm.space"] = authorizedSpaceId },
MetadataInFilters = new Dictionary<string, IReadOnlyList<string>>
{
["qm.id"] = allowedProjectionIds,
},
},
};
var paged = provider.GetRequiredService<IKnowledgePagedSearch>();
var first = await paged.SearchGroupsAsync(query, options, ct);
// Revalidate IAM and derive the same current filter/stamp before each continuation.
var next = first.HasMore
? await paged.SearchGroupsAsync(query, options with { Cursor = first.NextCursor }, ct)
: null;
var group = first.Groups.FirstOrDefault();
var moreHits = group?.HasMore == true
? await paged.SearchGroupHitsAsync(query, options with { Cursor = null },
group.GroupKey, group.NextCursor, ct)
: null;
The group-list cursor belongs in GroupedSearchOptions.Cursor; a group's hit cursor belongs
in the SearchGroupHitsAsync argument, with options.Cursor = null. Both calls require the
original query, grouping/page parameters and the current trusted scope stamp and filter.
Starting SearchGroupHitsAsync with a null cursor starts at that group's first chunk.
A valid group with no matching hits returns zero counts/empty hits. A missing/blank grouping
key argument is invalid. Any eligible matching document missing that metadata key, or carrying
a null/empty/whitespace-only value or a group value over 1024 UTF-8 bytes, fails the query
explicitly; such documents are not silently dropped or combined into an unnamed group.
Metadata group values are application-defined: Polly never infers a version from a title.
| Result field | Meaning, after all eligibility and lexical matching |
|---|---|
KnowledgeGroupPage.TotalGroups |
Distinct source-group metadata values |
TotalChunkHits / group ChunkHitCount |
Matching native Polly index chunks |
TotalDocuments / group DocumentCount |
Distinct matching Polly document IDs (application projection documents) |
CountAccuracy |
KnowledgeCountAccuracy.Exact, not a top-K estimate |
Groups / group Hits |
This bounded page only |
NextCursor / HasMore |
Opaque continuation, or null/false when exhausted |
KnowledgePagedHit.SegmentId |
Native index chunk ID; changes when that source is re-chunked |
KnowledgePagedHit.Result |
Existing KnowledgeResult with authoritative current document metadata and SourceId |
With one application segment per Polly document, DocumentCount counts matching application
segment projections, while ChunkHitCount may be larger. It is not the total parsed segments
in the source, nor a count of every stored chunk. Hit continuation pages native chunks and may
therefore contain the same projection document more than once. The consumer can label/deduplicate
using its metadata but must not relabel chunk counts as application segment totals.
Different version metadata values are separate groups even when their material ID is the same.
Group ordering is best chunk score descending, then group key using PostgreSQL COLLATE "C";
within a group it is score descending, then PostgreSQL UUID order. No display heading or synthetic
page/paragraph locator is used. Supply the actual Source version, application segment ID and
locator (for example qm.source.locator) at ingestion.
Limits are explicit and never truncate: group size and hits/group are 1..100; query is at most 4096 UTF-16 code units; scope stamp is nonblank and at most 1024; metadata keys are at most 256; filter values are at most 1024; Eq + In have at most 32 field entries; In has at most 4096 total values (including duplicates before normalization); each domains/types list has at most 4096 entries; combined filter keys/values have at most 262144 UTF-16 code units. Null and NUL-containing filter values are rejected; empty string values remain literal values. These limits apply to grouped queries and to legacy searches opting into In. Legacy Eq-only search retains its prior validation behavior. Oversize input throws an argument exception, not a successful partial page.
Cursor consistency, scope and operator upgrade
The rebuildable polly_knowledge_documents_v2 projection stores document eligibility data.
polly_knowledge_state_v2 stores a singleton schema/profile marker, durable UUID generation and a
random 256-bit AES-GCM key. polly_knowledge_segments_v1 retains its English column and gains a
lexical_mixed projection/index. These are not authoritative source/version archives.
All facade ingestion, metadata-only updates, removal and rebuild update the generation in the same Storage transaction as authoritative and projection writes. A rollback restores the old generation and old content. Ingestion that merely refreshes properties also invalidates cursors. Invalidation is conservatively store-wide, including writes outside the current filter and reserved Memory writes. No process-local counter is used. All readers and writers use the existing transaction-scoped PG advisory lock: generation, filtered counts, ordering/page and bounded authoritative result materialization see the same logical database view even across connections. The lock ends with the request transaction; there is no transaction held across browser pages. Direct collection/SQL writes or writers not maintaining the registered projection remain unsupported.
Cursors are encrypted/authenticated with random 96-bit nonces and a versioned purpose context.
Group-list versus group-hit purpose and group identity cannot be interchanged. The binding includes
NFC/whitespace-normalized query, normalized ordinal filter sets, grouping/sorting/page sizes,
lexical profile, durable generation and caller's current AccessScopeStamp. Tokens contain no
usable authorization snapshot; the library does not authenticate the caller. They are limited to
4096 characters and expire 30 minutes after issuance; deployments should synchronize clocks.
New continuation tokens get a new expiry. Malformed, tampered, expired, wrong-purpose or stale
tokens throw KnowledgeCursorException; callers should restart the query, not treat this as zero
matches. The library never returns keys in status/results or logs them. Protect the database and
backups as server secrets and do not enable sensitive SQL-parameter logging; neither key nor SQL
appears in the cursor's plaintext.
For an existing pre-v2 projection or a changed lexical profile, ordinary setup/query reports
KnowledgeInitializationRequiredException and readiness is false. Deploy in this order:
- Stop/drain old writers; all processes sharing this store must use the new projection contract.
- In a separate Automatic + AutoCreate=true operator composition, select the intended profile
and explicitly call
IKnowledgeIndexMaintenance.RebuildIndexAsync(ct). This transaction adds missing v2 structures, sets the profile and rebuilds from the authoritative mapped collections. It never drops those collections or re-embeds sources. Failure rolls back schema/data/profile/ generation changes. The original English/vector column and configured dimensions remain intact. - Grant runtime/status roles access to the new tables as appropriate, then verify
IKnowledgeReadinessand startRequireInitialized + AutoCreate=falseprocesses with that profile.
For a new store, the existing operator SetupAsync creates/backfills everything; legacy Automatic
auto-setup behavior is retained. Strict runtime/status never create keys, tables, columns or indexes,
nor repair state. Strict rebuild is rejected. Rebuild always changes the generation, invalidating
all outstanding cursors, but normally retains the encryption key. Reconstructing missing state
creates a new key as well. No historical snapshot/replay across revisions is promised.
Optional Chinese / mixed-language lexical profile
Set the same immutable backend profile for operator and runtime compositions:
services.AddPostgreSqlKnowledgeIndex(options =>
{
options.EnableVectorSearch = false;
options.LexicalProfile = PostgreSqlLexicalProfile.HanRunePhraseV1;
});
This local deterministic algorithm uses Unicode Han scalar unigrams, contiguous non-Han
letters/digits/combining marks as word tokens, and literal arithmetic/other-symbol tokens (including
ASCII hyphen and Unicode minus). NFC and invariant lowercasing are applied only to the search
projection. Tokens are safely hex-encoded before PostgreSQL simple parsing. There is no model,
dictionary download, external service or domain-specific keyword list. Original documents,
chunk content and locator metadata are unchanged. It uses PostgreSQL ts_rank_cd, not fabricated
fallback scores, BM25, vector semantics or comprehensive Chinese NLP.
Within each whitespace-separated query fragment, all emitted tokens must occur contiguously
in the indexed token stream (phrase semantics); fragments are ANDed, in any order and at any
distance within the same chunk. Query syntax such as quotes/& does not become a raw tsquery.
Ordinary punctuation is a delimiter and is not itself indexed; whitespace separates indexed words.
Consequently punctuation/spacing can be equivalent at the lexical-token level, not as source
characters or mathematics. Word tokens are whole words; Han phrases can match within longer
sentences. Phrases do not span Polly chunks; choose suitable chunk size/overlap at ingestion.
| Indexed text | Query | Result |
|---|---|---|
本章介绍条件概率及其应用 |
条件概率 or the full sentence |
Match |
研究报告讨论风险管理和预算约束 |
风险管理 |
Match |
| Same text | 预算约束 风险管理 |
Match: AND of two phrases, not adjacency |
| Same text | 风险预算 or 无关词 |
No match |
AI 条件概率 P(A) = −3.5 |
条件概率 AI |
Match: phrase plus Latin word |
| Same text | AIT or -3.5 |
No match: whole word / different minus character |
café |
decomposed cafe + combining acute |
Match after NFC |
running |
run |
No match in this profile; default English profile still stems |
Decimal punctuation is token-delimiting (−3.5 has minus, 3, 5); this is not numeric
normalization or mathematical equivalence. NFC does not convert traditional/simplified Chinese,
translate languages, or prove semantic equivalence of formulas. Han extensions in U+3400..4DBF,
U+4E00..9FFF, U+F900..FAFF and U+20000..323AF are treated as scalar tokens, including supplementary
characters; other Unicode letters/symbols follow the word/symbol rules above.
PostgreSQL position/lexeme limits are enforced before indexing: at most 16000 tokens per chunk, 256 occurrences of one encoded lexeme, 512 UTF-8 bytes per unencoded word, and 524288 encoded ASCII characters. Queries allow at most 512 tokens. Exceeding a bound throws and rolls back ingestion/rebuild; the tokenizer never truncates and then claims complete retrieval. Use smaller chunks for repetitive or very long text. Scalar unigrams increase index size and positional work relative to word segmentation; this tradeoff avoids dictionary coverage gaps but is not benchmarked.
Changing profile requires the explicit rebuild above, which recalculates lexical projections
without altering mapped source text or obtaining embeddings. The mixed profile identifier also
participates in the ingestion fingerprint; re-ingest to update stored document fingerprints and
any changed chunking profile (mandatory before vector/hybrid reuse). Rebuild alone does not
rewrite those document fingerprints. Default English fingerprints, English FTS and legacy
Keyword scoring remain unchanged. This profile is PostgreSQL-only; SQLite retains its prior
porter unicode61 behavior and has no claimed equivalent Chinese/profile or paging capability.
Ingestion and index compatibility
Same-source replacements now preserve DocumentId and CreatedAt. Metadata-only changes
update title, domains, keywords, metadata and UpdatedAt without embedding. Replaced segments
receive new IDs. This intentionally corrects the previous regenerated document identity.
Content hashes are not sufficient to reuse embeddings. Stored documents additionally record a
fingerprint of model ID, requested dimensions, chunk strategy/size/overlap, processor type and
KnowledgeOptions.IndexVersion. Change IndexVersion when a provider binding or custom processor
changes semantics without changing those values. Do not mutate options while operations are running.
Re-ingest all relevant sources after changing the profile: vector/hybrid queries reject incompatible
documents, including legacy documents without a fingerprint, rather than mixing embeddings.
Lexical retrieval remains available during migration. Rebuilding the search projection does not
re-embed or upgrade a document profile. A separate database is another explicit namespace option.
This is compatibility checking, not a content/model revision-history system.
Document writes use Storage optimistic versions before touching child segments. A concurrent conflict surfaces as a Storage exception for the caller to retry; it cannot silently orphan the other writer's segments or commit an out-of-sync projection.
Batch size must be positive. Batches respect cancellable EmbeddingBatchDelay; responses must have
exact cardinality, non-empty finite vectors and consistent requested/inferred dimensions across batches.
Chunking validates overlap is smaller than size and bounds every emitted chunk, including long lines
and Markdown sections. Markdown uses Markdig AST headings, not heading-like text inside code blocks.
WholeDocument uses document-typed chunks but still obeys the size cap.
Boundaries and source mapping
This local adaptation was authorized by the author of the source Knowledge module. Behavioral
comparison used source snapshot 1d1a3218b6750a668cbe454d2847116f3cefddf2, module
Platform\Framework\Knowledge from BizChatLTDash. Its surrounding build metadata declares
Microsoft Corporation / All rights reserved and private artifacts; this work does not override
those statements or grant permission to publish that source. The implementation here extends Polly
and uses public algorithms/libraries; it does not copy the source module, its assembly attributes,
SQL scripts, AI project, internal build imports or credentials.
| Source responsibility | Polly implementation |
|---|---|
| Pipeline/query/facade and models | Existing KnowledgeService, Knowledge.cs, extended in place |
| AI embedding interface/cache | Existing Polly.AI.Embeddings and Polly.AI.Runtime |
| SQLite vector store + FTS | Existing mapped Storage + Knowledge.Sqlite FTS5 projection |
| PostgreSQL vector/FTS store | Existing PostgreSqlStorage + Knowledge.PostgreSql projection, UUID/vector schema, exact metrics and optional HNSW |
| Text/Markdig processors | TextChunker with explicit bounded/AST strategies |
| Strategy resolver/vector/BM25/hybrid | Explicit SearchStrategy, shared vector math and weighted RRF |
| Schema/setup scripts | Transactional, provider-owned initialization, no external SQL file path |
| Source metadata | Authoritative document metadata; no stale metadata copies in projection |
The PostgreSQL adapter is optional. Its text-only lifecycle and missing-schema isolation have been run against an isolated PostgreSQL 17 container without pgvector. Vector/HNSW integration still requires explicit pgvector-enabled test configuration; lexical execution does not prove ANN behavior or performance.
Neither source nor this library supplies a real cross-encoder reranker, IAM/ACL enforcement,
tenant authorization, arbitrary facets/count trees or full historical version governance.
Polly's optional grouped full-text paging above supplies scoped invalidatable cursors and exact
matching counts, not an authorization engine or a historical snapshot service.
SourceId is globally unique within a store. Domains and metadata are retrieval filters, not
authorization; an application must derive allowed scope from trusted identity context and isolate
stores where required. The memory: namespace remains inaccessible through public Knowledge,
including native full text; trusted Memory uses IReservedKnowledgeService.
Existing PDF/DOCX/image/text extractors and Knowledge-backed Memory are retained. Markdig is BSD-2-Clause; the native backend dependencies retain their own package licenses. No package push, remote publication or source-repository merge is performed by this migration.
Verification
Run core/SQLite and Memory tests without external services:
dotnet test tests\Polly.Knowledge.Tests\Polly.Knowledge.Tests.csproj
dotnet test tests\Polly.Memory.Tests\Polly.Memory.Tests.csproj
dotnet test tests\Polly.Storage.Tests\Polly.Storage.Tests.csproj --filter "FullyQualifiedName~NativeTransactionTests|FullyQualifiedName~SqliteConcurrencyTests|FullyQualifiedName~RelationalInitializationTests"
The PostgreSQL suite is tests\Polly.Knowledge.PostgreSql.Tests\Polly.Knowledge.PostgreSql.Tests.csproj.
Its live tests are explicitly skipped/inconclusive unless
POLLY_KNOWLEDGE_POSTGRES_TEST_CONNECTION names a dedicated database ending in _tests.
The vector contract needs pgvector installed; the separate text-only contract does not. It creates and removes only a unique per-run schema, never drops a fixed
database or uses application data. Invalid configuration or database failures are test failures, not
skips. Do not interpret a skipped live test as PostgreSQL verification. To run just the lexical
and schema-isolation paths against ordinary PostgreSQL without pgvector:
dotnet test tests\Polly.Knowledge.PostgreSql.Tests\Polly.Knowledge.PostgreSql.Tests.csproj --filter "FullyQualifiedName!~PostgreSqlNativeProjectionContractInUniqueSchema"
| Product | Versions Compatible and additional computed target framework versions. |
|---|---|
| .NET | net10.0 is compatible. net10.0-android was computed. net10.0-browser was computed. net10.0-ios was computed. net10.0-maccatalyst was computed. net10.0-macos was computed. net10.0-tvos was computed. net10.0-windows was computed. |
-
net10.0
- DocumentFormat.OpenXml (>= 3.5.1)
- Microsoft.Extensions.DependencyInjection.Abstractions (>= 10.0.14)
- PdfPig (>= 0.1.16)
NuGet packages (2)
Showing the top 2 NuGet packages that depend on PollySdk.Documents:
| Package | Downloads |
|---|---|
|
PollySdk.AgentFramework.Extensions.Knowledge
Knowledge ingestion, extraction, retrieval, embeddings, and Knowledge-backed Memory. |
|
|
PollySdk.Documents.Services
Optional, explicitly configured PaddleOCR worker and ONLYOFFICE Docs conversion adapters. |
GitHub repositories
This package is not used by any popular GitHub repositories.