Repository navigation
Roadmap update: 4.0 and beyond #52
jordansrowles
announced in
The Cooking Pot
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Roadmap update: 4.0 and beyond
The version numbering in this roadmap needs a reset.
While finishing the work originally planned for 3.2.0, we ended up requiring breaking changes significant enough that the next release can no longer reasonably ship as another 3.x minor. That work will therefore ship as 4.0.0.
The important bit is that this does not mean the architectural work originally assigned to 4.x has suddenly been pulled forward. We've effectively consumed the 4.0 major version while completing the current feature line.
New version progression
So the existing numbered sections below should be treated as describing scope and ordering, rather than the final release numbers, until I rewrite the roadmap properly.
Nothing is being dropped because of the renumbering. The distinction between the two phases also remains:
Completion work first: analysis, suggestions/parsers, vector/range work, alternate indexing/query models, telemetry/diagnostics, languages and text similarity.
Architectural work afterwards: optimiser contracts, query planning/execution, lifecycle/integrity, and retrieval/ranking/relevance unification.
The main change is simply that breaking API/index-format evolution has forced the major-version counter forward earlier than originally expected.
Apparently "make a roadmap to stop feature creep" did not protect me from version creep.
Revised release roadmap
Rather than mechanically renumbering every old minor release, the remaining roadmap is being consolidated around the actual compatibility boundaries.
The old 3.3 through 3.8 milestones now form 5.0.0.
The old 4.0 through 4.3 milestones now form 6.0.0.
This makes the intent considerably clearer:
4.0.0 - Geo Shapes
Originally planned as 3.2.0.
The majority of this work is now complete. Existing
GeoPointField,GeoBoundingBoxQueryandGeoDistanceQueryprovided the original point-search foundation; 4.0 completes the broader spatial model.Geometry model
LatLonShapeXYPointGeoGeometry/XYGeometryabstractionsSpatial querying
Shape DocValues
Search and aggregation
Geometry infrastructure
The boundary remains intentional. LeanCorpus is gaining a proper spatial-search foundation, not becoming a GIS library.
Geohash/geotile grids, spatial prefix trees, projection conversion, GeoJSON ingestion and similar GIS-specific functionality remain outside the scope unless there is a concrete search-engine requirement for them.
5.0.0 - Feature Completion
5.0 consolidates everything previously planned for 3.3 through 3.8.
The purpose of this release is to finish incomplete feature families, fill important gaps around existing primitives and improve the tooling required to understand what LeanCorpus is actually doing before the optimiser work begins.
Analysis
Filters and analysers
LimitTokenCountFilterinto an analyser wrapperTrimFilterStemmedOverrideConditionalTokenFilterKeywordRepeatCodepointCountClassicFilteralready exists)CollationKeyanalyserPatternCaptureGroupTokenFilterFingerprintFilterTypeTokenFilterCommonGramsQueryFilterNormalisation
Analysis diagnostics
Add an analysis inspector/debugger capable of exposing:
It should be usable from tests, documentation and contributor tooling rather than being tied to a particular UI.
Token-graph correctness remains an implementation-quality requirement across positions, position lengths, offsets, flattening and phrase/query-time handling.
Suggestions & Parsers
Suggestions
ContextSuggesterFuzzySuggesterwhere it provides a useful surface beyondDidYouMeanSuggesterQuery parsing
StandardQueryParserSurroundQueryParserFormalise the parser pipeline as:
syntax -> AST -> processors -> QueryProcessors should support:
The parser pipeline must also expose proper extensibility hooks rather than requiring consumers to fork or replace the parser.
Vector & Range Search
Multi-vector documents and byte-vector execution remain the parts most likely to require careful design work rather than simply extending the existing vector machinery.
Alternative Indexing, Querying & Storage Models
MemoryIndexMonitorQueryThis remains the most miscellaneous part of the completion release, but the common theme is supporting alternative ways of indexing, querying, storing or consuming data without changing the fundamental LeanCorpus execution architecture.
Telemetry & Diagnostics
General telemetry
ActivitySourcecoverageQuery execution diagnostics
Expose enough structured information to explain what actually happened during execution:
For HNSW:
For hybrid/retrieval pipelines:
The aim of 5.0 telemetry is still to record what actually happened.
6.0 can then add what was estimated, what was planned, what alternatives were rejected, and why.
Telemetry should also provide enough evidence to revisit filesystem and persistence behaviour using real measurements rather than blind micro-optimisation.
Languages & Text Similarity
6.0.0 - Architecture & Execution
6.0 consolidates the architecture work previously spread across 4.0 through 4.3.
This is the point where LeanCorpus starts reasoning about search execution as a complete system rather than simply exposing increasingly capable individual primitives.
API & Optimiser Contracts
API cleanup
Optimiser-facing execution contracts
Revisit query, scorer and collector abstractions so implementations can expose capabilities required by the optimiser.
These include:
Custom queries and extensions should be able to participate properly in optimisation rather than becoming opaque operators that silently force slow paths.
Query Planning & Execution
Introduce a first-class planning model covering both logical intent and physical execution.
Planning
Adaptive execution
Allow the planner to select physical strategies such as:
Planner diagnostics
There should be one execution-plan model, not several overlapping concepts for logical plans, physical plans, explanations and diagnostics.
Logical intent, selected physical operators, estimates and observed execution should instead be different layers or views of the same query-execution model.
Index Lifecycle & Integrity
These should form a coherent lifecycle model rather than a collection of unrelated maintenance utilities.
Retrieval, Ranking & Relevance
6.0 also consolidates retrieval, fusion, ranking and evaluation around one architecture.
Retrieval
topKRetrieval diagnostics should expose:
Ranking
Examples include:
exact symbol > prefix > fuzzy > semanticor:
exact phrase > proximity > loose term matchThese must remain general ranking primitives rather than becoming code-search-specific policy.
Relevance evaluation
Build a judgement-driven experiment harness over the ranking metrics already available.
Support:
Capture systems measurements alongside relevance:
And support:
The Linux kernel code-search E2E can continue serving as an early combined relevance and systems workload.
The goal is to consolidate retrieval and ranking architecture rather than creating separate query, retriever, fusion and ranking-plan hierarchies that all model different views of the same execution process.
ADR016 also remains the baseline: Matryoshka, RaBitQ, PQ or compressed late interaction do not return to the roadmap without a materially different design and new benchmark evidence.
Research & validation
Alongside the normal product roadmap, I have started treating some of LeanCorpus's lower-level behaviour as a systems research problem rather than just an implementation problem.
Research work around LeanCorpus
There is now a separate research programme looking at parts of LeanCorpus's storage, filesystem and persistence architecture in considerably more depth than would make sense inside the normal project roadmap.
The aim is not to turn every optimisation into a paper.
It is to take areas where LeanCorpus has exposed an interesting systems-level question and subject them to a much higher standard of evidence than:
That means separating exploratory measurements from actual evidence, defining what is being measured precisely, reproducing results under controlled conditions, testing across relevant environments, and being explicit about what a result does and does not establish.
The work is being organised separately from normal feature development and includes:
This is also one reason telemetry and diagnostics have become more prominent in the roadmap.
Instrumentation should not exist solely for production dashboards. It should also make it possible to answer questions about what the engine actually did, rather than trying to infer behaviour from a single end-to-end timer.
Some findings from this work may feed back into later LeanCorpus releases, particularly around:
Others may simply establish that an apparently promising optimisation is workload-specific, platform-specific, or not worth pursuing.
Both results are useful.
The detailed hypotheses, experimental designs and unpublished results intentionally live outside this roadmap. I'll publish the interesting parts when they're ready rather than pre-announcing the entire research programme here.
This also changes how I want to approach some of the more speculative items below.
For substantial systems changes, particularly ones that claim performance or architectural advantages, the preferred progression is increasingly:
observation -> investigation -> controlled evidence -> implementationrather than:
interesting idea -> implementation -> benchmark afterwardsThat won't apply to every feature, but it should apply wherever the complexity or long-term architectural cost is high enough to justify it.
Beyond 6.0 - Future Core directions
Anything beyond the architecture work should be treated differently from the releases above.
These are directions, not promises for 7.0, 8.0 or any other specific release. Some may become full release themes, some may arrive incrementally, and some may disappear entirely once they are investigated properly.
The roadmap should deliberately become less version-specific at this point. There is little value in pretending we know today exactly which compatibility boundary a feature several releases away will eventually require.
Classification
LeanCorpus currently provides the primitives needed to retrieve and rank documents, but not a first-class way to classify them.
A future Core direction is therefore index-backed document and text classification, broadly in the same problem space as Lucene's classification facilities while remaining appropriate to LeanCorpus's own architecture.
Possible areas include:
Vector search also gives this a natural future extension point:
This should remain an index/search feature, not an attempt to turn LeanCorpus into a general-purpose machine-learning framework.
The exact public APIs, classifier set and implementation model are deliberately undecided at this stage. Those decisions belong much closer to implementation, once the preceding retrieval and execution architecture has stabilised.
Classification should also be considered alongside the existing
Document classificationitem in the 5.0 completion work. If 5.0 only introduces the necessary primitive or basic classifier, the broader classification surface described here can continue evolving later without forcing all of the classification work into that release.Learned & weighted models
There is a potentially useful middle ground between completely hand-written heuristics and embedding a general-purpose machine-learning runtime into LeanCorpus.
LeanCorpus already contains several places where decisions are based on costs, scores, probabilities, candidate ranking or path selection. Some of those may eventually benefit from parameters learned offline rather than permanently encoding every decision as a fixed heuristic.
The important constraint is that this should not turn LeanCorpus into an ML framework.
The direction being investigated is much smaller:
Possible applications exist across several areas of the engine, including:
This does not imply that all of those will be implemented.
The useful architectural question is whether LeanCorpus can consume small learned artefacts in the same spirit that it already consumes indexes, dictionaries and other immutable data structures: validate them once, load them cheaply and execute them predictably.
Large neural inference remains outside Core.
Applications that require transformers, tensor runtimes, GPUs or substantial model-lifecycle infrastructure should generally perform that inference externally and integrate through bounded retrieval/reranking surfaces.
The attraction of the smaller model class is that it could provide some of the advantages of learned behaviour without giving up the characteristics LeanCorpus is deliberately designed around:
This is still exploratory work.
Individual mechanisms should only move into the numbered roadmap when they solve a concrete search problem and demonstrate a meaningful improvement over the existing deterministic implementation.
Further execution evolution
The 6.0 optimiser should be the foundation rather than the final state of query execution.
Future work can investigate increasingly data-oriented and adaptive execution techniques where they show measurable benefit, including:
This may eventually include carefully bounded learned decision-making where it can outperform static thresholds without making execution unpredictable.
Any such mechanism would need a conservative deterministic fallback and should be treated as an optimisation rather than a correctness dependency.
These techniques should not be adopted merely because equivalent ideas exist in database engines, ML systems or other search engines.
They need evidence against representative LeanCorpus workloads.
Search intelligence
Once the execution and relevance layers are stable, there is room for higher-level search behaviour without baking application-specific policy into Core.
Potential directions include:
Some of this may remain heuristic. Some may eventually use the compact learned-model approach described above.
Either way, LeanCorpus should preserve deterministic and inspectable execution wherever practical.
The engine should be able to explain why a strategy was selected or why a document's score changed rather than hiding important retrieval decisions behind an opaque model.
Corpus-aware codecs and storage
CodecKit gives LeanCorpus the mechanism to support multiple persistent representations.
A later step could investigate using corpus characteristics to help choose or configure those representations.
Potential areas include:
Automatic selection would need to remain inspectable and overrideable.
The goal would be to use CodecKit's flexibility intelligently, not create formats whose behaviour cannot be reasoned about or reproduced.
Investigations, not release commitments
Current investigations
These remain outside the numbered roadmap until evidence is strong enough to justify promotion:
The distinction between future directions and investigations is intentional.
Future directions describe capabilities that make sense for LeanCorpus if the prerequisite architecture and use cases exist.
Investigations are specific mechanisms that still have to prove they deserve to exist at all.
Neither implies a release commitment.
The bar for moving an investigation into a release remains a representative spike showing measurable benefit over the simpler LeanCorpus baseline.
That means evidence in the dimensions that actually matter:
Every learned feature also needs a non-learned baseline.
A more complicated model is not an improvement simply because it was trained.
Interesting is not enough to get something onto the numbered roadmap.
All reactions