Skip to content

Realtime Query Command

Sergiu Ciumac edited this page Oct 15, 2026 · 6 revisions

Real-time query command

Oftentimes the source of the query comes from real-time data: broadcast, stream, microphone, a continuous stream of files, etc. This sets additional challenges in correct striding and real-time matches management that continue across multiple discrete query chunks.

To handle these uses cases, a special query command has been developed: RealtimeQueryCommand. It accepts an IAsyncEnumerable that continuously feeds the algorithm with data to fingerprint and query.

RealtimeQueryCommand should be viewed as a generic consumer of realtime files, samples, or hashes, which queries the underlying IModelService. The producer is abstracted by IAsyncEnumerable interface.

Query from a continuous stream of files

Below is a short example of how to create real-time query command from a continuous list of files generated by a given producer:

/// <summary>
///  Instance of model service with stored fingerprints to query.
/// </summary>
private readonly IModelService modelService = new InMemoryModelService();

/// <summary>
///  Media service to use for audio/video decoding.
/// </summary>
/// <remarks>
///  Install SoundFingerprinting.Emy package to get access to FFmpegAudioService.
/// </remarks>
private readonly IMediaService mediaService = new FFmpegAudioService();

public void StartQueryingFromFiles(IAsyncEnumerable<string> files, CancellationToken token)
{
    _ = QueryCommandBuilder.Instance
        .BuildRealtimeQueryCommand()
        .From(files, MediaType.Audio)
        .WithRealtimeQueryConfig(config =>
        {
            // provide a success callback that will be invoked for matches that pass result entry filter
            config.SuccessCallback = result =>
            {
                foreach (var entry in result.ResultEntries)
                {
                    Console.WriteLine($"Successfully matched {entry.TrackId}");
                }
            };

            // configure result entry filter
            config.ResultEntryFilter = new TrackCoverageLengthEntryFilter(5d, waitTillCompletion: true);
            return config;
        })
        .UsingServices(modelService, mediaService)
        .Query(token);
} 

This will start a query that will continue until the cancellation token is invoked. Similarly, you can provide the list of already downsampled AudioSamples, by using the appropriate overload. Make sure the audio samples adhere to the format required by the algorithm: mono audio, sampled at 5512 Hz. Other overloads accept IAsyncEnumerable<StreamingFile> (a file path with its capture time), IAsyncEnumerable<AVTrack> (decoded audio and video) and IAsyncEnumerable<AVHashes> (fingerprints generated elsewhere, for example on an edge device that only ships hashes).

Notice how we pass a SuccessCallback to capture successful matches and ResultEntryFilter to filter false positives.

Querying from the broadcasted stream

Realtime streams are typical radio broadcasts available over the web. Here is an example of CNN radio that you can listen to online: https://tunein.streamguys1.com/cnn-new

Using a real-time stream as your query source gives the opportunity to monitor the stream 24/7. Below is an example of how you can generate a real-time query command referencing CNN stream.

/// <summary>
///  Instance of model service with stored fingerprints.
/// </summary>
private readonly IModelService modelService = new InMemoryModelService();

/// <summary>
///  Media service to use for audio/video decoding.
/// </summary>
/// <remarks>
///  Install SoundFingerprinting.Emy package to get access to FFmpegAudioService.
/// </remarks>
private readonly IMediaService mediaService = new FFmpegAudioService();

public void StartQueryingRealtimeSource(CancellationToken cancellationToken)
{
    string url = "https://tunein.streamguys1.com/cnn-new";

    _ = QueryCommandBuilder.Instance
        .BuildRealtimeQueryCommand()
        .From(url, chunkLength: 60, MediaType.Audio)
        .WithRealtimeQueryConfig(config =>
        {
            config.ResultEntryFilter = new TrackCoverageLengthEntryFilter(5d, waitTillCompletion: true);
            config.SuccessCallback = result => Console.WriteLine($"Found {result.ResultEntries.Count()} matches");
            config.ErrorCallback = (exception, _) => Console.WriteLine($"An exception occured while querying the data source: {exception.Message}");
            config.RestoredAfterErrorCallback = () => Console.WriteLine("Query connection re-established");
            config.DidNotPassFilterCallback = result => Console.WriteLine($"Found {result.ResultEntries.Count()} matches, but they did not pass result entry filter.");
            return config;
        })
        .InterceptHashes(hashes =>
        {
            Console.WriteLine($"Querying the source with hashes {hashes} captured at {hashes.RelativeTo:O}");
            return hashes;
        })
        .UsingServices(modelService, mediaService)
        .Query(cancellationToken);
}

Things to understand:

  • The data source with tracks is provided by IModelService. Available sources listed here.
  • Queries will be generated every 60 seconds, after getting captured from the source stream.
  • The query thread will continue until the cancellation token is not canceled.
  • Matches will be relative to the DateTime.UtcNow, to be able to correctly identify the match time relative to the universal time.

How the stream is read

From(url, chunkLength, mediaType) hands the URL to the media service passed in UsingServices. FFmpegAudioService.ReadAVTrackFromRealtimeSource opens the URL with FFmpeg, so any container and codec FFmpeg can demux works (HLS, Icecast/SHOUTcast MP3 or AAC, RTMP, MPEG-TS over HTTP or UDP, RTSP). The decoded audio is downsampled to 5512 Hz mono, and every chunkLength seconds one AudioSamples chunk is yielded, stamped with RelativeTo, the UTC time at which the chunk was captured. Each chunk is fingerprinted and queried on arrival, and consecutive chunks that match the same track are stitched into one result entry, so a match is not cut at chunk boundaries.

Choosing chunkLength:

  • It is the detection latency: a match is at the earliest reported one chunk after it starts.
  • It bounds the stride of the query loop. A 30 second ad in a 60 second chunk is identified from one query; in 5 second chunks it is accumulated over six.
  • It does not limit the length of a match. A 28 minute program is one result entry whatever the chunk length.
  • Values between 5 and 60 seconds work well; the examples in this page use 60.

What happens when the stream or the storage fails:

  • A read error on the URL, or a failed query against IModelService, invokes ErrorCallback with the exception and the hashes that could not be queried. The command does not stop.
  • The hashes are kept in OfflineStorage (default EmptyOfflineStorage, which keeps nothing; implement IOfflineStorage to buffer them on disk) and are queried again, in order, once the connection is restored. RestoredAfterErrorCallback fires at that point.
  • Retries follow ErrorBackoffPolicy (default RandomExponentialBackoffPolicy).
  • Between chunks the command waits according to DelayStrategy; the default RandomDelayStrategy(1, 5) spreads queries from many monitors over time.

Recommended settings for broadcast monitoring:

  • ResultEntryFilter = new TrackCoverageLengthEntryFilter(seconds, waitTillCompletion: true) or new TrackRelativeCoverageEntryFilter(0.4, waitTillCompletion: true), so one airing is reported once, when it ends. Combine both with ChainedRealtimeEntryFilter when you need a relative bar for short tracks and an absolute one for long tracks.
  • OngoingEntryStalenessCeiling = 120 when the storage holds long tracks (programs, episodes). Without it a few seconds that brush against a 28 minute track keep that entry open for the remaining 28 minutes.
  • SeamJumpPolicy = SeamJumpPolicy.NewOccurrenceAbove(120) when stored tracks can repeat their own audio, for example programs that contain their advertising spots. See Realtime Seam Jump Policy.
  • Keep IncludeQueryHashesInResponse = false unless you need the query fingerprints in AVQueryResult.QueryHashes; they are large.

Query from a microphone

Below is a short example of a program that queries the underlying IModelService using samples captured over the microphone. To run this example, you will need NAudio package that provides means for recording from available recording devices.

class Program
{
    static BlockingCollection<AudioSamples> realtimeSource;
    static WaveInEvent waveSource;
    static int sampleRate = 5512;


    static async Task Main(string[] args)
    {
        _ = Task.Factory.StartNew(RecordMicNAudio);

        var tokenSource = new CancellationTokenSource();
        realtimeSource = new BlockingCollection<AudioSamples>();

        IModelService modelService = GetModelService(); // initialize your data source

        Console.WriteLine("Start Matching [...]");
        _ = GetBestMatchForStream(realtimeSource, modelService, tokenSource.Token);

        Console.WriteLine("Press any key to cancel.");
        Console.ReadKey();
        tokenSource.Cancel();
    }

    /**
      * Since NAudio microphone recording is based on event-driven architecture we will push
      * recorded samples into a blocking collection, that RealtimeQueryCommand will consume
      * Typical producer-consumer pattern.
      */
    static void RecordMicNAudio()
    {
        Console.WriteLine($"Available devices {WaveIn.DeviceCount}. Will use device 0 for recording.");
        for (int device = 0; device < WaveIn.DeviceCount; ++device)
        {
            var capabilities = WaveIn.GetCapabilities(device);
            Console.WriteLine($"Device {device} Name {capabilities.ProductName}, Channels {capabilities.Channels}");
        }

        waveSource = new WaveInEvent();
        waveSource.DeviceNumber = 0;
        waveSource.WaveFormat = new NAudio.Wave.WaveFormat(rate: sampleRate, bits: 16, channels: 1);
        waveSource.DataAvailable += (_, e) =>
        {
            // using short because 16 bits per sample is used as input wave format
            short[] samples = new short[e.BytesRecorded / 2];
            Buffer.BlockCopy(e.Buffer, 0, samples, 0, e.BytesRecorded);
            // converting to [-1, +1] range
            float[] floats = Array.ConvertAll(samples, (sample => (float)sample / short.MaxValue));
            realtimeSource.Add(new AudioSamples(floats, string.Empty, sampleRate));
        };
        waveSource.RecordingStopped += (_, _) => Console.WriteLine("Recording stopped.");
        waveSource.BufferMilliseconds = 1000;
        waveSource.StartRecording();
    }

    /**
     * Wrapping blocking collection to an adapter that implements IAsyncEnumerable interface.
     * BlockingRealtimeCollection adapter is provided within SoundFingerprinting package.
     */
    public static async Task<double> GetBestMatchForStream(BlockingCollection<AudioSamples> audioSamples, IModelService modelService, CancellationToken token)
    {
        double seconds = await QueryCommandBuilder.Instance
            .BuildRealtimeQueryCommand()
            .From(new BlockingRealtimeCollection<AudioSamples>(audioSamples))
            .WithRealtimeQueryConfig(config =>
            {
                // match only those entries got at least 5 seconds of query match
                config.ResultEntryFilter = new TrackCoverageLengthEntryFilter(5d, waitTillCompletion: true);

                // provide a success callback that will be invoked for matches that pass the result entry filter
                config.SuccessCallback = result =>
                {
                    foreach (var entry in result.ResultEntries)
                    {
                        Console.WriteLine($"Successfully matched {entry.TrackId}");
                    }
                };

                config.DidNotPassFilterCallback = (queryResult) =>
                {
                    foreach (var result in queryResult.ResultEntries)
                    {
                        Console.WriteLine($"Did not pass filter {result.TrackId}");
                    }
                };

                return config;
            })
            .UsingServices(modelService)
            .Query(token);

        Console.WriteLine($"Realtime query stopped. Issued {seconds} seconds of query.");
        return seconds;
    }
    
    private static IModelService GetModelService()
    {
        return new InMemoryModelService();
    }
}

Since samples are captured in an event-driven way, an adapter class BlockingRealtimeCollection is provided to facilitate their consumption via IAsyncEnumerable.

Realtime configuration options

RealtimeQueryCommand can be configured similar to QueryCommand, as it supersedes the number of available options for configuration. Below are listed additional options that you may want to fine-tune for your application.

  • SuccessCallback - invoked with an AVQueryResult for the entries that pass ResultEntryFilter.
  • ResultEntryFilter - decides which result entries reach SuccessCallback, and when. Every filter takes a waitTillCompletion flag: true holds the entry until it can no longer continue in the next chunk, so a track that spans several chunks is reported once with its full coverage; false reports it as soon as the bar is met.
    • TrackCoverageLengthEntryFilter(secondsThreshold, waitTillCompletion) - passes entries whose TrackCoverageWithPermittedGapsLength is above the threshold in seconds.
    • TrackRelativeCoverageEntryFilter(coverage, waitTillCompletion) - passes entries whose TrackRelativeCoverage is at or above the threshold. 0.4 means that tracks matched over less than 40% of their length are disregarded.
    • ChainedRealtimeEntryFilter(filters) - passes an entry when any filter in the chain passes it (an OR). Typical: relative coverage 0.4 OR 10 seconds, so short and long tracks each get a sensible bar.
    • CompletedRealtimeMatchResultEntryFilter - passes every entry, but only once it can no longer continue. Use it when you want everything, without partial matches.
    • PassThroughRealtimeResultEntryFilter - passes every entry immediately.
    • NoPassRealtimeResultEntryFilter - blocks everything; useful together with OngoingCallback when only the live view matters.
  • DidNotPassFilterCallback - invoked for the entries that ResultEntryFilter rejected, with the same AVQueryResult shape.
  • OngoingResultEntryFilter and OngoingCallback - the "what is playing right now" view. Entries that are still accumulating are passed through OngoingResultEntryFilter after every chunk, and OngoingCallback(streamId, entries) receives the ones that pass. Use a filter with waitTillCompletion: false here, for example new ChainedRealtimeEntryFilter([new TrackRelativeCoverageEntryFilter(0.2, false), new TrackCoverageLengthEntryFilter(5, false)]). Defaults: PassThroughRealtimeResultEntryFilter and a callback that does nothing.
  • ErrorCallback - invoked with the exception and the AVHashes that could not be queried. The command keeps running; cancel the token inside the callback to stop.
  • RestoredAfterErrorCallback - invoked when the first query after an error succeeds.
  • OfflineStorage - IOfflineStorage that keeps the hashes generated while queries fail; they are queried again once the connection is restored. Default EmptyOfflineStorage keeps nothing.
  • ErrorBackoffPolicy - delay between retries after an error. Default RandomExponentialBackoffPolicy.
  • DelayStrategy - delay between consecutive queries. Default RandomDelayStrategy(1, 5) seconds.
  • OngoingEntryStalenessCeiling - the longest non-matching stretch, in seconds, that an accumulating entry may bridge before it completes. Default unbounded (the entry waits for the remaining track length). Set it when the storage holds long tracks.
  • SeamJumpPolicy - what to do when the track position jumps ahead of the stream between two consecutive chunks: Stitch (default), SourceSkip for a source that seeks, or NewOccurrenceAbove(seconds) for a continuous source matched against tracks that repeat their own audio. See Realtime Seam Jump Policy.
  • AutomaticSkipDetection - obsolete shorthand for SeamJumpPolicy: true selects SourceSkip, false selects Stitch.
  • IncludeQueryHashesInResponse - whether AVQueryResult.QueryHashes carries the fingerprints of the chunks that produced the result. Default true; set false for long-running monitors to save memory.
  • QueryConfiguration - the AVQueryConfiguration used for every chunk (stride, threshold votes, permitted gap, meta field filters); see Algorithm configuration.

There are more properties marked with Experimental flag. Those are not listed here and are subject to change in the future.

Clone this wiki locally