Testing

This file describes the testing methodology used in SVS and includes documentation on how update reference results for integration tests.

The test suite defined in the tests directory are broken into two categories: unit tests that test individual components and integration tests which exercise more cross-unit functionality. The latter includes index searching and index construction. Now, we cannot necessarily make our integration tests exhaustive for a number of reasons. First, integration tests should have high coverage of the many different combinations of indexes, distance functions, compile time optimizations, etc., each of which takes time. Second, these integration tests must all reference the same binary dataset to minimize the repository size. This dataset may not be appropriate for all distance functions.

Hence, integration tests compare against reference behavior (i.e., recall for a given configuration generated at some point in the past) and fail if the observed behavior deviates too far. These reference recalls are generated by executables in the benchmarking framework.

See the section on building the library for CMake options related to building integration tests into the test executable (SVS_FORCE_INTEGRATION_TESTS) and building reference result test generators into the benchmarking executable (SVS_BUILD_BENCHMARK_TEST_GENERRATORS).

Testing-Related Manifest

A summary of the testing related files in the SVS repository and their role is given below.

data/
|-- test_dataset/ : Directory containing the reference dataset used for testing as well
|   |               as reference recall values.
|   |
|   |-- reference/ : Directory containing reference configuration/recall values for
|   |   |            index implementations.
|   |   |
|   |   +-- vamana_reference.toml: Reference configurations and recall for the Vamana
|   |   |                          index. Auto-generated by the benchmarking framework.
|   |   |
|   |   +-- inverted_reference.toml: Reference configurations and recall for the
|   |                                Inverted index. Auto-generated by the benchmarking
|   |                                framework.
|   |
|   |-- data_f32.fvecs : The dataset upon which reference results are generated.
|   |-- data_f32.svs : A copy of `data_f32.fvecs` but encoded in the SVS format.
|   |-- graph_f32.svs : A reference Vamana index constructed for the test dataset.
|   |-- groundtruth_cosine.ivecs : The groundtruth for the queries over the test dataset
|   |                              for the CosineSimilarity distance measure.
|   |
|   |-- groundtruth_euclidean.ivecs : The groundtruth for the queries for the L2
|   |                                 similarity measure.
|   |
|   |-- groundtruth_mip.ivecs : The groundtruth for the queries for the L2
|   |                           similarity measure.
|   |
|   |-- known_f32.fvecs : A small "fvecs" file with known contents. The expected
|   |                     contents are returned by ``test_dataset::reference_file_contents()``
|   |
|   |-- known_f32.svs : A copy of `data_f32.fvecs` but encoded in the SVS format.
|   |-- queries_f32.svs : The queries to use for the test dataset.
|   |-- leanvec_data_matrix.fvecs : LeanVec OOD matrix to transform data. Shape(128, 64)
|   |-- leanvec_query_matrix.fvecs : LeanVec OOD matrix to transform queries. Shape(128, 64)
|   +-- vamana_config.toml : The serialized ``VamanaIndexParameters`` of the reference
|                            Vamana graph.
|
+-- tools/
    |
    +-- benchmark_inputs/
        |
        +-- vamana/test-generator.toml : Input file used to generate the reference
        |   results (data/test_dataset/reference/vamana_reference.toml) from the
        |   benchmarking framework.
        |
        +-- vamana/test-generator.toml : Input file used to generate the reference
            results (data/test_dataset/reference/inverted_reference.toml) from the
            benchmarking framework.

Generating Test Inputs

Complete scripts for compiling, executing, and updating reference inputs are collected here.

Vamana

SVS_NUM_THREADS=10
CC=gcc-11
C++=g++-11

mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release \
    -DSVS_BUILD_BENCHMARK_TEST_GENERATORS=YES
make -j
# Generate the expected results.
./benchmark/svs_benchmark vamana_test_generator \
    ../tools/benchmark_inputs/vamana-test-generator.toml \
    ./vamana_reference.toml \
    ${SVS_NUM_THREADS} \
    ../data/test_dataset
# After checking that the results look good, update the reference file.
cp ./vamana_reference.toml ../data/test_dataset/vamana_reference.toml