VDMS Configuration File

VDMS uses a configuration file (written in JSON) that can be specified when starting the server by using the -cfg flag: 

./vdms -cfg config-vdms.json

If no configuration file is specified, VDMS will try to open the default file (config-vdms.json), and will fail to initiate if the file is not found.

Parameters

All the parameters in the configuration file are optional, as VDMS has default values for all of them.

Param Explanation Default
autodelete_interval_s Time interval (in seconds) to delete specifc entries (Should be greater than 0) -1
autoreplicate_interval Time interval to backup the DB folder (Should be greater than 0) -1
aws_log_level This parameter is optional and it is used when "storage_type" parameter is set to "aws" value, it is used to control the level of verbosity of AWS logging system. The acccepted values are "off", "fatal", "error", "warn", "info", "debug", "trace" off
backup_flag Boolean whether to use the auto-replication thread false
backup_path Path to store the backup DB db_root_path
blobs_path Path to folder where blobs will be stored blobs (db/blobs)
bucket_name Bucket name for AWS storage vdms_bucket
db_root_path Path to the root folder where all filed/objects will be stored db
descriptors_path Path to folder where descriptors will be stored descriptors (db/descriptors)
endpoint_override Server address (including scheme and port number) to override the S3 storage address server. This parameter is valid when the "storage_type" parameter is set to "aws" value and "use_endpoint" parameter is set to true. http://127.0.0.1:9000 when "use_endpoint" parameter is set to true and the "storage_type" parameter is set to "aws"
expiration_time Time interval (in seconds) to automatically delete entries
flinng_cells_per_row For the FLINNG indexing for descriptors, controls the number of bits in the distance sensitive LSH vector for each row 1000
flinng_hashes_per_table For the FLINNG indexing for descriptors, controls the number of hash functions to be used per table for group testing 12
flinng_num_hash_tables For the FLINNG indexing for descriptors, controls the number of hash tables (permutations) to be used for the dataset for group testing 10
flinng_num_rows For the FLINNG indexing for descriptors, controls the number of distance sensitive LSH vectors 3
hnsw_efConstruction For the HNSW indexing for descriptors, controls the breadth of the search during the index construction phase 96
hnsw_efsearch For the HNSW indexing for descriptors, controls the breadth of the search during the search query 64
hnsw_M For the HNSW indexing for descriptors, controls the maximum number of neighbors that each descriptor can have at each layer 48
images_path Path to folder where images (all formats) will be stored images (db/images)
ivf_nlist For the IVF FLAT indexing for descriptors, specify the number of partitions to create using the k-means algorithm 16
k8s_container Boolean whether to use Kubernetes orchestration false
max_simultaneous_clients Number of max simultaneous connections open 500
pmgd_num_allocators Number of allocators when creating a new PMGD graph (this will only be used when creating a new graph, and ignored if the graph already exist) 1
pmgd_path Path to folder where PMGD graph will be stored db
port TCP port for incoming connections 55555
proxy_host Address of the proxy (optional). Example: "a.proxy.from.intel.com"
proxy_port Port number of the proxy. This parameter is needed when "proxy_host" parameter is set
proxy_scheme Scheme used by the proxy, accepted values are "http" and "https". This parameter is needed when "proxy_host" and "proxy_port" parameters are set
query_handler Specifies the query handler to use. Accepted values: pmgd (PMGD), or neo4j (Neo4j) pmgd
neo4j_conn_pool_sz Sets the pool size of neo4j client connections. This parameter is only used when "query_handler" is set to neo4j 32
replication_time -1
storage_type Database storage type. Accepted values: local (local storage), or aws (AWS S3) local
tmp_path Path to the temporary directory (optional) /tmp/tmp
unit Unit of the autoreplicate_interval variable. Accepted values: h (hour), m (minute), or s (seconds) s
use_endpoint Boolean whether to use an AWS storage mocking server (MinIO). This parameter is valid when the "storage_type" parameter is set to "aws" value false

Config File Example

// VDMS Config File
// This is the run-time config file
// Sets database paths and other parameters
{
    "port": 55555,
    "cert_file": "cert.pem",
    "key_file": "key.pem",
    "ca_file": "ca.pem",
    "autoreplicate_interval":-1, // it should be > 0
    "unit":"s",
    "max_simultaneous_clients": 100,
    // "backup_path":"backups_test", // set this if you want different path to store the back up file
    "db_root_path": "db",
    "backup_flag" : "false",
    "storage_type": "local", //local, aws, etc
    "bucket_name": "vdms_bucket",
    "more-info": "github.com/IntelLabs/vdms",
    "use_endpoint": false, // storage_type is set to local
    "endpoint_override": "http://127.0.0.1:9000",// Format "scheme://ip:port"
    "k8s_container": false,
    "proxy_host": "a.proxy.from.intel.com",
    "proxy_port": 912,
    "proxy_scheme": "http", // [http|https] valid values,
    "aws_log_level": "debug", // [off|fatal|error|warn|info|debug|trace]
    "tmp_path": "/tmp/tmp"
}

Default Directories Structure

By default, VDMS will create a directory structure as follows:

db
├── blobs
├── descriptors
├── graph
│   ├── allocator.jdb
│   ├── edges.jdb
│   ├── graph.jdb
│   ├── indexmanager.jdb
│   ├── journal.jdb
│   ├── nodes.jdb
│   ├── stringtable.jdb
│   └── transaction.jdb
└── images
    ├── jpg
    ├── png
    └── tdb