Over the last few years, vector search has become a critical piece of AI infrastructure, powering use cases from RAG and semantic search to agentic memory and context layers. With the rise of agentic systems, companies are striving to provide as much context as possible to agents, which drives vector database indexes to scale from an initial million or tens of millions of vectors to hundreds of millions or even billions. By 2026, as agentic workloads become mainstream, the cost of storing these indexes and associated data in RAM can reach thousands of dollars per month, making HNSW a scalability bottleneck.
In this article, I will dive into the details of what makes semantic search fast and efficient: approximate nearest neighbor (ANN) algorithms, the different options available, and their trade-offs.
Deep Dive into Vector Databases
Vector databases consist of three main components:
- Embeddings – the numeric representation of the corpus.
- Search algorithm and index structure – the algorithm defines search quality and speed.
- Storage – how data is stored (in memory, on disk, payload together with embeddings, etc.). Whether in RAM or on disk, it determines costs and latency at scale.
Embeddings are already well defined and discussed in many articles. This one focuses on search algorithms, specifically ANN ones. There are generally two approaches for search execution:
- Exact search – delivers the best retrieval metrics but does not scale well in terms of latency.
- Approximate nearest neighbor (ANN) – trades retrieval quality for latency and scalability.
Exact search loops through all entries in the index and calculates the distance between the query and existing data. There are no losses from approximation or generalization, with the trade-off being latency and scalability. It is a great approach for very small indexes or experimentation, but not suitable for production scale.
Interesting fact: Many modern vector databases allow you to bypass index building for small collections, falling back to kNN search because the overhead of building an index is not worth it for a few thousand vectors.
The second option is approximate nearest neighbor algorithms, a group of algorithms designed to improve scalability by avoiding visiting all entries in the index. The idea is to provide shortcuts to speed up search and ingestion. Implementations vary, though many modern ANN algorithms (e.g., HNSW and DiskANN) rely on a graph structure for low query latency.
Approximate Nearest Neighbor Algorithms
It is important to note that although all ANN algorithms follow a similar concept—providing a short path to the end result—there are different implementations with unique trade-offs, making it crucial to select one that fits your exact use case.
In this article, I focus on two groups of ANN algorithms:
- RAM-based – optimized for storing all or a large chunk of data in memory, providing extremely low latency with costs as a trade-off. A standard example is HNSW (Hierarchical Navigable Small World).
- On-disk – minimize RAM usage and rely heavily on disk to load required data. Examples include DiskANN or SPANN.
It is possible to store the underlying data structures of both algorithm groups on disk or in memory (at least partially), but they are optimized for a specific storage type and deliver the best results when used as designed.
In-Memory ANN
HNSW (Hierarchical Navigable Small World)
[Image placeholder – original source: https...]