oded valin: Speed up PostgreSQL pgvector queries with indexes

er sideUsing AI, it’s possible to find similar text information, photos or products in a database. As the number of searches increases, performance can be a problem, though.In this article, I’ll show you how indexes can help. We’re going to use PostgreSQL’s pgvector extension. It enables you to store AI embeddings, which are representations of information, and enables you to perform similarity searches on them. Pgvector has a couple of features that make it particularly popular. It supports hybrid searches, mixing standard and vector queries. For example, someone shopping online might want to search for shoes that look like a photo of their existing pair (a vector search), and that cost less than $100 (a standard database query). Pgvector also enables retrieval-augmented generation (RAG), where information from the database gives additional context to generative AI, improving the quality of the output and reducing hallucinations. This tutorial explains: The basics of vector similarity How pgvector can be used for vector similarity searches How indexes of various types can help to speed up searches The possible tradeoffs in output quality when using indexes See our previous blog for a refresher on what embeddings are and how pgvector can be used for face recognition. To get insights about your database performance and improvement suggestions, check out EverSQL by Aiven Understanding vector queries In the AI world, embeddings are a representation of a piece of information (such as text, image, sound, or video). They are expressed as an array of numbers. AI models such as OpenAI’s text-embedding-ada-002can generate embeddings from text (see the image below). These embeddings are always the same length, no matter how long the source text is. In the case of ada-002, the embeddings are 1536 elements long. The embeddings can be compared to find text with a similar meaning. But how does this embedding comparison work? We can picture the embeddings as p[...]