AI concepts
Embeddings (vector embeddings)
A representation of text, an image, or another artefact as a numerical vector in a high-dimensional space, where geometric closeness corresponds to semantic closeness. Embeddings are the foundation of modern semantic search, recommendation systems, and RAG architecture. The dimension of a typical embedding: 768 to 3,072 numbers.
Primary source: Google word2vec (Mikolov 2013), OpenAI text-embedding-3 documentation 2024, Pinecone Vector Database Whitepaper 2024
The concept comes from the foundational paper "Efficient Estimation of Word Representations in Vector Space" by Tomas Mikolov and a Google team in 2013 (word2vec). The idea: every word can be represented as a vector of 300 numbers, in which similar words have geometrically similar vectors. The famous discovery: "King" minus "Man" plus "Woman" gives a vector close to "Queen". Over a decade the idea grew into representations of sentences, documents, images, and sound.
The technical mechanism
An embedding model (separate from the LLM, though often from the same family) converts the input into a numerical vector of fixed length. OpenAI text-embedding-3-large documentation from 2024 describes a typical text embedding as a vector of 3,072 floating-point numbers. Each number represents a value on some abstract semantic axis whose meaning cannot be directly interpreted.
Vectors are compared using cosine or Euclidean distance. Two documents on a similar topic have vectors with a small angle between them, two documents on different topics have divergent vectors.
Enterprise applications
The Pinecone Vector Database Whitepaper from 2024 lists five dominant use cases. First, semantic search over a knowledge base (for example, I search for "how to solve problem X" and the system finds documents that match conceptually, not just by keyword). Second, RAG for an LLM (find the context, hand it to the model for generation). Third, a recommendation system (find similar products, clients, content). Fourth, deduplication and clustering (group records with similar meaning). Fifth, anomaly detection (flag a record that is unusual within the corpus).
Choosing an embedding model
Three main categories. Proprietary API: OpenAI text-embedding-3, Cohere embed-v3, Voyage AI. Higher quality, cost of 0.02 to 0.13 dollars per million tokens. Open-weights: bge-large, e5-large, multilingual-e5. Can be hosted locally, lower operational cost, lower benchmark quality. Specialised: legal-bert, finbert, biobert. Better than general-purpose models for narrow domains.
For a Polish firm with a sizeable knowledge base in Polish, multilingual-e5-large is often a good compromise. It has reasonable Polish quality and can be hosted locally (16 GB GPU).
Operational cost
A vector database (Pinecone, Weaviate, Qdrant, ChromaDB) plus an embedding API are two cost items. For a base of 100 thousand documents: a one-off 50 to 200 zloty for embedding (proprietary API), 5 to 15 dollars a month for a hosted vector DB. Enterprise scale (10 million documents): 5 to 20 thousand zloty one-off, 200 to 2,000 dollars a month.
Embeddings are the basis of RAG architecture, which we implement within AI implementation into processes.