Skip to main content

Embedder

An Embedder converts text into fixed-length numerical vectors, called embeddings, that represent its meaning. Texts with similar meanings tend to produce vectors that are close together, enabling semantic search and document comparison.

LogicalDOC provides two implementations:

  • Paragraph Embedder: a trainable model based on Doc2Vec. This is the model previously called simply Embedder.
  • Mini Embedder: a pre-trained model based on MiniLM that does not require training inside LogicalDOC.

The resulting embeddings are stored in a Vector Store. A document can be divided into text chunks, with a separate vector generated for each chunk.

Paragraph Embedder

The Paragraph Embedder uses the Doc2Vec algorithm, through DL4J’s ParagraphVectors implementation, to learn vocabulary and contextual relationships from a supplied collection of text.

During training, the model learns which words and paragraphs occur in similar contexts. Once trained, it can convert document chunks and search queries into vectors.

Training Data

A Sampler supplies the text used to train the model. You can prepare training data in two ways:

  • Paragraph Sampler: reads a prepared text file and creates one sample for each paragraph. Paragraphs must be separated by blank lines.
  • Content Sampler: collects the indexed text of documents in a selected repository folder, optionally including its subfolders. Each document becomes one training sample.

For example, a file used by the Paragraph Sampler could contain:

Invoices record the products or services supplied to a customer.
They usually include payment terms and the total amount due.

Contracts define the responsibilities of the parties involved.
They may also specify deadlines and conditions for termination.

The quality of the embeddings depends on the training corpus. Use enough representative text to cover the languages, subjects and terminology of your documents.

Words absent from the trained vocabulary cannot contribute to an embedding. If none of the input words are recognized, the result is a zero vector with no useful semantic information.

Properties

The Properties tab contains the settings for Doc2Vec training, inference and text chunking.

  • Seed: value used to initialize random number generation.
  • Workers: number of threads used for training.
  • Window size: number of nearby words considered by the Doc2Vec algorithm when learning context.
  • Vector Size: number of elements in each vector. The default for the Paragraph Embedder is 300. The Mini Embedder uses a fixed size of 384.
  • Min. word freq: minimum number of occurrences required for a word to enter the training vocabulary. Less frequent words are discarded.
  • Max chunks: maximum number of text chunks processed for each document.
  • Chunk size: target number of tokens in each chunk.
  • Min. chunk size: minimum number of tokens and characters required for a chunk to be processed.
  • Alpha: initial learning rate, which controls the size of model updates. The default is 0.025.
  • Min Alpha: minimum learning rate. During Doc2Vec inference, the learning rate decreases linearly toward this value over the inference epochs. The default is 0.0001.

The Doc2Vec settings apply to the Paragraph Embedder. The Mini Embedder does not require local training.

Mini Embedder

The Mini Embedder uses a pre-trained MiniLM-L6-v2 transformer to convert text into 384-dimensional vectors.

The model already contains language representations learned from an external training corpus. It does not build its vocabulary from your repository and does not use a Sampler or provide a local training step.

How It Works

A compatible Hugging Face tokenizer converts the input text into tokens. The MiniLM model generates a representation for each token, and LogicalDOC averages the representations of valid tokens to produce one embedding. Padding positions are excluded from this average.

LogicalDOC then normalizes the resulting vector to unit length for similarity comparisons.

Model Configuration

The Mini Embedder uses a pre-trained model in ONNX format, executed locally through ONNX Runtime. Document text does not need to be sent to an external AI API for embedding.

Its vector size is fixed at 384 and must not be configured as 300. Doc2Vec properties, such as Alpha, Min Alpha, Window size and Minimum Word Frequency, do not apply to this model.

Choosing an Embedder

The Paragraph Embedder uses Doc2Vec through DL4J’s ParagraphVectors implementation. It requires training data supplied by a Sampler and supports training inside LogicalDOC. Its vector size is configurable, with a default of 300. Choose this model when you have a representative corpus and want it to learn vocabulary and contextual relationships from your domain text.

The Mini Embedder uses a pre-trained MiniLM-L6-v2 model that you import into LogicalDOC. It does not require a Sampler or support local training, and its vector size is fixed at 384. Choose this model when you want to generate semantic embeddings without preparing a local training corpus.

Generating Document Embeddings

After training the Paragraph Embedder or importing the Mini Embedder, configure an Embedding Scheme that connects the model to a Vector Store.

The scheduled Embedder task generates and stores embeddings for repository documents. Preparing the model and generating document embeddings are separate steps.

If you change the embedder used by an Embedding Scheme, regenerate the stored document embeddings so that document and query vectors remain compatible.