Visualising Text Embeddings: A Quick Exploration with Ollama and Dimensionality Reduction

Text embeddings map sentences to high-dimensional vectors where semantic similarity translates to geometric proximity. Two sentences about the same topic should land near each other; two sentences about unrelated topics should land far apart.

This post walks through a small experiment: embed 25 hand-picked sentences from 5 different topics using a local embedding model, then project the resulting 768-dimensional vectors down to 3D and check whether topical clusters emerge visually.

Setup: Ollama + nomic-embed-text

The embedding model runs locally via Ollama↗, which wraps open-weight models behind a simple API. The model used here is nomic-embed-text, a 137M-parameter model that produces 768-dimensional embeddings. It is small enough to run on a laptop CPU yet competitive with much larger models on retrieval benchmarks.

Generating all 25 embeddings takes under a second:

import ollama
import numpy as np

response = ollama.embed(model="nomic-embed-text", input=texts)
vectors = np.array(response["embeddings"])
# → shape: (25, 768)

The Sentences

To test whether the embedding space captures topic structure, we need sentences that a human would clearly group into categories. Five topics, five sentences each:

The sentences are intentionally short and topically unambiguous, as this is not a stress test. If the model cannot separate "Bach wrote intricate counterpoint for string instruments" from "VO2 max measures aerobic capacity in athletes", the embedding space is not useful.

The full set:

texts = [
    # ML / AI
    "Deep learning trains neural networks on large datasets",
    "Gradient descent optimizes model weights iteratively",
    "Transformers use attention mechanisms for NLP",
    "Graph neural networks operate on graph-structured data",
    "Convolutional networks excel at image recognition",
    # Security / Android
    "Android APK analysis detects malicious behaviour",
    "Static analysis inspects code without executing it",
    "Malware can exfiltrate contacts and location data",
    "Permission misuse is a common Android vulnerability",
    "Dynamic analysis runs the app in a sandbox environment",
    # Space / Science
    "Satellites capture multispectral imagery of Earth",
    "Copernicus monitors climate change from orbit",
    "Rocket propulsion relies on Newton's third law",
    "Orbital mechanics governs spacecraft trajectories",
    "Remote sensing detects land use and deforestation",
    # Music / Arts
    "The cello produces rich, resonant low-frequency tones",
    "Bach wrote intricate counterpoint for string instruments",
    "Music theory describes harmony, rhythm, and melody",
    "Practice and repetition build muscle memory in musicians",
    "Chamber music involves small ensemble performances",
    # Running / Sports
    "Marathon training requires long slow distance runs",
    "VO2 max measures aerobic capacity in athletes",
    "Nutrition and hydration are critical during endurance races",
    "Salomon makes trail running shoes for ultramarathons",
    "Recovery sleep is essential after intense workouts",
]

From 768 Dimensions to 3

768 dimensions are impossible to visualise directly. We need dimensionality reduction: algorithms that project high-dimensional data to 2D or 3D while preserving some notion of distance or neighbourhood structure. The experiment compares three popular methods.

t-SNE

t-SNE↗ (t-distributed Stochastic Neighbour Embedding) converts pairwise distances to conditional probabilities and minimises the KL divergence between the high-dimensional and low-dimensional distributions. It excels at revealing local clusters but does not preserve global distances: the relative positions of clusters are essentially arbitrary.

Key parameter: perplexity (set to 5 here, appropriate for 25 points). It roughly controls the effective number of neighbours each point considers.

from sklearn.manifold import TSNE

tsne = TSNE(n_components=3, perplexity=5, random_state=42, max_iter=1000)
coords = tsne.fit_transform(vectors)

UMAP

UMAP↗ (Uniform Manifold Approximation and Projection) constructs a fuzzy topological representation of the high-dimensional data and optimises a low-dimensional layout to match. Compared to t-SNE, it tends to better preserve global structure (the relative arrangement of clusters) while still giving tight local groupings. It is also significantly faster on large datasets.

import umap

reducer = umap.UMAP(
    n_components=3, n_neighbors=10,
    min_dist=0.1, random_state=42, metric="cosine"
)
coords = reducer.fit_transform(vectors)

Using metric="cosine" is natural here because cosine similarity is the standard metric for comparing text embeddings.

PaCMAP

PaCMAP↗ (Pairwise Controlled Manifold Approximation Projection) is a newer method that explicitly balances local structure (nearby points stay together), mid-range structure (moderately distant points are handled carefully), and global structure (far-apart points are pushed away). It does this through three types of point pairs with a phased training schedule.

import pacmap

reducer = pacmap.PaCMAP(
    n_components=3, n_neighbors=10,
    MN_ratio=0.5, FP_ratio=2.0, random_state=42
)
coords = reducer.fit_transform(vectors)

MN_ratio controls the weight of mid-near pairs relative to near pairs, while FP_ratio sets the repulsion strength from far pairs. Higher FP_ratio pushes clusters further apart.

Results

UMAP produces really good clusters, PaCMAP almost as good, and t-SNE has the least visible clusters. The interactive visualisation below shows the 25 embedded sentences projected to 3D. Drag to rotate, hover for details, and switch between reduction methods.

What the Plots Show

t-SNE shows the weakest separation in this run. The clusters are still visible, but they overlap slightly and the overall layout is unstable across seeds.

UMAP yields the clearest clusters. The topics separate cleanly, and the global arrangement is consistent enough to show ML/AI and Security nearer to each other than to Music or Sports.

PaCMAP is close to UMAP, with good separation and a readable global structure, though a couple of points sit nearer their neighbours across topics than in the UMAP view.

Observations

1. The embedding model separates all five topics cleanly. With just 5 sentences per topic, nomic-embed-text produces vectors that cluster perfectly by subject. The model has not seen these exact sentences during training; it generalises from its pretraining data.

2. The methods differ noticeably. t-SNE shows more overlap and less stable separation, while UMAP is the clearest. PaCMAP is close to UMAP but slightly less clean in this run.

3. Interesting near-misses. The Security and ML/AI clusters tend to be closer to each other than to Music or Sports. This reflects genuine semantic overlap: both talk about software, models, data, and analysis. Similarly, "Practice and repetition build muscle memory in musicians" sits at the edge of the Music cluster, closest to Sports, because practice and muscle memory are concepts shared by both domains.

Applications

Text embeddings appear in several practical settings:

• Retrieval-augmented generation (RAG): embed a query, retrieve the nearest document chunks, pass them to an LLM.
• Semantic search: match by meaning rather than keywords.
• Clustering and topic modelling: group documents by content without predefined categories.
• Classification features: use embedding vectors as input to downstream classifiers.

Visualising the embedding geometry, as done here, serves as a basic sanity check before relying on the vectors in any of these pipelines.

Reproducing This

Requirements:

Install the Ollama model with:

ollama pull nomic-embed-text

Conclusion

With just 25 sentences, a compact embedding model, and a few lines of dimensionality reduction, you can clearly see how well the model separates topics in practice. This experiment was run on a GPU for speed, but the model is small enough to run on a CPU as well. The results provide an intuitive sense of what text embeddings capture, making this a useful first step before using them for retrieval or classification tasks.

WSL

If you are running on WSL, Ollama runs on the Windows side and is not reachable at localhost from within WSL. You need to:

1. Set Ollama to listen on all interfaces on the Windows host. Open PowerShell as administrator and run:

[System.Environment]::SetEnvironmentVariable("OLLAMA_HOST", "0.0.0.0", "Machine")

Then restart Ollama.

2. In your Python code, point the client at the Windows host IP before importing ollama:

import os, subprocess

windows_ip = subprocess.check_output(
    ["bash", "-c", "ip route show | grep default | awk '{print $3}'"]
).decode().strip()
os.environ["OLLAMA_HOST"] = f"{windows_ip}:11434"

This resolves the default gateway (i.e. the Windows host) and sets the environment variable so the Ollama Python client connects to the correct address.