obsidianUMAP — UMAP Graph for Obsidian

Current Status

🟢 Settings UI Complete (v0.1.3): Comprehensive settings interface with clustering algorithms, gradient strategies, filtering system, and interactive visualization features.

🟢 Prototype proven: in-Obsidian indexing + working interactive view (pan/zoom/hover/double-click open) with large-vault performance optimizations validated on ~5.8k notes.

🟡 Optimization Phase: Testing K-scaling formulas, UMAP parameter sensitivity, and cluster geometry optimization on clippings test vault (n=570).

Recent Updates (January 29, 2026)

• ✅ Comprehensive settings UI organized by process (clustering, visualization, filtering, performance, presets)
• ✅ Multiple clustering algorithms (K-means implemented, HDBSCAN/DBSCAN placeholders)
• ✅ Gradient strategies (by cluster, similarity, tag, metadata)
• ✅ Rule-based filtering system with multiple operators
• ✅ Interactive legend panel with gradient selector and preset switcher
• ✅ Cluster highlighting on click
• ✅ Enhanced tooltips with metadata
• ✅ Fixed null reference error in view loading
• ✅ Test vault feature flag for development workflow

See LESSONS-LEARNED.md for development insights and recommendations.

Phases to Completion (Roadmap)

flowchart TD
  A[Phase 0: Prototype v0.1.3] --> B[Phase 1: Data & Clustering Baseline]
  B --> C[Phase 2: Optimization & Validation]
  C --> D[Phase 3: Live-ish Updates]
  D --> E[Phase 4: Native-Feeling Toggle]
  E --> F[Phase 5: Optional Enhancements]

  B --> B1[TF-IDF + K-means + Silhouette]
  B --> B2[Clippings test vault]
  B --> B3[Baseline UMAP 2D view]

  C --> C1[Optimal K scaling]
  C --> C2[UMAP parameter sweep]
  C --> C3[Cluster geometry & tightness tuning]
  C --> C4[Validation metrics + visual proofs]

  F --> F1[Semantic embeddings]
  F --> F2[3D view]
  F --> F3[Subject-matter labeling]

Optimization Phase (Current Focus)

We are in Phase 2: Optimization & Validation, with two primary goals:

1. Optimal K Scaling
   Determine how optimal cluster count (K) scales with input size (n).
   Output: A defensible scaling rule (e.g., K ≈ n / α or K ≈ β√n) with evidence from parameter sweeps.

2. Cluster Geometry & Tightness
   Improve visual separation without overclaiming semantic accuracy.
   Output: Parameter guidance (UMAP min_dist, spread, repulsion_strength) and visual comparison grids.

Scope Constraints:

• In scope: Empirical optimization of K-scaling formulas, UMAP parameter tuning, cluster tightness metrics.
• Out of scope (for now): Full semantic embeddings, subject-matter modeling, 3D visualization.
• Lightweight enhancement: Cluster labeling via top TF-IDF terms + tags (not full topic modeling).

Meta-Analysis (Decision Support Evidence)

This folder holds cross-epoch evidence that justifies project-wide decisions (not single-epoch artifacts): /home/dsowerby/Cursor AI Projects/Plugins - Obsidian/obsidianUMAP/analysis-output/meta-analysis

K-Scaling Decision (Logarithmic Asymptote Formula)

Evolution of the decision:

1. Epoch 4 (small dataset, 570 notes): Tested 8 formulas, n_div_6 performed best
2. Epoch 5 (large dataset, 10k notes): n_div_6 produced 50+ clusters, losing meaningful relationships
3. Solution: Logarithmic asymptote formula that scales gracefully

Current default: K_max = min(110, max(6, 18 + 12 * log10(n/100)))

Why this formula:

• Typical range: 6-24 clusters for most practical vault sizes (100-2000 notes)
• Rarely exceeds: 80 clusters even on very large datasets
• Asymptote: Approaches ~110 clusters for massive datasets (100k+ notes)
• Logarithmic growth: Prevents cluster explosion on large datasets while maintaining granularity

Behavior by dataset size:

• 100 notes → 18 clusters (baseline, within typical range)
• 500 notes → ~24 clusters (top of typical range)
• 1,000 notes → ~28 clusters (slightly above typical, reasonable)
• 10,000 notes → ~46 clusters
• 100,000+ notes → ~66-110 clusters (asymptote)

Visualization constraint:
Graph geometry distorts with >20 clusters. This is a visualization issue, not a clustering quality issue. The clustering algorithm can produce meaningful clusters beyond 20, but 2D UMAP projection becomes harder to interpret.

See: K_SCALING_RATIONALE.md for complete analysis and comparison with previous formulas.

Validation Process (Specific and Evidence-Based)

We validate in stages and document each stage with artifacts:

1. Data Scope Definition

• Test vault: Clippings subset from 2025+2026 vaults (~570 files)
• Rationale: Real content with semantic meaning, safe subset for testing
• Evidence: File count verification, vault structure checks
• Limitation: Results apply to this specific dataset; generalization requires broader testing

2. Embedding Quality (TF-IDF Baseline)

• Check: Dimensions, non-empty vectors, token consistency
• Evidence: TF-IDF shape (n_docs × vocab_size), sample token inspection
• Limitation: Lexical similarity only; semantic relationships may be missed

3. Clustering Quality

• Metric: Silhouette Score (primary), Davies-Bouldin Index (secondary validation)
• Evidence: K vs. Silhouette curves, parameter sweep grids
• Success criteria (qualified):
  • Look for relative improvement in silhouette across parameter sweeps
  • Avoid overclaiming semantic "truth"; interpret as structure-separation indicator
  • Higher silhouette = better cluster separation, not necessarily "correct" semantic grouping

4. Projection Quality (UMAP)

• Metric: Qualitative separation and stability across parameter grid
• Evidence: 8×8 parameter grids with labeled axes, cluster hull overlays
• Success criteria: Clusters are visually distinct and stable across parameter choices
• Limitation: UMAP affects layout only; clustering quality is determined before projection

5. Visual Readability & Interpretation

• Cluster hull overlays: 2% opacity boundaries for visual reinforcement
• Legends and axis labels: Clear parameter identification
• Evidence: Comparison grids, annotated plots
• Success criteria: Reader can interpret what each axis represents and compare results

6. Limitations (Explicit)

• UMAP parameters: Affect visualization layout, not clustering assignments
• TF-IDF: Lexical similarity only; misses semantic nuance
• Cluster "quality": Relative to dataset and metrics; not absolute semantic truth
• Generalization: Results from clippings subset may not apply to all vault types

Success Criteria (Qualified)

We consider progress successful when:

• K scaling: Shows stable, defensible pattern across subsample sizes (epoch_1, epoch_4)
• UMAP parameter sweeps: Show consistent cluster separation without extreme distortions (epoch_3)
• Visual output: Interpretable (labels, legends, cluster boundaries) and reproducible
• Claims remain bounded: No overstatement beyond what metrics evidence supports

Evidence artifacts:

• Parameter sweep grids (8×8) with labeled axes
• K-scaling analysis plots (n vs K, ratio analysis, formula comparison)
• Silhouette score distributions by parameter
• Cluster geometry metrics (intra-cluster vs inter-cluster distances)

What This Plugin Is Trying To Become

An Obsidian graph mode that helps you navigate a big vault by similarity (semantic + tags + near-duplicate similarity) and then overlays cluster-quality analytics:

• D (Top priority): a “best clusters” workflow (quality curve over K, suggest local minima)
• E: live-ish behavior (incremental updates + background rebuilds)
• A: native-feeling toggle (quick switch between native graph and UMAP mode)

The Key Concept: UMAP Is Projection, Not Clustering

UMAP is the step that projects high-dimensional structure down to 2D/3D; it does not by itself decide clusters. This is explained well here: https://pair-code.github.io/understanding-umap/.

That means our pipeline is:

1. Represent notes as vectors / signals
2. Cluster (or at least compute neighborhoods / similarity graph)
3. Project with UMAP (2D first, 3D later)
4. Render + interact like a graph view

What Already Works (v0.1.3)

• Index build inside Obsidian
• Progress modal during indexing
• Per-vault cache written to:
  • .obsidian/plugins/umap-graph-view/<vaultId>.umap.json
• Interactive view
  • pan / zoom
  • hover tooltip
  • double-click opens note
  • edges drawn with LOD caps
• Performance optimizations
  • O(1) link lookups via noteMap
  • viewport culling
  • node/link caps for smooth frames

Leverage Obsidian’s Native Capabilities (What We Get “For Free”)

This project should lean on Obsidian’s built-in indexing/caching as much as possible:

• Vault file inventory: list markdown files from the vault (avoid manual filesystem walks).
• Metadata cache: get link structure and per-file caches without reparsing everything constantly.
  • “resolved links” (for edges)
  • “unresolved links” (for ghosts / broken links)
  • frontmatter/tags/links caches (for tag-informed clustering and filters)
• Vault events: react to file create/modify/rename/delete so we can do incremental updates and batch rebuilds rather than full rescans every time.

D — Cluster Quality Curve + “Best K” Suggestions (Primary Deliverable)

User experience

• A panel that can run “cluster experiments” for a range of K values
• A plotted curve:
  • x-axis: number of clusters (K)
  • y-axis: cluster-quality score (we previously called this “entropy”)
• Highlight local minima / best candidates and let the user switch the active clustering choice.

Critical decision: what is the y-axis?

Decision (based on research): Default to Silhouette Score for K-means clustering.

Rationale:

• Silhouette Score (ranges -1 to 1, higher is better): Measures how similar a point is to its own cluster vs. other clusters. Most intuitive and widely used for K-means.
• Davies–Bouldin Index (lower is better): Alternative geometry-based metric; can be used as secondary validation.
• Graph modularity: Use if clustering from link/similarity graph (Louvain/Leiden algorithms).
• Tag/topic purity entropy: Domain-specific metric; useful for measuring semantic coherence within clusters.

Implementation note: For the quality curve, we'll compute Silhouette Score across a range of K values. Local minima in the curve (where score drops) may indicate suboptimal K; we'll highlight local maxima instead (where clustering quality peaks).

Why n_div_6 for K scaling (current best)

We tested 8 K-scaling formulas on the clippings dataset (n≈570). Each formula sets the maximum K we allow K-means to test. The goal is to pick a K range that is large enough to separate distinct topics without fragmenting too much.

Result: K_max = n / 6 produced the highest mean silhouette score across subsample levels, with n_div_8 nearly tied (slightly lower mean, lower variance).

Why it beats the others:

• Conservative formulas (like sqrt or hard caps at 20) force too few clusters, which mixes unrelated notes and lowers separation quality.
• Linear scaling (like n / 6) keeps K proportional to dataset size, allowing the model to express more fine-grained structure as n grows.
• Silhouette score improves when clusters are tighter and further apart; more available K values let K-means find that better separation.

Non-coding summary: Think of it like organizing a library. With too few shelves, different topics get forced together. With more shelves, books can be grouped more accurately. n / 6 gave us enough shelves to separate topics without going to the extreme of “one shelf per book.”

E — Live-ish Updates Without Constant Full Recompute

Full recluster + full UMAP for every edit is too expensive. Target behavior:

• Fast path (immediate UI response): when a note changes, update metadata + embedding for that note and do an approximate placement (nearest cluster centroid / nearest neighbors).
• Slow path (accuracy): schedule a debounced background rebuild (cluster + UMAP), show progress, then swap in the refined layout.

A — Native-Feeling Toggle (Without Fighting Obsidian Internals)

Pragmatic interpretation of “toggle”:

• Obsidian native graph stays native.
• This plugin provides a UMAP Graph View that is one click away (command/ribbon).
• Optional later: try UI integration inside the native graph pane, but treat it as brittle.

Clustering on Text Files: Practical Options (In-Plugin)

We need a clustering pipeline that works within Obsidian’s constraints (CPU, memory, UI responsiveness).

Representations (fast → heavy)

Recommended progression (start simple, add sophistication):

1. TF‑IDF (sparse vectors) — Start here for v1

   • Good baseline for lexical similarity
   • Cheap, local, no external dependencies
   • Works well with K-means
   • Preprocessing: tokenization, lowercasing, stop-word removal, optional stemming/lemmatization

2. Hybrid signals (TF-IDF + tags + link structure)

   • Combine text similarity with Obsidian-native signals
   • Weighted combination: w_text * TF-IDF + w_tags * tag_overlap + w_links * link_similarity
   • Near-duplicate detection: SimHash/MinHash for content fingerprints (stronger clustering signal)

3. Semantic embeddings (Sentence Transformers / BERT-based)

   • Best quality for semantic similarity
   • Options:
     • Local (in-plugin): ONNX runtime + lightweight model (e.g., all-MiniLM-L6-v2 ~80MB)
     • External preprocessing: Generate embeddings via CLI/indexer, cache in vault
   • Trade-off: quality vs. bundle size + compute time

For v1 (Priority D): Start with TF-IDF + tags. This gives us a working cluster-quality curve without heavy dependencies. Semantic embeddings can be added as an optional "enhanced mode" later.

Clustering algorithms (fast → fancy)

Primary choice for Priority D: K-means (fast, requires K, but we're building best-K tooling).

Alternatives (for future exploration):

• DBSCAN / HDBSCAN: Density-based; finds arbitrary-shaped clusters, handles noise. Good for exploratory clustering when K is unknown.
• Hierarchical clustering: Builds dendrogram; useful for exploring K choices interactively, but can be expensive for large vaults.
• Graph community detection (Louvain/Leiden): Excellent if we build a similarity/kNN graph first; naturally finds optimal number of communities.
• Spectral clustering: Uses eigenvalues of similarity matrix; good for complex distributions but computationally heavier.

For v1 (Priority D): Start with K-means + Silhouette Score curve. This gives us the "best K" workflow immediately. We can add DBSCAN/HDBSCAN later as an alternative clustering mode.

Known Unknowns → Resolved vs. Needs Experimentation

✅ Resolved (Based on Research)

• Quality metric for y-axis: Silhouette Score (default for K-means). Davies-Bouldin as secondary option.
• Text representation for v1: TF-IDF (baseline) + tags. Semantic embeddings as optional enhancement.
• Clustering algorithm for v1: K-means (fast, works with our best-K workflow). DBSCAN/HDBSCAN for future exploration.

🔬 Needs Experimentation (Implementation Decisions)

D — "Best clusters" analytics

• Efficiency: How fast can we compute Silhouette Score curves for K=2..50 on 1k/5k/10k notes?
  • Hypothesis: K-means is O(n·k·i) where n=notes, k=clusters, i=iterations. Should be feasible with debouncing.
• Curve smoothing: How do we make "local maxima" detection robust (avoid noise from random K-means initialization)?
  • Options: multiple runs per K (take average), smoothing filter, or hierarchical pre-clustering to seed K-means.

E — Live updates

• Event signals: Which Obsidian events are most reliable?
  • Test: vault.on('modify'), metadataCache.on('resolve'), metadataCache.on('changed')
  • Need to verify: do these fire for frontmatter/tag changes, or only file content?
• Debounce window: What delay feels "live" without thrashing?
  • Start with: 2-5 seconds for fast path, 30-60 seconds for background rebuild.
• Stability: Can we keep approximate placement stable (avoid nodes jumping on every edit)?
  • Strategy: only update position if similarity change exceeds threshold; use animation/transition.

Text similarity feasibility

• TF-IDF performance: How does TF-IDF + K-means perform on real vaults vs. semantic embeddings?
  • Test on: technical docs, personal notes, mixed content.
• Semantic embeddings path: If we add embeddings, should we:
  • Bundle ONNX + lightweight model (increases plugin size ~80-100MB)?
  • Or require external CLI/indexer (better performance, but setup friction)?

Scaling and storage

• Cache size limits: How big can embeddings + indices get before Obsidian performance degrades?
  • Test: 5k notes ≈ ~10-20MB (TF-IDF sparse) vs. ~40-80MB (dense embeddings 384-dim).
  • Strategy: lazy-load embeddings, compress cache, or cap to recent N notes.
• Approximate nearest neighbor (ANN): Do we need an ANN index for fast nearest-neighbor placement?
  • For incremental updates: yes, if we want O(log n) lookup instead of O(n) brute force.
  • Options: HNSW (hierarchical navigable small world) or LSH (locality-sensitive hashing).

Research Synthesis (28Jan26)

Sources: Text clustering algorithms, embedding models, similarity metrics, evaluation methods.

Key findings:

• Clustering algorithms: K-means (fast, requires K), DBSCAN/HDBSCAN (density-based, no K needed), hierarchical (exploratory, expensive), graph community detection (Louvain/Leiden for similarity graphs).
• Text representation: TF-IDF (baseline, cheap), Sentence Transformers (semantic, heavier), hybrid approaches (combine multiple signals).
• Similarity metrics: Cosine similarity (most common for text, orientation-based), Euclidean distance (magnitude-based), Jaccard (set-based).
• Evaluation metrics: Silhouette Score (standard for K-means, -1 to 1, higher better), Davies-Bouldin Index (lower better), Calinski-Harabasz Index, Adjusted Rand Index (requires ground truth).
• Best practices: Start with pretrained/baseline models, use dimensionality reduction (UMAP/t-SNE) for visualization, iterate and evaluate with both quantitative metrics and visual validation.

Implementation guidance:

• For Priority D: K-means + Silhouette Score curve is the standard approach. Compute quality scores for K=2..K_max, plot curve, highlight local maxima (best K candidates).
• Preprocessing pipeline: tokenization → lowercasing → stop-word removal → (optional) stemming/lemmatization → vectorization (TF-IDF or embeddings).
• Performance: K-means is O(n·k·i); should handle 5k-10k notes with reasonable K ranges (2-50) in seconds to minutes.

Testing Environment

Primary: Use real clippings files from your 2025 and 2026 vaults. These are actual content with semantic meaning, perfect for testing clustering.

Clippings Collector: collect-clippings.js

• Queries datalake metadata to find files tagged "clippings"
• Collects files from both Clippings/ and 11 Clippings/ folders
• Copies ~500-900 real files to a test vault
• Preserves tags, links, and frontmatter

Quick Start:

# Collect all clippings from both vaults
node collect-clippings.js --copy-to ~/Vaults/Test-Clippings-UMAP

# List clippings without copying
node collect-clippings.js --list-only

Fallback: Synthetic test vault generator (test-vault-generator.js) for rapid iteration and controlled scenarios.

See TESTING.md for detailed testing strategy and scenarios.

File Layout (Minimal)

• README.md — single working document (this file)
• DESIGN-PRINCIPLES.md — visual/UX notes (keep)
• TESTING.md — testing guide and scenarios
• test-vault-generator.js — synthetic test vault generator
• main.js — plugin entrypoint
• cli/ — optional external indexer (currently exists)