VLMs OCR for Obsidian

VLMs OCR turns Obsidian into an intelligent multimodal transcription workspace. Extract text, LaTeX mathematical formulas, and tables from images and PDFs, or generate in-depth structured descriptions of charts, plots, and diagrams—completely offline using local Vision-Language Models (VLMs) or in the cloud.

Powered locally by high-efficiency VLMs like GLM-OCR (0.9B) or any vision model via llama.cpp / Ollama, as well as the Hugging Face Inference API.

---

✨ Features

• 🧠 Auto-detect & Intelligent OCR Modes:
  • Auto-detect: Intelligently classifies content and outputs formatted Markdown, inline/display LaTeX, or converts charts and diagrams into rich textual explanations.
  • Describe Image / Chart: Generates structured markdown summaries of graphs, plots, architectures, workflows, and infographics.
  • Full Document: Converts book pages, slides, papers, and scanned notes into clean Markdown.
  • Formulas Only: Formats math expressions directly into Obsidian-compatible LaTeX ($$...$$ and $ ... $).
  • Tables: Parses complex tabular data into pristine Markdown tables.
  • Plain Text & Custom Prompt: Extract unformatted text or provide your own instruction prompt.
• ⚡ Optimized for Low-VRAM Hardware (MX450 / 2GB–4GB GPUs):
  • Smart automatic downscaling preserves aspect ratio while preventing token explosion and out-of-memory errors.
  • Zero-temp-file in-memory clipboard pipeline processes images directly from RAM.
  • One-click Low-VRAM Preset configures context window and memory limits automatically.
• 📋 Seamless Workflow & Hotkeys:
  • Paste image directly as OCR: Press Ctrl+Alt+V (configurable) to grab an image from your clipboard and immediately insert transcribed Markdown/LaTeX at your cursor.
  • Non-blocking inline preview: Real-time markdown placeholder (<!-- [OCR is being generated...] -->) keeps your editor responsive while the model runs.
  • Context Menu & Ribbon: Right-click any image or PDF in your file explorer to transcribe or describe it.
• 🔒 100% Local & Private or Zero-Install Cloud:
  • llama.cpp: Native, blazing-fast GGUF execution with optional GPU offloading.
  • Ollama: Connects to your local Ollama instance with auto-start support.
  • Hugging Face API: Free cloud inference with zero local dependencies or GPU requirements.

---

🚀 Quick Start

Option A: Cloud Inference (Zero-Install, No GPU Required)

1. Create a free account at huggingface.co.
2. Generate a read token in Hugging Face Token Settings.
3. In Obsidian, go to Settings → VLMs OCR:
   • Turn off Use local model.
   • Paste your token into the API Key field.

---

Option B: Run Locally with llama.cpp (Recommended for Speed & Low VRAM)

Running GLM-OCR locally in GGUF format requires less than 1.2 GB of VRAM, making it ideal even for entry-level GPUs (like the NVIDIA GeForce MX450 / GTX 1650) or pure CPU execution.

1. Obtain llama-server:
   • Download the latest llama.cpp release with your preferred acceleration (CUDA or CPU) from llama.cpp Releases.
2. Download GLM-OCR GGUF weights:
   • Model: GLM-OCR.Q4_K_M.gguf (~530 MB)
   • Multimodal projector: GLM-OCR.mmproj-Q8_0.gguf (~484 MB)
3. Configure Settings:
   • In Obsidian, go to Settings → VLMs OCR.
   • Enable Use local model and choose Local backend: llama.cpp.
   • Set llama.cpp executable path to your llama-server.exe (or llama-server on Linux/macOS).
   • Click Apply Low-VRAM preset or set arguments:

     -m "/path/to/GLM-OCR.Q4_K_M.gguf" --mmproj "/path/to/GLM-OCR.mmproj-Q8_0.gguf" -ngl 99 -c 4096 --host 127.0.0.1 --port 8080
     

---

Option C: Run Locally with Ollama

1. Install Ollama.
2. Pull your desired vision model (e.g., ollama pull glm-ocr or ollama pull minicpm-v).
3. In Obsidian Settings → VLMs OCR:
   • Enable Use local model and select Local backend: Ollama.
   • Enter your model name in Ollama model.

---

⌨️ How to Use

1. Clipboard Paste (Fastest)

1. Copy any screenshot, math formula, or document snippet to your clipboard (Win + Shift + S on Windows, Cmd + Shift + 4 on macOS).
2. In your Obsidian note, run the command "OCR image from clipboard" (or bind it to Ctrl+Alt+V).
3. The recognized Markdown or LaTeX expression will appear right at your cursor position!

2. Vault Files

• Right-click any supported image (.png, .jpg, .webp, .bmp, .gif) or .pdf in the file explorer.
• Select Perform OCR (Auto / Active mode) to copy the text to your clipboard, or Describe Image / Chart for an analytical breakdown.

---

⚙️ Configuration Reference

Setting	Description	Default
OCR mode	Choose default parsing behavior (Auto-detect, Describe, Document, Formulas, Tables, Text, Custom)	Auto-detect
Max image dimension	Proportional image downscaling threshold to conserve VRAM (set 1536 for 2GB GPUs, 0 to disable)	1536
In-memory clipboard	Process clipboard captures entirely in memory without writing temporary files to disk	Enabled
Show status bar	Displays live backend status indicators (Ready ✅, Loading ⚙️, Offline ❌)	Enabled

---

🛠️ Requirements & Compatibility

• Obsidian: v1.4.0 or newer.
• Operating Systems: Windows, macOS, Linux (Desktop only).
• Supported File Types: PNG, JPG, JPEG, WEBP, BMP, GIF, PDF.

---

🔒 Security & Disclosures

In accordance with the Obsidian Developer Policies:

• Desktop Only (isDesktopOnly: true): This plugin is strictly for desktop platforms because local Vision-Language Model execution requires native binary execution and filesystem access.
• Shell Execution (child_process):
  • Used exclusively to spawn and manage the local inference servers chosen and configured by the user (llama-server.exe or ollama).
  • The plugin does not execute arbitrary shell commands. It only spawns the configured local server executable with inference parameters (such as --port, -m, --mmproj, -c).
• Filesystem Access (fs):
  • Local Model Files: Reads user-selected local model weights (.gguf files) and local server executables configured in settings.
  • Vault Files & Images: Reads user-selected image and PDF files from the vault or file picker to convert them into base64 visual inputs for OCR processing.
  • No system files outside the configured paths and selected attachments are modified or deleted.
• Network Use:
  • Local Mode (Default): Connects strictly to localhost / 127.0.0.1 (to the local llama-server or Ollama HTTP REST API). Zero data leaves your computer.
  • Cloud Mode (Optional): If and only if the user explicitly switches the backend to Hugging Face API, image data is transmitted directly over HTTPS to Hugging Face Inference API using the user-provided API key.

---

📄 License & Attribution

• Released under the GNU General Public License v3.0 (GPL-3.0).
• Based on and inspired by the original obsidian-latex-ocr by lucasvanmol.
• Model architecture powered by GLM-OCR by the Zhipu AI & GLM research team.