Vision2Canvas

Convert physical handwritten note photos, whiteboard sketches, and mind maps into native, structured Obsidian .canvas whiteboards with Vision AI.

English README | 繁體中文說明文件

📖 Overview

Vision2Canvas bridges the gap between analog handwritten thinking and digital Obsidian knowledge bases. Using multimodal Vision Large Language Models (Vision MLLMs like Google AI Studio Gemini, OpenAI, Claude, or OpenClaw), this Obsidian plugin not only extracts text OCR but also:

• Preserves Spatial Layout: Maps handwritten sections to absolute (x, y) coordinates and card dimensions.
• Translates Visual Connections: Converts arrows, flowchart paths, and group boxes into native Obsidian Canvas nodes, edges, and groups.
• Desktop Workflow: Instantly process clipboard screenshots or images directly in your vault.

🌟 Key Features

1. Spatial Bounding Box Mapping
   Maps handwritten columns, bullet points, and margin notes to Canvas nodes with proper spacing and color categorization.

2. Smart Arrow & Hierarchy Detection
   Automatically detects connecting arrows, labels them, and attaches them to the best attachment sides (top, bottom, left, right).

3. OpenAI & Google AI Studio Compatible API
   Seamlessly connects to Google AI Studio (Gemini), OpenAI API, or local MLLM gateways.

4. Multi-Channel Desktop Triggers

   • Clipboard Command: Copy any image or screenshot to clipboard → Convert Clipboard Image to Canvas.
   • Vault File Menu: Right-click any image (.png, .jpg, .jpeg, .webp) in Obsidian file explorer → Convert to Obsidian Canvas Whiteboard.

🏗 System Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                           Input Triggers                                │
│           ┌─────────────────────┐       ┌───────────────┐               │
│           │ Clipboard Screenshot│       │ Vault Image   │               │
│           └──────────┬──────────┘       └───────┬───────┘               │
└──────────────────────┼──────────────────────────┼───────────────────────┘
                       │                          │
                       ▼                          ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                      Vision2Canvas Plugin Core                          │
│  ┌───────────────────────────────────────────────────────────────────┐  │
│  │ VisionClient (Google AI Studio / OpenAI Compatible Vision API)   │  │
│  └──────────────────────────────────┬────────────────────────────────┘  │
│                                     │ (Spatial OCR + JSON Schema Output)│
│  ┌──────────────────────────────────▼────────────────────────────────┐  │
│  │ CanvasBuilder & CanvasValidator                                   │  │
│  │ - Coordinate Normalization (scaleX, scaleY)                       │  │
│  │ - Edge Connection Sides calculation (top, bottom, left, right)     │  │
│  └──────────────────────────────────┬────────────────────────────────┘  │
└─────────────────────────────────────┼───────────────────────────────────┘
                                      │
                                      ▼
                      ┌───────────────────────────────┐
                      │ Obsidian Vault (.canvas File) │
                      └───────────────────────────────┘

🚀 Quick Start & Installation

1. Build & Install Plugin

# Clone repository into your vault's plugin directory
cd /path/to/your/vault/.obsidian/plugins/
git clone https://github.com/yourname/vision2obsidian.git
cd vision2obsidian

# Install dependencies and compile bundle
npm install
npm run build

Then in Obsidian: Settings → Community Plugins → Enable Vision2Canvas.

2. Configure Settings

In Obsidian Settings → Vision2Canvas:

• AI API Endpoint: https://generativelanguage.googleapis.com/v1beta/openai (or https://api.openai.com/v1)
• API Key: YOUR_API_KEY
• Vision Model Name: gemini-flash-latest (or gpt-4o)

🧪 Testing & Verification

Run the included standalone test script to test converting ref/sample.jpeg into output_sample.canvas:

npm run test:sample

Output:

--- TEST RESULTS ---
Generated 15 nodes and 4 edges.
Saved output canvas to: /Users/user/projects/vision2obsidian/output_sample.canvas

📜 License

MIT License © 2026 beno66edu