Document Merge and Dedupe

English | 简体中文

Merge overlapping Markdown clips and paginated documents in Obsidian without repeating content.

The plugin is useful when a long conversation, article, or multi-page resource was saved as several Markdown files. It aligns overlapping paragraphs, keeps an ordered union, and writes a separate verification report for uncertain gaps or conflicts.

Features

• Merge clips grouped by a normalized source URL.
• Recognize ChatGPT and Gemini conversation URLs.
• Recognize common pagination formats such as ?page=2, ?paged=2, and /page/2.
• Use explicit merge_group and merge_order frontmatter for custom sets.
• Select arbitrary Markdown documents from a searchable picker.
• Detect exact duplicate files and overlapping paragraphs.
• Preserve Markdown tables, code blocks, links, and attachment names.
• Write a verification report with coverage, anchors, conflicts, and possible gaps.
• Optionally move source files to the system or Obsidian trash after validation.

Commands

• Scan mergeable web clips: inspect source-based groups without changing files.
• Merge current clip group: merge the group that contains the active note, or choose a detected group.
• Select and merge Markdown documents: choose any two or more notes from a searchable picker. Source files are always preserved for this manual flow.

Obsidian displays command names in the language used by the plugin. The current user interface is Chinese.

Automatic grouping

The plugin groups clips in this order:

1. merge_group or clip_group in YAML frontmatter.
2. Stable ChatGPT or Gemini conversation IDs.
3. Common pagination parameters or /page/number paths.
4. The normalized full URL after removing fragments and common tracking parameters.

Similar titles alone never cause unrelated documents to be merged.

For sites without a recognizable page pattern, add:

merge_group: "my-long-article"
merge_order: 1

Use the same group and increment the order on later pages.

Output and safety

• Merged documents default to Clippings.
• Verification reports default to 00 待整理/文档合并待审核.
• New installations preserve all source files by default.
• Automatic cleanup is opt-in. Before cleanup, the plugin verifies output, report, and source SHA-256 hashes.
• Cleanup uses the system trash first and Obsidian's local trash as a fallback.
• The plugin cannot reconstruct content that was never captured by any source file; it reports possible gaps instead.

Privacy

All processing happens locally inside Obsidian.

• No network requests.
• No analytics or telemetry.
• No account or API key.
• No data is sent to the developer or any third party.

Installation

Community plugins

After the plugin is accepted into the Obsidian community directory:

1. Open Settings → Community plugins → Browse.
2. Search for Document Merge and Dedupe.
3. Install and enable it.

Manual installation

Download main.js, manifest.json, and styles.css from the latest GitHub release, then place them in:

<vault>/.obsidian/plugins/document-merge-dedupe/

Reload Obsidian and enable the plugin.

Development

npm install
npm test
npm run build

The production release is built with:

npm run build:release

License

MIT