Llm Docs Builder

Optimize your documentation for LLMs and RAG systems. Reduce token consumption by 67-95%.

llm-docs-builder transforms markdown documentation to be AI-friendly and generates llms.txt files. It normalizes links, removes unnecessary content, optimizes documents for LLM context windows, and enhances documents for RAG retrieval with hierarchical heading context and metadata.

The Problem

When LLMs fetch documentation, they typically get HTML pages designed for humans - complete with navigation bars, footers, JavaScript, CSS, and other overhead. This wastes 70-90% of your context window on content that doesn't help answer questions.

Real-world results from Karafka documentation (10 pages analyzed):

Average reduction: 83% fewer tokens

Quick Start

Measure Your Current Token Waste

# Using Docker (no Ruby installation needed)
docker pull mensfeld/llm-docs-builder:latest

# Compare your documentation page
docker run mensfeld/llm-docs-builder compare \
  --url https://yoursite.com/docs/getting-started.html

Example output:

============================================================
Context Window Comparison
============================================================

Human version:  127.4 KB (~32,620 tokens)
  Source: https://karafka.io/docs/Pro-Virtual-Partitions/ (User-Agent: human)

AI version:     46.3 KB (~11,854 tokens)
  Source: https://karafka.io/docs/Pro-Virtual-Partitions/ (User-Agent: AI)

------------------------------------------------------------
Reduction:      81.1 KB (64%)
Token savings:  20,766 tokens (64%)
Factor:         2.8x smaller
============================================================

Transform Your Documentation

# Single file
llm-docs-builder transform --docs README.md

# Fetch and transform a remote page
llm-docs-builder transform --url https://yoursite.com/docs/page.html

# Bulk transform with config
llm-docs-builder bulk-transform --config llm-docs-builder.yml

HTML to Markdown Conversion: The transformer automatically detects and converts HTML content to clean markdown format. This works seamlessly with both local files and remote URLs, converting HTML tables, code blocks, and other elements into their markdown equivalents.

Installation

Docker (Recommended)

docker pull mensfeld/llm-docs-builder:latest
alias llm-docs-builder='docker run -v $(pwd):/workspace mensfeld/llm-docs-builder'

RubyGems

gem install llm-docs-builder

Features

Automatic HTML to Markdown Conversion

The tool automatically detects and converts HTML content to clean markdown:

• HTML Tables → Markdown tables
• HTML Code Blocks → Fenced code blocks
• Figures & Captions → Clean markdown equivalents
• Seamless Integration - Works with local files and remote URLs without special configuration

# Transform HTML content automatically
llm-docs-builder transform --docs page-with-html.md
llm-docs-builder transform --url https://site.com/docs/api.html

Measure and Compare

# Compare what your server sends to humans vs AI
llm-docs-builder compare --url https://yoursite.com/docs/page.html

# Compare remote HTML with local markdown
llm-docs-builder compare \
  --url https://yoursite.com/docs/api.html \
  --file docs/api.md

Generate llms.txt

# Create standardized documentation index
llm-docs-builder generate --config llm-docs-builder.yml

Configuration