Generate llms.txt and llms-full.txt for your site or any docs site

AI assistants and AI search engines increasingly look for a file called /llms.txt: a short Markdown index that tells a language model what a site is about and where its important pages are — the way robots.txt and sitemap.xml do for search crawlers. Writing one by hand for a site with hundreds of pages is tedious, and keeping it current is worse. This guide shows how to generate llms.txt and llms-full.txt automatically from any site, for $1 per 1,000 pages.

What you get

Two files in the run's key-value store, plus one dataset row per page:

A real dataset row from a recorded run:

{
  "site": "crawlee.dev",
  "url": "https://crawlee.dev/js/docs/quick-start",
  "title": "Quick Start",
  "description": "With this short tutorial you can start scraping with Crawlee in a minute or two. To learn more, read the Introduction.",
  "section": "Docs",
  "wordCount": 1484,
  "tokens": 4413,
  "markdown": "Version: 3.18\n\n# Quick Start\n\nWith this short tutorial you can start scraping with Crawlee in a minute or two. To learn in-depth how Crawlee works, read the [Introduction](https://crawlee.dev/js/docs/introduction), which is a comprehensive step-by-step guide for creating your …"
}

Titles are cleaned (the repeated "| Site Name" suffix is removed) and descriptions come from each page's meta description — or its first paragraph when the meta description is missing or the same boilerplate on every page.

Step-by-step

1. Open the tool and enter one or more Start URLs — your home page, or the entry page of a docs site.
2. Set Max pages per website (100 by default). This is the billed unit.
3. A start URL inside a folder keeps the crawl in that folder: https://example.com/docs/intro reads only /docs pages. Use Include path prefixes to choose folders yourself.
4. Leave Use sitemap on so the crawl starts from the site's real pages.
5. Click Start, then open the Output tab and download llms.txt and llms-full.txt.
6. To publish, upload llms.txt to your web root so it is served at https://yoursite.com/llms.txt.

Run llms.txt Generator (Website to llms.txt & llms-full.txt) on Apify →

Call it from code

import json
import os
from apify_client import ApifyClient
client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("webdatatools/llms-txt-generator").call(run_input=json.loads("{\n    "startUrls": [\n        {\n            "url": "https://crawlee.dev/js/docs/quick-start\"\n        }\n    ],\n    "maxPages": 50,\n    "maxDepth": 3\n}"))
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('webdatatools/llms-txt-generator').call({
    "startUrls": [
        {
            "url": "https://crawlee.dev/js/docs/quick-start"
        }
    ],
    "maxPages": 50,
    "maxDepth": 3
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

curl -X POST "https://api.apify.com/v2/acts/webdatatools~llms-txt-generator/run-sync-get-dataset-items" 
-H "Authorization: Bearer $APIFY_TOKEN" 
-H "Content-Type: application/json" 
-d '{"startUrls":[{"url":"https://crawlee.dev/js/docs/quick-start"}],"maxPages":50,"maxDepth":3}'

Recipes

Publish an llms.txt for your own product docs

Run the tool on your docs URL, review the generated llms.txt (section names come from your URL structure), and deploy it with your site. Schedule a monthly run and diff the new file against the published one to catch pages that were added or removed.

Give an AI assistant a library's documentation

Generate llms-full.txt for a framework or API you're working with and attach it to your assistant or IDE. The tokens count in SUMMARY tells you up front whether the whole file fits the model's context window; if not, narrow the crawl with Include path prefixes.

Build a RAG index from a docs site

Pass the run's dataset to RAG Text Chunker with inputDatasetId set to this run's dataset ID. It finds the Markdown column automatically and returns heading-aware, token-counted chunks with the page URL attached, ready for embeddings.

How much does it cost?

Primary event rates from a catalog snapshot; this is not the total price of a run. Additional charge events and Actor Start fees may apply; start fees can scale with memory. Check the Actor's Pricing tab on Apify for all events, conditions and current plan-specific rates, and set a maximum run budget before starting.

The llms.txt and llms-full.txt files themselves are free; you pay per page read. Pages that fail to load are listed in the ERRORS record and cost nothing.

How it compares

FAQ

What is llms.txt?

A proposed standard, described at llmstxt.org, for a Markdown file at /llms.txt that tells language models what a site contains and where its key pages are.

The site already has an llms.txt. Why generate one?

SUMMARY tells you when it does. A generated file is still useful to compare coverage, or when you need llms-full.txt and the site doesn't publish one.

Can I generate files for several sites in one run?

Yes. Each domain gets its own pair of files, prefixed with the domain name, for example docs.example.com-llms.txt.

Does it work on JavaScript-heavy sites?

Only partly. It reads the HTML the server sends, so sites that render their text in the browser come out short. Server-rendered docs sites such as Docusaurus work well.

How are pages grouped into sections?

Each page goes under its first folder in the URL — /docs/… pages under ## Docs, /guides/… under ## Guides — with language and version folders such as /en/ or /v2/ skipped, and top-level pages under ## Main. Blog, news, changelog, careers, legal and similar sections are placed under ## Optional so models can skip them when context is tight.