Turn any website into clean Markdown for your LLM or RAG pipeline

Large language models read Markdown far better than raw HTML. A typical documentation page is 80–90 % markup, scripts and navigation; the part you actually want an LLM to see — headings, paragraphs, code blocks, tables — is buried inside it. This guide shows how to crawl a whole site and get one clean Markdown document per page, with the boilerplate removed, in a few minutes and for about $1 per 1,000 results.

What you get

Each crawled page becomes one row with the page's url, title, description, markdown (or plain text), a wordCount, the page's heading outline and its outgoing links. Here is a real row from a recorded run, trimmed for length:

{
  "url": "https://news.ycombinator.com/",
  "finalUrl": "https://news.ycombinator.com/",
  "statusCode": 200,
  "title": "Hacker News",
  "description": null,
  "lang": "en",
  "canonical": null,
  "markdown": "| 1. |  | RepRapMicron (github.com/vikolliver) |\n| --- | --- | --- |\n|  | 121 points by RobotToaster 1 hour ago \\| hide \\| 19 comments |  |\n| 2. |  | Will it be a golden age or a dark age for mathematics (and science)? (terrytao.wordpress.com) |\n|  | 24 points by smilelamp 43 …",
  "text": null,
  "wordCount": 711,
  "headings": [],
  "links": [
    "https://news.ycombinator.com/",
    "https://news.ycombinator.com/news",
    "https://news.ycombinator.com/newest"
  ],
  "depth": 0,
  "contentHash": "974809c9ad54cf36318814149195fa2101108c9e",
  "crawledAt": "2026-10-18T15:45:32.665Z",
  "error": null,
  "chunks": null,
  "changeStatus": null
}

The crawler keeps what matters for retrieval and drops what doesn't:

Step-by-step: crawl a site in the Apify Console

1. Open the tool and paste a start URL — for example the root of a docs site such as https://docs.apify.com/platform.
2. Set Max pages. This is the billed unit and the crawl stops at exactly that number, so a 200-page crawl can never cost more than 200 results.
3. Leave Use sitemap on. The crawler reads /sitemap.xml (including nested sitemap indexes) and starts from real content pages instead of discovering them link by link.
4. Optional: add Include path prefixes such as /docs to stay inside one section of a large site.
5. Click Start. A 50-page docs crawl usually finishes in under a minute; download the results as JSON, CSV or Excel, or read them through the API.

Run Website to Markdown — Content Crawler for LLM & RAG on Apify →

Call it from code

The same crawl from Python, JavaScript or the command line. Replace the token with your own from the Apify Console (Settings → API & Integrations).

import json
import os
from apify_client import ApifyClient
client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("webdatatools/website-to-markdown").call(run_input=json.loads("{\n    "startUrls": [\n        {\n            "url": "https://docs.apify.com/platform\"\n        },\n        {\n            "url": "https://crawlee.dev/docs\"\n        }\n    ],\n    "maxPages": 50,\n    "maxDepth": 3\n}"))
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('webdatatools/website-to-markdown').call({
    "startUrls": [
        {
            "url": "https://docs.apify.com/platform"
        },
        {
            "url": "https://crawlee.dev/docs"
        }
    ],
    "maxPages": 50,
    "maxDepth": 3
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

curl -X POST "https://api.apify.com/v2/acts/webdatatools~website-to-markdown/run-sync-get-dataset-items" 
-H "Authorization: Bearer $APIFY_TOKEN" 
-H "Content-Type: application/json" 
-d '{"startUrls":[{"url":"https://docs.apify.com/platform"},{"url":"https://crawlee.dev/docs"}],"maxPages":50,"maxDepth":3}'

Getting RAG-ready chunks in the same run

Most pipelines split each page into chunks before embedding. Turn on Add RAG chunks (chunkForRag: true) and every row gets a chunks array: heading-aware pieces of the Markdown, each with its token count and the heading path it sits under (["Platform", "Actors", "Running"]). Chunk size is set with chunkSize (500 tokens by default). Because chunks never cross a heading, a retrieved chunk always comes with the context of the section it belongs to — which measurably improves answer quality compared with fixed-length splitting.

Keeping a knowledge base fresh

Re-crawling a 2,000-page site every day just to find the ten pages that changed is wasteful. With Change detection on, the crawler remembers each page's content hash from the previous run with the same start URLs and marks every row new, changed or unchanged. Unchanged pages can come back without their body text (skipUnchangedContent), so your pipeline only re-embeds what actually changed. Schedule the run daily in Apify and connect a webhook to your ingestion job.

How much does it cost?

Primary event rates from a catalog snapshot; this is not the total price of a run. Additional charge events and Actor Start fees may apply; start fees can scale with memory. Check the Actor's Pricing tab on Apify for all events, conditions and current plan-specific rates, and set a maximum run budget before starting.

A practical example: a 300-page documentation site costs 300 results on the free plan price above. There is no separate charge for compute, bandwidth or proxies — those are included in the per-result price.

How it compares

Rule of thumb: if you can see the content with JavaScript disabled in your browser, this HTTP crawler will get it — at a fraction of browser cost.

FAQ

Can it crawl sites that need JavaScript?

No. It fetches raw HTML without running page scripts. Most documentation, blogs and help centers render on the server and work well; client-side single-page apps need a browser-based crawler instead.

Does it respect robots.txt?

Yes, by default. URLs disallowed by the site's robots.txt are skipped. You can turn this off only for sites you own or have permission to crawl.

How do I crawl only part of a site?

Use Include path prefixes (for example /docs or /blog) and Exclude patterns for things like /login or file downloads. Max depth limits how many links deep the crawl goes from the start URL.

What happens if my run hits its timeout?

The crawler stops shortly before the run timeout and finishes successfully with every page crawled so far. You are only billed for pages that were delivered.

Can an AI agent call it directly?

Yes. Every Apify tool is available through the Apify MCP server, so assistants such as Claude or ChatGPT with MCP support can run the crawl and read the Markdown themselves.