How to convert a web page to clean Markdown for LLMs and RAG

The minimal request

You need an API key (created once by a person at api.relayted.de/account) and a small balance. Then one call creates the job and waits for it, and a second call downloads the Markdown:

export RELAYTED_KEY="RAPI-XXXX-XXXX-XXXX-XXXX-XXXX"

# 1. create the job and wait up to 20 seconds for it
curl -s "https://api.relayted.de/v1/websitepro/jobs?wait=20" \
  -H "Authorization: Bearer $RELAYTED_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com/article","outputs":["markdown"]}' > job.json

# 2. download the Markdown result
JOB=$(jq -r '.job.id' job.json)
curl -s "https://api.relayted.de/v1/jobs/$JOB/results/markdown" \
  -H "Authorization: Bearer $RELAYTED_KEY" -o article.md

The response of step 1 contains a job object with status, cost, charged and a results array, plus your new balance. If job.status is still queued or processing when the wait time is over, ask again with GET /v1/jobs/<id>?wait=20 until it is completed or failed. The longest wait the API honours per call is 25 seconds.

Want to know the price first? POST /v1/websitepro/quote takes the same body, costs nothing and returns cost, the price of every item and whether your balance is sufficient.

Why Markdown and not raw HTML

Raw HTML is mostly markup, scripts, navigation and tracking code. Fed into a language model it wastes context and adds noise to embeddings. The markdown output keeps the main content of the page: it is extracted with a readability step, navigation and boilerplate are removed, and headings, lists, links and code stay as Markdown. That is the format most chunkers and RAG pipelines expect anyway.

How the page is fetched

The service first fetches the page over plain HTTP. If that returns too little content, for example on JavaScript-heavy pages or when plain requests are blocked, a real Chromium browser renders the page. A job usually takes 1 to 10 seconds, and up to about 40 seconds when the browser is needed. The page is fetched once, even if you ask for several outputs.

Which output for which job

One job can ask for several outputs of the same page. Each output is one result and is priced on its own:

10,000 Payloads are 1 EUR. The live numbers are always in /v1/pricing; the table above is the state when this article was published.

For RAG, markdown plus metadata is a good pair: the Markdown goes into your chunks, the metadata gives you the title, canonical URL and language for citations and filters. Together that is 65 Payloads per page.

A small ingestion script in Python

This script uses only the Python standard library. It converts a list of URLs and writes one Markdown file per page. It handles the two answers you will meet in practice: a job that needs a little longer, and an empty balance.

import json, os, re, time, urllib.request, urllib.error

API = "https://api.relayted.de/v1"
KEY = os.environ["RELAYTED_KEY"]
URLS = [
    "https://example.com/docs/getting-started",
    "https://example.com/docs/configuration",
]

def call(method, path, body=None):
    req = urllib.request.Request(API + path, method=method,
        data=json.dumps(body).encode() if body is not None else None,
        headers={"Authorization": "Bearer " + KEY, "Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=60) as r:
        return r.read()

for url in URLS:
    try:
        job = json.loads(call("POST", "/websitepro/jobs?wait=20",
                              {"url": url, "outputs": ["markdown"]}))["job"]
    except urllib.error.HTTPError as e:
        err = json.loads(e.read() or b"{}").get("error", {})
        print("skipped", url, e.code, err.get("code"))
        if e.code == 402:      # insufficient_funds: top up, then run again
            break
        continue
    while job["status"] in ("queued", "processing"):
        time.sleep(1)
        job = json.loads(call("GET", "/jobs/%s?wait=20" % job["id"]))["job"]
    done = [r for r in job["results"] if r["status"] == "done"]
    if not done:
        print("not delivered (refunded):", url)
        continue
    md = call("GET", "/jobs/%s/results/markdown" % job["id"]).decode("utf-8")
    name = re.sub(r"[^a-z0-9]+", "-", url.lower()).strip("-")[:80] + ".md"
    with open(name, "w", encoding="utf-8") as f:
        f.write(md)
    print("saved", name, "charged", job["charged"])

What it costs in practice

• One page as Markdown: 45 Payloads, that is 0.45 cent.
• 1 EUR buys about 222 Markdown pages.
• Refreshing 50 documentation pages every night: 50 × 45 = 2,250 Payloads per day, about 67,500 Payloads or 6.75 EUR in 30 days.

There is no base fee and no minimum volume. Before a large run, a free quote tells you the exact price.

Limits you should know

• Only public targets: the URL must be http:// or https://, URLs with credentials in them are rejected (bad_url), and private, local or reserved addresses answer blocked_address.
• Results are kept for 24 hours, then deleted. After that the result URL answers 410 expired.
• Every output that is not delivered (for example timeout, not_html, empty or too_large) is refunded automatically. Rejected requests cost nothing.
• 600 jobs per hour and key; request bodies up to 64 KB. A 503 busy means nothing was charged; retry shortly.

Without code

The same conversion is available as a web interface at websitepro.relayted.de, without an account and paid per use. The API is the better fit as soon as a script, a scheduled job or an agent does the work.