# tatzel.io — a personal site and a living résumé. Machines are welcome here.
#
# POSTURE, set 2026-09-16: AI-FORWARD, WITH ONE NARROW RESERVATION.
#
# Everything is open to everyone, with a single exception: the two long-form
# evidence guides under /guides/ are reserved from TRAINING crawlers. They stay
# fully open to humans, to search engines, and to every retrieval agent that
# cites its sources — because a guide quoted with a link is worth something to
# its author, and a guide absorbed silently is not.
#
# The reasoning, so nobody has to re-derive it: the portfolio content on this
# site is NAME-LINKED FACT. A model that trains on it can later answer "who is
# David Tatzel?" and attach nine patents and seven shipped apps to the name.
# That is the whole point of the site, so it is open to training crawlers too.
# The guides are UNATTRIBUTED SUBSTANCE. A model that trains on them learns the
# rankings and will never name the source, so training-inclusion there returns
# nothing while costing the only content on this site with standalone value.
# Same site, opposite economics — hence one line of difference.
#
# This is a reservation, not a wall. See https://tatzel.io/ai-policy and the
# machine-readable form at /.well-known/tdmrep.json (W3C TDM Reservation
# Protocol), which is the instrument for "you may read this, you may not mine
# it" and is recognised under EU DSM Article 4.
#
# ⚠️ robots.txt IS ADVISORY AND IT IS NOT THE ONLY GATE. Cloudflare's edge sits
# in front of it and wins. On 2026-09-16 this zone 403'd nine training crawlers
# for a period while this file said "Allow: /" -- the crawler read permission and
# was then refused. Cause: MANAGED ruleset 3e677e63d4e9479382576f3fa66279e7, rule
# 7bd01eeccb6b420fa0be30264603a5cb "Manage AI bots", blocking category AI Crawler.
# It appears in NO settings list, and "Configure AI bot policies" RENDERED Allow
# while storing Block. Only Security > Analytics > Events named it. If the edge
# and this file ever disagree again, read the EVENT LOG first.
#
# Shape enforced by scripts/check_robots_policy.py, which evaluates this file
# with a real robots parser rather than by grepping it, and which fails if a
# search or retrieval agent ever loses access to /guides/.
#
# Structured summary for language models: https://tatzel.io/llms.txt

User-agent: *
Allow: /

# ⚠️ GROUP ORDER IS LOAD-BEARING — TRAINING FIRST, ON PURPOSE.
# The spec says the most SPECIFIC user-agent group wins, but several parsers in
# the wild (CPython's urllib.robotparser among them) take the FIRST group whose
# declared token is a substring of the requesting agent. "Applebot" is a
# substring of "Applebot-Extended", so with the retrieval group first,
# Applebot-Extended would match Applebot's group and be granted the guides --
# silently, and in the one direction that loses the reservation. Training first
# inverts that safely: no training token is a substring of a retrieval agent's
# name, but the reverse is true, so this order is correct under BOTH matching
# rules. scripts/check_robots_policy.py asserts exactly this pair.

# ---------------------------------------------------------------------------
# Training and corpus crawlers — welcome on everything EXCEPT /guides/.
# Google-Extended and Applebot-Extended are training-use signals rather than
# crawlers; naming them here is how you reserve content they have already
# fetched, so they belong in this group and not with Googlebot/Applebot above.
# ---------------------------------------------------------------------------
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: CCBot
User-agent: cohere-ai
User-agent: Amazonbot
User-agent: Bytespider
User-agent: PetalBot
User-agent: Diffbot
User-agent: omgili
User-agent: omgilibot
User-agent: Timpibot
User-agent: meta-externalagent
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Google-CloudVertexBot
User-agent: ProRataInc
User-agent: Novellum AI Crawl
User-agent: Cloudflare Crawler
Allow: /
Disallow: /guides/

# ---------------------------------------------------------------------------
# Search, retrieval and answer engines — FULL access, guides included.
# These cite and link, which is an exchange rather than a taking.
# ---------------------------------------------------------------------------
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
User-agent: DuckDuckBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: DuckAssistBot
User-agent: MistralAI-User
User-agent: Meta-ExternalFetcher
User-agent: Manus Bot
Allow: /

# ---------------------------------------------------------------------------
# Link-preview and social unfurlers — FULL access. A shared guide must still
# render its card, so these are never restricted anywhere.
# ---------------------------------------------------------------------------
User-agent: facebookexternalhit
User-agent: FacebookBot
User-agent: Twitterbot
User-agent: LinkedInBot
User-agent: Slackbot
User-agent: Slackbot-LinkExpanding
User-agent: Discordbot
User-agent: WhatsApp
User-agent: TelegramBot
Allow: /

# ---------------------------------------------------------------------------
# Archivers — FULL access. A living résumé benefits from a public record of
# what it said and when.
# ---------------------------------------------------------------------------
User-agent: archive.org_bot
User-agent: ia_archiver
User-agent: Arquivo-web-crawler
Allow: /

Sitemap: https://tatzel.io/sitemap.xml