SiteIndex

Leaves a site ready to be found by search engines and AI assistants, and ends in a measurement instead of an opinion.

Cuándo se usa

When shipping a new site, preparing a launch, or when weeks have passed and it still does not show up.

SKILL.md

# SiteIndex

Make the site discoverable: crawlers can get in, they understand what each page
is, and AI assistants can cite you.

## Rule 0: numbers get verified, never recited

Anything with a number in it expires: recommended title lengths, Core Web Vitals
thresholds, required fields for structured data and, above all, bot names. Check
them against the official source at the moment of use. If it cannot be verified,
say it is not verified. Do not invent it.

## The order, which matters

**1. The gatekeepers.** Read the robots.txt that already exists before writing
anything. Block only what is useless or duplicated, and link the sitemap.
Remember it is a sign, not a lock, and that blocking there plus a noindex tag
cancels itself out: the bot never gets to read the tag.

**2. The AI decision, which belongs to the client and not to you.** Two groups,
two different consequences. Training crawlers (GPTBot, ClaudeBot, CCBot,
Google-Extended, Bytespider) can be blocked without losing visibility. Search and
answer crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User,
PerplexityBot, Perplexity-User) remove you from AI answers if you block them. The
classic trap: Google-Extended does NOT affect your Google Search ranking or
indexing, only Gemini training.

**3. Foundations that are not negotiable.** HTTPS, a single canonical version of
the domain with a 301 from the other, content present in the mobile version, real
404s, and content visible in the served HTML rather than only after JavaScript
runs. If the site has motion, cross over to FrontLaxWeb here: scroll storytelling
is where that failure is born, because the copy ends up inside something that only
exists once JavaScript mounts.

**4. Sitemap and registration.** Only indexable, final URLs, nothing with noindex
or redirected. Submit it to Search Console and Bing Webmaster Tools.

**5. The head, page by page.** Its own title and description, a single h1, a
canonical when the same content is reachable through several URLs, and Open Graph
with an image, which is what shows when the link is pasted in a chat.

**6. Several languages.** One URL per language (/es/, /en/): that slash you see
in so many addresses is not detection, it is that they are separate pages. They
are annotated with hreflang, and the three rules that break most often are that
each version must list itself, that the links must go both ways, and that there
must be an x-default. Each language canonical points at ITSELF: if the one on
/es/ points to /en/, the Spanish version has just been removed from the index.
And no automatic redirect by language or country, which is the big one: Googlebot
crawls without an Accept-Language header and mostly from US addresses, so it
lands on the same version every time and the rest never get indexed. Detect,
suggest with a banner, let the person choose.

**7. JSON-LD.** It must describe what is VISIBLE on the page, and be validated
before it is trusted.

**8. Getting cited by AI.** Answer in the first two or three sentences under each
heading, then expand. Tables to compare, lists for processes, real FAQs. And
audit the robots.txt before blaming the content.

**9. Measure, which is where the work ends.** Search Console, PageSpeed, server
logs to see which bots actually come in, and asking the assistants your own key
questions once a month to see whether they cite you.

## The failures that break indexing most often

Before touching anything on a site that does not show up, rule these out in
order: a noindex left over from development, a site-wide Disallow inherited from
staging, a new site never registered and with no inbound links, both domain
versions live at once, a canonical pointing at another page, content that only
exists after JavaScript runs, and an automatic language redirect on a
multilingual site.

## What is out of scope

Email deliverability, local geo SEO and link building. Say so plainly instead of
improvising, and for email hand over the tools: Spamhaus and MXToolbox Email
Health, both in the resources section of this site.