--- name: optimise-seo description: Implements Next.js crawlability, metadata, canonicals, status codes, JSON-LD, hreflang, and crawler policy with served-page evidence. Use when asked to "fix SEO", "add a sitemap", "fix soft 404s", or "add llms.txt". For demand research, briefs, or Search Console monitoring use seo-program. ---

# Optimise SEO

- **IS:** crawlability, metadata, structured data, canonicals, redirects, status codes, hreflang, AI-crawler policy and extractable page structure, Core Web Vitals, programmatic SEO, security and privacy headers, and error-page behaviour, implemented in a Next.js App Router codebase. - **IS NOT:** visual redesign, layout, or page-level UI quality (`ui-design` Direction, Build, and Audit modes), demand research, briefs, question maps, or Search Console monitoring (`seo-program`), tenant routing for `robots.txt` and `sitemap.xml` (`multi-tenant-architecture`), or writing the article (external `ghostwriter`).

Allowed surface: `metadata` and `generateMetadata`, JSON-LD, semantic HTML, internal links, alt text, `app/sitemap.ts`, `app/robots.ts`, `app/feed.xml/route.ts`, `app/llms.txt/route.ts`, `proxy.ts`, `next.config.ts` redirects and headers, error pages, and performance work. Component styling and layout stay as they are; when a fix needs a layout change, name it as a handoff to `ui-design`.

Before writing code, read the matching guide in `node_modules/next/dist/docs/` (resolved from the app directory; in a monorepo the `next` package is not visible from the repo root). The bundled docs match the installed version; the notes in `references/nextjs-implementation.md` were checked against 16.3 and say where behaviour changed.

## References

| File | Read when | |---|---| | [references/indexing-policy.md](references/indexing-policy.md) | Redirects, duplicate URLs, index decisions, or pages generated at scale | | [references/nextjs-implementation.md](references/nextjs-implementation.md) | Before writing code in steps 2-4: metadata, sitemap and robots, redirects, indexing, streaming 404s, JSON-LD, OG images, CSP | | [references/answer-engines.md](references/answer-engines.md) | The task touches AI crawlers, `robots.ts` rules for GPTBot or ClaudeBot, `llms.txt`, Content-Signal, or "why are we not cited" | | [references/internationalisation.md](references/internationalisation.md) | The site has more than one locale | | [references/technical-hardening.md](references/technical-hardening.md) | Headers, cookies, consent, GPC, `security.txt`, or maintenance and error-status tasks | | [references/validation-evidence.md](references/validation-evidence.md) | Step 5: served-page commands and evidence expectations | | [references/seo-checklist.md](references/seo-checklist.md) | Step 5: copy into the report and mark each line with evidence |

## Workflow

Copy and track this checklist:

```text SEO progress: - [ ] Step 1: Inventory routes and decide index intent per route - [ ] Step 2: Fix crawl/index foundations (sitemap, robots, canonicals, redirects, status codes) - [ ] Step 3: Implement metadata and structured data - [ ] Step 4: Improve semantics, extractable answers, internal links, and Core Web Vitals - [ ] Step 5: Validate with references/seo-checklist.md and report evidence ```

Step 1 is a table: route, index or noindex, canonical, reason. Every later step reads from it, and a route with no row gets no work.

## Must-have on every site

- `app/sitemap.ts` lists every indexable URL with `lastModified` derived from content; `app/robots.ts` names it - One canonical per page, one host, one casing, one trailing-slash policy, set through `alternates.canonical` with `metadataBase` in the root layout. Next.js enforces the slash policy itself: by default it 308s `/about/` to `/about`, and `trailingSlash: true` inverts that (files with extensions and `.well-known/` paths are exempt either way) - Preserve the property's existing ownership verification. Use `metadata.verification` only for a supplied token and the chosen verification method; DNS verification needs no duplicate meta tag - Root layout `robots` lifts Google's default preview caps: `googleBot: { 'max-snippet': -1, 'max-image-preview': 'large', 'max-video-preview': -1 }`. Without them Google truncates the snippet and the image preview, and the snippet cap is what AI surfaces read against when deciding how much of a page they may quote - `authors` and `creator` set explicitly in the root layout, so person-level attribution does not live only in footer HTML and JSON-LD - Unique title and description per page; title and H1 lead with the non-brand primary keyword and agree in intent (a brand-led title on a category page competes only for navigational queries the site already wins) - Every page opens with a short, plain-text answer to the question it exists to answer; the h2s are the questions people actually ask - Open Graph and Twitter Card tags with a 1200x630 image - JSON-LD: Organization and WebSite once in a root `@graph` with stable `@id`s, BreadcrumbList on inner pages, then Article, Product, ProfilePage, or LocalBusiness where the content type matches. Define an entity once and reference it by `@id` - One h1, logical h2-h6, descriptive alt text, internal links between related pages - Core Web Vitals in target at the p75 of CrUX field data: LCP 2.5 s, INP 200 ms, CLS 0.1 (INP replaced FID in March 2024; a guide still naming FID is stale)

## Redirects and indexing policy

Load `references/indexing-policy.md` when routes move, duplicates need consolidation, or page indexing policy changes. Verify behavior against the installed Next.js version and served responses.

## AI crawlers and answer engines

Google needs no special files: AI Overviews and AI Mode run on Googlebot and core ranking, and `nosnippet` or `max-snippet:0` is the only opt-out (it removes the normal snippet too). Everything else is a robots decision per crawler class: search and citation bots (`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`) put a site in answers; training bots (`GPTBot`, `ClaudeBot`, `CCBot`) do not; `Google-Extended` and `Applebot-Extended` are usage tokens with no crawler behind them. `llms.txt` costs one route and earns almost nothing measurable. The full table, `robots.ts` rules, Content-Signal, and the Markdown-alternate patterns are in `references/answer-engines.md`.

## Programmatic SEO

For pages generated at scale, load `references/indexing-policy.md`. Require demand evidence and unique page value before indexing a new pattern.

## Audit triage order

1. Crawl and index: robots, sitemap, stray `noindex`, canonicals, redirect chains, soft 404s 2. Technical: HTTPS, Core Web Vitals field data, mobile and desktop parity, AI-crawler access through the CDN 3. On-page: title and H1 uniqueness and non-brand lead, extractable answer, internal links, thin pages 4. Orphans: indexable pages with no internal link or sitemap entry; give them a crawl path or `noindex` and drop them. A sitewide footer link is not a crawl path on a low-authority domain: Google discounts footer boilerplate, and a section linked only from the footer can sit at "URL is unknown to Google" for months while nav-linked pages beside it are crawled daily

Report in the same order: what blocks indexation first, then on-page and schema gaps, then the single next fix. Twenty equal-weight findings get none of them done. Report length follows what was found, not the section list. Treat third-party audit tools as leads, not findings: several flag Next.js output falsely (a missing `<meta name="title">` when `<title>` is present, "duplicate" `theme-color` tags that are light and dark variants, repeated `article:tag`), and at least one agent-skill auditor silently skips modules unless given an explicit module list and passes a `FAQPage` whose answers never render. Verify each lead against the served HTML before it goes in the report.

## Gotchas

- Under `cacheComponents: true` (the `scaffold-nextjs` default) the `dynamic`, `dynamicParams`, `revalidate`, and `fetchCache` segment configs are build errors: `export const revalidate` on `app/sitemap.ts`, `dynamic = 'force-static'` on a `robots.txt/route.ts` handler, `fetchCache` anywhere (`runtime`, `maxDuration`, `instant`, and `prefetch` remain valid). Move the content fetch into a separate `'use cache'` helper with `cacheLife('hours')` and `cacheTag`, called from the page or `GET` (the directive cannot sit on the `GET` export itself), and `revalidateTag(tag, 'max')` from the publish webhook. Revalidation does not purge a CDN in front of the app; purge it in the same webhook. - `generateSitemaps()` writes `/product/sitemap/0.xml`, `/product/sitemap/1.xml`, and no index file. List each generated file in `robots.ts` (`sitemap` takes an array) or hand-write the index at `app/sitemap.xml/route.ts`; otherwise Google sees only the ones it stumbles on. - `lastModified: new Date()` on every sitemap row marks the whole site as changed on every deploy; Google then ignores `lastmod` sitewide. `priority` and `changeFrequency` are ignored outright, so emitting them buys nothing and clutters the diff. - No `Host:` line in `robots.txt`. Yandex dropped the directive in 2018 in favour of a 301 to the preferred host, which `next.config.ts` already serves, and auditors flag the leftover line as an unknown directive. Leave `host` out of the `robots.ts` return value. - Next.js replaces object-valued metadata (`openGraph`, `twitter`, `robots`, `alternates`) wholesale when a child segment declares the key; it does not deep-merge. A page that adds `openGraph: { title }` loses the inherited `siteName`, `images`, and `locale`, and `images: undefined` counts as a declaration. When you add one of these blocks to a route, every field the parent supplied becomes yours to restate. Scalars (`title`, `description`, `authors`, `creator`) merge normally. - `title.template` applies to `<title>` only. `og:title` resolves against `openGraph.title.template`, a separate mechanism, so stripping a hand-rolled suffix from a page title silently strips the product name from the share card. Set both templates in the layout, or neither. A bare string `title` in a nested layout also resets the parent template to null for that subtree; return `{ default, template }` instead. - Never set `openGraph.url` in a root layout. It is not per-page, every route inherits it, and every share collapses onto the home URL. A child cannot fix it without declaring `openGraph` and losing the rest of the block. Absent beats wrong: consumers fall back to the URL they fetched, and `alternates.canonical` is already per-page. - `middleware.ts` is deprecated in Next.js 16. New request-time logic (slug checks, CSP nonces, `Accept` negotiation) goes in `proxy.ts` with an exported `proxy` function; `npx @next/codemod@canary middleware-to-proxy .` moves an existing file. Proxy is Node.js only; `runtime` is not a valid export there. - `next.config.ts` redirects match in array order and `:path*` swallows everything beneath it, so a specific source placed after a catch-all is dead code, and a bare `/old` must sit above `/old/:path*` or it lands on `/new/`. Vercel caps the list at 1,024 rules; past that, move the map into `proxy.ts`. - Next owns the `Vary` header on App Router HTML responses and replaces it with its RSC list (`rsc, next-router-state-tree, ...`). A `Vary: Accept` set in `proxy.ts` on `NextResponse.next()`, or in `headers()`, never reaches the client. It survives only on a `Response` you construct yourself, so content-negotiated Markdown must be returned directly, not rewritten to. - `notFound()` after a Suspense boundary or `loading.tsx` has flushed returns 200 with a `noindex` meta, not 404. Without Cache Components, call it before the first suspending `await`; with Cache Components the docs are explicit that every dynamic route streams a static shell first, so the existence check moves to `proxy.ts`. Analytics and uptime monitors otherwise see a health

--- name: optimise-seo description: Implements Next.js crawlability, metadata, canonicals, status codes, JSON-LD, hreflang, and crawler policy with served-page evidence. Use when asked to "fix SEO", "add a sitemap", "fix soft 404s", or "add llms.txt". For demand research, briefs, or Search Console monitoring use seo-program. ---

# Optimise SEO

- **IS:** crawlability, metadata, structured data, canonicals, redirects, status codes, hreflang, AI-crawler policy and extractable page structure, Core Web Vitals, programmatic SEO, security and privacy headers, and error-page behaviour, implemented in a Next.js App Router codebase. - **IS NOT:** visual redesign, layout, or page-level UI quality (`ui-design` Direction, Build, and Audit modes), demand research, briefs, question maps, or Search Console monitoring (`seo-program`), tenant routing for `robots.txt` and `sitemap.xml` (`multi-tenant-architecture`), or writing the article (external `ghostwriter`).

Allowed surface: `metadata` and `generateMetadata`, JSON-LD, semantic HTML, internal links, alt text, `app/sitemap.ts`, `app/robots.ts`, `app/feed.xml/route.ts`, `app/llms.txt/route.ts`, `proxy.ts`, `next.config.ts` redirects and headers, error pages, and performance work. Component styling and layout stay as they are; when a fix needs a layout change, name it as a handoff to `ui-design`.

Before writing code, read the matching guide in `node_modules/next/dist/docs/` (resolved from the app directory; in a monorepo the `next` package is not visible from the repo root). The bundled docs match the installed version; the notes in `references/nextjs-implementation.md` were checked against 16.3 and say where behaviour changed.

## References

| File | Read when | |---|---| | [references/indexing-policy.md](references/indexing-policy.md) | Redirects, duplicate URLs, index decisions, or pages generated at scale | | [references/nextjs-implementation.md](references/nextjs-implementation.md) | Before writing code in steps 2-4: metadata, sitemap and robots, redirects, indexing, streaming 404s, JSON-LD, OG images, CSP | | [references/answer-engines.md](references/answer-engines.md) | The task touches AI crawlers, `robots.ts` rules for GPTBot or ClaudeBot, `llms.txt`, Content-Signal, or "why are we not cited" | | [references/internationalisation.md](references/internationalisation.md) | The site has more than one locale | | [references/technical-hardening.md](references/technical-hardening.md) | Headers, cookies, consent, GPC, `security.txt`, or maintenance and error-status tasks | | [references/validation-evidence.md](references/validation-evidence.md) | Step 5: served-page commands and evidence expectations | | [references/seo-checklist.md](references/seo-checklist.md) | Step 5: copy into the report and mark each line with evidence |

## Workflow

Copy and track this checklist:

```text SEO progress: - [ ] Step 1: Inventory routes and decide index intent per route - [ ] Step 2: Fix crawl/index foundations (sitemap, robots, canonicals, redirects, status codes) - [ ] Step 3: Implement metadata and structured data - [ ] Step 4: Improve semantics, extractable answers, internal links, and Core Web Vitals - [ ] Step 5: Validate with references/seo-checklist.md and report evidence ```

Step 1 is a table: route, index or noindex, canonical, reason. Every later step reads from it, and a route with no row gets no work.

## Must-have on every site

- `app/sitemap.ts` lists every indexable URL with `lastModified` derived from content; `app/robots.ts` names it - One canonical per page, one host, one casing, one trailing-slash policy, set through `alternates.canonical` with `metadataBase` in the root layout. Next.js enforces the slash policy itself: by default it 308s `/about/` to `/about`, and `trailingSlash: true` inverts that (files with extensions and `.well-known/` paths are exempt either way) - Preserve the property's existing ownership verification. Use `metadata.verification` only for a supplied token and the chosen verification method; DNS verification needs no duplicate meta tag - Root layout `robots` lifts Google's default preview caps: `googleBot: { 'max-snippet': -1, 'max-image-preview': 'large', 'max-video-preview': -1 }`. Without them Google truncates the snippet and the image preview, and the snippet cap is what AI surfaces read against when deciding how much of a page they may quote - `authors` and `creator` set explicitly in the root layout, so person-level attribution does not live only in footer HTML and JSON-LD - Unique title and description per page; title and H1 lead with the non-brand primary keyword and agree in intent (a brand-led title on a category page competes only for navigational queries the site already wins) - Every page opens with a short, plain-text answer to the question it exists to answer; the h2s are the questions people actually ask - Open Graph and Twitter Card tags with a 1200x630 image - JSON-LD: Organization and WebSite once in a root `@graph` with stable `@id`s, BreadcrumbList on inner pages, then Article, Product, ProfilePage, or LocalBusiness where the content type matches. Define an entity once and reference it by `@id` - One h1, logical h2-h6, descriptive alt text, internal links between related pages - Core Web Vitals in target at the p75 of CrUX field data: LCP 2.5 s, INP 200 ms, CLS 0.1 (INP replaced FID in March 2024; a guide still naming FID is stale)

## Redirects and indexing policy

Load `references/indexing-policy.md` when routes move, duplicates need consolidation, or page indexing policy changes. Verify behavior against the installed Next.js version and served responses.

## AI crawlers and answer engines

Google needs no special files: AI Overviews and AI Mode run on Googlebot and core ranking, and `nosnippet` or `max-snippet:0` is the only opt-out (it removes the normal snippet too). Everything else is a robots decision per crawler class: search and citation bots (`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`) put a site in answers; training bots (`GPTBot`, `ClaudeBot`, `CCBot`) do not; `Google-Extended` and `Applebot-Extended` are usage tokens with no crawler behind them. `llms.txt` costs one route and earns almost nothing measurable. The full table, `robots.ts` rules, Content-Signal, and the Markdown-alternate patterns are in `references/answer-engines.md`.

## Programmatic SEO

For pages generated at scale, load `references/indexing-policy.md`. Require demand evidence and unique page value before indexing a new pattern.

## Audit triage order

1. Crawl and index: robots, sitemap, stray `noindex`, canonicals, redirect chains, soft 404s 2. Technical: HTTPS, Core Web Vitals field data, mobile and desktop parity, AI-crawler access through the CDN 3. On-page: title and H1 uniqueness and non-brand lead, extractable answer, internal links, thin pages 4. Orphans: indexable pages with no internal link or sitemap entry; give them a crawl path or `noindex` and drop them. A sitewide footer link is not a crawl path on a low-authority domain: Google discounts footer boilerplate, and a section linked only from the footer can sit at "URL is unknown to Google" for months while nav-linked pages beside it are crawled daily

Report in the same order: what blocks indexation first, then on-page and schema gaps, then the single next fix. Twenty equal-weight findings get none of them done. Report length follows what was found, not the section list. Treat third-party audit tools as leads, not findings: several flag Next.js output falsely (a missing `<meta name="title">` when `<title>` is present, "duplicate" `theme-color` tags that are light and dark variants, repeated `article:tag`), and at least one agent-skill auditor silently skips modules unless given an explicit module list and passes a `FAQPage` whose answers never render. Verify each lead against the served HTML before it goes in the report.

## Gotchas

- Under `cacheComponents: true` (the `scaffold-nextjs` default) the `dynamic`, `dynamicParams`, `revalidate`, and `fetchCache` segment configs are build errors: `export const revalidate` on `app/sitemap.ts`, `dynamic = 'force-static'` on a `robots.txt/route.ts` handler, `fetchCache` anywhere (`runtime`, `maxDuration`, `instant`, and `prefetch` remain valid). Move the content fetch into a separate `'use cache'` helper with `cacheLife('hours')` and `cacheTag`, called from the page or `GET` (the directive cannot sit on the `GET` export itself), and `revalidateTag(tag, 'max')` from the publish webhook. Revalidation does not purge a CDN in front of the app; purge it in the same webhook. - `generateSitemaps()` writes `/product/sitemap/0.xml`, `/product/sitemap/1.xml`, and no index file. List each generated file in `robots.ts` (`sitemap` takes an array) or hand-write the index at `app/sitemap.xml/route.ts`; otherwise Google sees only the ones it stumbles on. - `lastModified: new Date()` on every sitemap row marks the whole site as changed on every deploy; Google then ignores `lastmod` sitewide. `priority` and `changeFrequency` are ignored outright, so emitting them buys nothing and clutters the diff. - No `Host:` line in `robots.txt`. Yandex dropped the directive in 2018 in favour of a 301 to the preferred host, which `next.config.ts` already serves, and auditors flag the leftover line as an unknown directive. Leave `host` out of the `robots.ts` return value. - Next.js replaces object-valued metadata (`openGraph`, `twitter`, `robots`, `alternates`) wholesale when a child segment declares the key; it does not deep-merge. A page that adds `openGraph: { title }` loses the inherited `siteName`, `images`, and `locale`, and `images: undefined` counts as a declaration. When you add one of these blocks to a route, every field the parent supplied becomes yours to restate. Scalars (`title`, `description`, `authors`, `creator`) merge normally. - `title.template` applies to `<title>` only. `og:title` resolves against `openGraph.title.template`, a separate mechanism, so stripping a hand-rolled suffix from a page title silently strips the product name from the share card. Set both templates in the layout, or neither. A bare string `title` in a nested layout also resets the parent template to null for that subtree; return `{ default, template }` instead. - Never set `openGraph.url` in a root layout. It is not per-page, every route inherits it, and every share collapses onto the home URL. A child cannot fix it without declaring `openGraph` and losing the rest of the block. Absent beats wrong: consumers fall back to the URL they fetched, and `alternates.canonical` is already per-page. - `middleware.ts` is deprecated in Next.js 16. New request-time logic (slug checks, CSP nonces, `Accept` negotiation) goes in `proxy.ts` with an exported `proxy` function; `npx @next/codemod@canary middleware-to-proxy .` moves an existing file. Proxy is Node.js only; `runtime` is not a valid export there. - `next.config.ts` redirects match in array order and `:path*` swallows everything beneath it, so a specific source placed after a catch-all is dead code, and a bare `/old` must sit above `/old/:path*` or it lands on `/new/`. Vercel caps the list at 1,024 rules; past that, move the map into `proxy.ts`. - Next owns the `Vary` header on App Router HTML responses and replaces it with its RSC list (`rsc, next-router-state-tree, ...`). A `Vary: Accept` set in `proxy.ts` on `NextResponse.next()`, or in `headers()`, never reaches the client. It survives only on a `Response` you construct yourself, so content-negotiated Markdown must be returned directly, not rewritten to. - `notFound()` after a Suspense boundary or `loading.tsx` has flushed returns 200 with a `noindex` meta, not 404. Without Cache Components, call it before the first suspending `await`; with Cache Components the docs are explicit that every dynamic route streams a static shell first, so the existence check moves to `proxy.ts`. Analytics and uptime monitors otherwise see a health

--- name: optimise-seo description: Implements Next.js crawlability, metadata, canonicals, status codes, JSON-LD, hreflang, and crawler policy with served-page evidence. Use when asked to "fix SEO", "add a sitemap", "fix soft 404s", or "add llms.txt". For demand research, briefs, or Search Console monitoring use seo-program. ---

# Optimise SEO

- **IS:** crawlability, metadata, structured data, canonicals, redirects, status codes, hreflang, AI-crawler policy and extractable page structure, Core Web Vitals, programmatic SEO, security and privacy headers, and error-page behaviour, implemented in a Next.js App Router codebase. - **IS NOT:** visual redesign, layout, or page-level UI quality (`ui-design` Direction, Build, and Audit modes), demand research, briefs, question maps, or Search Console monitoring (`seo-program`), tenant routing for `robots.txt` and `sitemap.xml` (`multi-tenant-architecture`), or writing the article (external `ghostwriter`).

Allowed surface: `metadata` and `generateMetadata`, JSON-LD, semantic HTML, internal links, alt text, `app/sitemap.ts`, `app/robots.ts`, `app/feed.xml/route.ts`, `app/llms.txt/route.ts`, `proxy.ts`, `next.config.ts` redirects and headers, error pages, and performance work. Component styling and layout stay as they are; when a fix needs a layout change, name it as a handoff to `ui-design`.

Before writing code, read the matching guide in `node_modules/next/dist/docs/` (resolved from the app directory; in a monorepo the `next` package is not visible from the repo root). The bundled docs match the installed version; the notes in `references/nextjs-implementation.md` were checked against 16.3 and say where behaviour changed.

## References

| File | Read when | |---|---| | [references/indexing-policy.md](references/indexing-policy.md) | Redirects, duplicate URLs, index decisions, or pages generated at scale | | [references/nextjs-implementation.md](references/nextjs-implementation.md) | Before writing code in steps 2-4: metadata, sitemap and robots, redirects, indexing, streaming 404s, JSON-LD, OG images, CSP | | [references/answer-engines.md](references/answer-engines.md) | The task touches AI crawlers, `robots.ts` rules for GPTBot or ClaudeBot, `llms.txt`, Content-Signal, or "why are we not cited" | | [references/internationalisation.md](references/internationalisation.md) | The site has more than one locale | | [references/technical-hardening.md](references/technical-hardening.md) | Headers, cookies, consent, GPC, `security.txt`, or maintenance and error-status tasks | | [references/validation-evidence.md](references/validation-evidence.md) | Step 5: served-page commands and evidence expectations | | [references/seo-checklist.md](references/seo-checklist.md) | Step 5: copy into the report and mark each line with evidence |

## Workflow

Copy and track this checklist:

```text SEO progress: - [ ] Step 1: Inventory routes and decide index intent per route - [ ] Step 2: Fix crawl/index foundations (sitemap, robots, canonicals, redirects, status codes) - [ ] Step 3: Implement metadata and structured data - [ ] Step 4: Improve semantics, extractable answers, internal links, and Core Web Vitals - [ ] Step 5: Validate with references/seo-checklist.md and report evidence ```

Step 1 is a table: route, index or noindex, canonical, reason. Every later step reads from it, and a route with no row gets no work.

## Must-have on every site

- `app/sitemap.ts` lists every indexable URL with `lastModified` derived from content; `app/robots.ts` names it - One canonical per page, one host, one casing, one trailing-slash policy, set through `alternates.canonical` with `metadataBase` in the root layout. Next.js enforces the slash policy itself: by default it 308s `/about/` to `/about`, and `trailingSlash: true` inverts that (files with extensions and `.well-known/` paths are exempt either way) - Preserve the property's existing ownership verification. Use `metadata.verification` only for a supplied token and the chosen verification method; DNS verification needs no duplicate meta tag - Root layout `robots` lifts Google's default preview caps: `googleBot: { 'max-snippet': -1, 'max-image-preview': 'large', 'max-video-preview': -1 }`. Without them Google truncates the snippet and the image preview, and the snippet cap is what AI surfaces read against when deciding how much of a page they may quote - `authors` and `creator` set explicitly in the root layout, so person-level attribution does not live only in footer HTML and JSON-LD - Unique title and description per page; title and H1 lead with the non-brand primary keyword and agree in intent (a brand-led title on a category page competes only for navigational queries the site already wins) - Every page opens with a short, plain-text answer to the question it exists to answer; the h2s are the questions people actually ask - Open Graph and Twitter Card tags with a 1200x630 image - JSON-LD: Organization and WebSite once in a root `@graph` with stable `@id`s, BreadcrumbList on inner pages, then Article, Product, ProfilePage, or LocalBusiness where the content type matches. Define an entity once and reference it by `@id` - One h1, logical h2-h6, descriptive alt text, internal links between related pages - Core Web Vitals in target at the p75 of CrUX field data: LCP 2.5 s, INP 200 ms, CLS 0.1 (INP replaced FID in March 2024; a guide still naming FID is stale)

## Redirects and indexing policy

Load `references/indexing-policy.md` when routes move, duplicates need consolidation, or page indexing policy changes. Verify behavior against the installed Next.js version and served responses.

## AI crawlers and answer engines

Google needs no special files: AI Overviews and AI Mode run on Googlebot and core ranking, and `nosnippet` or `max-snippet:0` is the only opt-out (it removes the normal snippet too). Everything else is a robots decision per crawler class: search and citation bots (`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`) put a site in answers; training bots (`GPTBot`, `ClaudeBot`, `CCBot`) do not; `Google-Extended` and `Applebot-Extended` are usage tokens with no crawler behind them. `llms.txt` costs one route and earns almost nothing measurable. The full table, `robots.ts` rules, Content-Signal, and the Markdown-alternate patterns are in `references/answer-engines.md`.

## Programmatic SEO

For pages generated at scale, load `references/indexing-policy.md`. Require demand evidence and unique page value before indexing a new pattern.

## Audit triage order

1. Crawl and index: robots, sitemap, stray `noindex`, canonicals, redirect chains, soft 404s 2. Technical: HTTPS, Core Web Vitals field data, mobile and desktop parity, AI-crawler access through the CDN 3. On-page: title and H1 uniqueness and non-brand lead, extractable answer, internal links, thin pages 4. Orphans: indexable pages with no internal link or sitemap entry; give them a crawl path or `noindex` and drop them. A sitewide footer link is not a crawl path on a low-authority domain: Google discounts footer boilerplate, and a section linked only from the footer can sit at "URL is unknown to Google" for months while nav-linked pages beside it are crawled daily

Report in the same order: what blocks indexation first, then on-page and schema gaps, then the single next fix. Twenty equal-weight findings get none of them done. Report length follows what was found, not the section list. Treat third-party audit tools as leads, not findings: several flag Next.js output falsely (a missing `<meta name="title">` when `<title>` is present, "duplicate" `theme-color` tags that are light and dark variants, repeated `article:tag`), and at least one agent-skill auditor silently skips modules unless given an explicit module list and passes a `FAQPage` whose answers never render. Verify each lead against the served HTML before it goes in the report.

## Gotchas

- Under `cacheComponents: true` (the `scaffold-nextjs` default) the `dynamic`, `dynamicParams`, `revalidate`, and `fetchCache` segment configs are build errors: `export const revalidate` on `app/sitemap.ts`, `dynamic = 'force-static'` on a `robots.txt/route.ts` handler, `fetchCache` anywhere (`runtime`, `maxDuration`, `instant`, and `prefetch` remain valid). Move the content fetch into a separate `'use cache'` helper with `cacheLife('hours')` and `cacheTag`, called from the page or `GET` (the directive cannot sit on the `GET` export itself), and `revalidateTag(tag, 'max')` from the publish webhook. Revalidation does not purge a CDN in front of the app; purge it in the same webhook. - `generateSitemaps()` writes `/product/sitemap/0.xml`, `/product/sitemap/1.xml`, and no index file. List each generated file in `robots.ts` (`sitemap` takes an array) or hand-write the index at `app/sitemap.xml/route.ts`; otherwise Google sees only the ones it stumbles on. - `lastModified: new Date()` on every sitemap row marks the whole site as changed on every deploy; Google then ignores `lastmod` sitewide. `priority` and `changeFrequency` are ignored outright, so emitting them buys nothing and clutters the diff. - No `Host:` line in `robots.txt`. Yandex dropped the directive in 2018 in favour of a 301 to the preferred host, which `next.config.ts` already serves, and auditors flag the leftover line as an unknown directive. Leave `host` out of the `robots.ts` return value. - Next.js replaces object-valued metadata (`openGraph`, `twitter`, `robots`, `alternates`) wholesale when a child segment declares the key; it does not deep-merge. A page that adds `openGraph: { title }` loses the inherited `siteName`, `images`, and `locale`, and `images: undefined` counts as a declaration. When you add one of these blocks to a route, every field the parent supplied becomes yours to restate. Scalars (`title`, `description`, `authors`, `creator`) merge normally. - `title.template` applies to `<title>` only. `og:title` resolves against `openGraph.title.template`, a separate mechanism, so stripping a hand-rolled suffix from a page title silently strips the product name from the share card. Set both templates in the layout, or neither. A bare string `title` in a nested layout also resets the parent template to null for that subtree; return `{ default, template }` instead. - Never set `openGraph.url` in a root layout. It is not per-page, every route inherits it, and every share collapses onto the home URL. A child cannot fix it without declaring `openGraph` and losing the rest of the block. Absent beats wrong: consumers fall back to the URL they fetched, and `alternates.canonical` is already per-page. - `middleware.ts` is deprecated in Next.js 16. New request-time logic (slug checks, CSP nonces, `Accept` negotiation) goes in `proxy.ts` with an exported `proxy` function; `npx @next/codemod@canary middleware-to-proxy .` moves an existing file. Proxy is Node.js only; `runtime` is not a valid export there. - `next.config.ts` redirects match in array order and `:path*` swallows everything beneath it, so a specific source placed after a catch-all is dead code, and a bare `/old` must sit above `/old/:path*` or it lands on `/new/`. Vercel caps the list at 1,024 rules; past that, move the map into `proxy.ts`. - Next owns the `Vary` header on App Router HTML responses and replaces it with its RSC list (`rsc, next-router-state-tree, ...`). A `Vary: Accept` set in `proxy.ts` on `NextResponse.next()`, or in `headers()`, never reaches the client. It survives only on a `Response` you construct yourself, so content-negotiated Markdown must be returned directly, not rewritten to. - `notFound()` after a Suspense boundary or `loading.tsx` has flushed returns 200 with a `noindex` meta, not 404. Without Cache Components, call it before the first suspending `await`; with Cache Components the docs are explicit that every dynamic route streams a static shell first, so the existence check moves to `proxy.ts`. Analytics and uptime monitors otherwise see a health

--- name: optimise-seo description: Implements Next.js crawlability, metadata, canonicals, status codes, JSON-LD, hreflang, and crawler policy with served-page evidence. Use when asked to "fix SEO", "add a sitemap", "fix soft 404s", or "add llms.txt". For demand research, briefs, or Search Console monitoring use seo-program. ---

# Optimise SEO

- **IS:** crawlability, metadata, structured data, canonicals, redirects, status codes, hreflang, AI-crawler policy and extractable page structure, Core Web Vitals, programmatic SEO, security and privacy headers, and error-page behaviour, implemented in a Next.js App Router codebase. - **IS NOT:** visual redesign, layout, or page-level UI quality (`ui-design` Direction, Build, and Audit modes), demand research, briefs, question maps, or Search Console monitoring (`seo-program`), tenant routing for `robots.txt` and `sitemap.xml` (`multi-tenant-architecture`), or writing the article (external `ghostwriter`).

Allowed surface: `metadata` and `generateMetadata`, JSON-LD, semantic HTML, internal links, alt text, `app/sitemap.ts`, `app/robots.ts`, `app/feed.xml/route.ts`, `app/llms.txt/route.ts`, `proxy.ts`, `next.config.ts` redirects and headers, error pages, and performance work. Component styling and layout stay as they are; when a fix needs a layout change, name it as a handoff to `ui-design`.

Before writing code, read the matching guide in `node_modules/next/dist/docs/` (resolved from the app directory; in a monorepo the `next` package is not visible from the repo root). The bundled docs match the installed version; the notes in `references/nextjs-implementation.md` were checked against 16.3 and say where behaviour changed.

## References

| File | Read when | |---|---| | [references/indexing-policy.md](references/indexing-policy.md) | Redirects, duplicate URLs, index decisions, or pages generated at scale | | [references/nextjs-implementation.md](references/nextjs-implementation.md) | Before writing code in steps 2-4: metadata, sitemap and robots, redirects, indexing, streaming 404s, JSON-LD, OG images, CSP | | [references/answer-engines.md](references/answer-engines.md) | The task touches AI crawlers, `robots.ts` rules for GPTBot or ClaudeBot, `llms.txt`, Content-Signal, or "why are we not cited" | | [references/internationalisation.md](references/internationalisation.md) | The site has more than one locale | | [references/technical-hardening.md](references/technical-hardening.md) | Headers, cookies, consent, GPC, `security.txt`, or maintenance and error-status tasks | | [references/validation-evidence.md](references/validation-evidence.md) | Step 5: served-page commands and evidence expectations | | [references/seo-checklist.md](references/seo-checklist.md) | Step 5: copy into the report and mark each line with evidence |

## Workflow

Copy and track this checklist:

```text SEO progress: - [ ] Step 1: Inventory routes and decide index intent per route - [ ] Step 2: Fix crawl/index foundations (sitemap, robots, canonicals, redirects, status codes) - [ ] Step 3: Implement metadata and structured data - [ ] Step 4: Improve semantics, extractable answers, internal links, and Core Web Vitals - [ ] Step 5: Validate with references/seo-checklist.md and report evidence ```

Step 1 is a table: route, index or noindex, canonical, reason. Every later step reads from it, and a route with no row gets no work.

## Must-have on every site

- `app/sitemap.ts` lists every indexable URL with `lastModified` derived from content; `app/robots.ts` names it - One canonical per page, one host, one casing, one trailing-slash policy, set through `alternates.canonical` with `metadataBase` in the root layout. Next.js enforces the slash policy itself: by default it 308s `/about/` to `/about`, and `trailingSlash: true` inverts that (files with extensions and `.well-known/` paths are exempt either way) - Preserve the property's existing ownership verification. Use `metadata.verification` only for a supplied token and the chosen verification method; DNS verification needs no duplicate meta tag - Root layout `robots` lifts Google's default preview caps: `googleBot: { 'max-snippet': -1, 'max-image-preview': 'large', 'max-video-preview': -1 }`. Without them Google truncates the snippet and the image preview, and the snippet cap is what AI surfaces read against when deciding how much of a page they may quote - `authors` and `creator` set explicitly in the root layout, so person-level attribution does not live only in footer HTML and JSON-LD - Unique title and description per page; title and H1 lead with the non-brand primary keyword and agree in intent (a brand-led title on a category page competes only for navigational queries the site already wins) - Every page opens with a short, plain-text answer to the question it exists to answer; the h2s are the questions people actually ask - Open Graph and Twitter Card tags with a 1200x630 image - JSON-LD: Organization and WebSite once in a root `@graph` with stable `@id`s, BreadcrumbList on inner pages, then Article, Product, ProfilePage, or LocalBusiness where the content type matches. Define an entity once and reference it by `@id` - One h1, logical h2-h6, descriptive alt text, internal links between related pages - Core Web Vitals in target at the p75 of CrUX field data: LCP 2.5 s, INP 200 ms, CLS 0.1 (INP replaced FID in March 2024; a guide still naming FID is stale)

## Redirects and indexing policy

Load `references/indexing-policy.md` when routes move, duplicates need consolidation, or page indexing policy changes. Verify behavior against the installed Next.js version and served responses.

## AI crawlers and answer engines

Google needs no special files: AI Overviews and AI Mode run on Googlebot and core ranking, and `nosnippet` or `max-snippet:0` is the only opt-out (it removes the normal snippet too). Everything else is a robots decision per crawler class: search and citation bots (`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`) put a site in answers; training bots (`GPTBot`, `ClaudeBot`, `CCBot`) do not; `Google-Extended` and `Applebot-Extended` are usage tokens with no crawler behind them. `llms.txt` costs one route and earns almost nothing measurable. The full table, `robots.ts` rules, Content-Signal, and the Markdown-alternate patterns are in `references/answer-engines.md`.

## Programmatic SEO

For pages generated at scale, load `references/indexing-policy.md`. Require demand evidence and unique page value before indexing a new pattern.

## Audit triage order

1. Crawl and index: robots, sitemap, stray `noindex`, canonicals, redirect chains, soft 404s 2. Technical: HTTPS, Core Web Vitals field data, mobile and desktop parity, AI-crawler access through the CDN 3. On-page: title and H1 uniqueness and non-brand lead, extractable answer, internal links, thin pages 4. Orphans: indexable pages with no internal link or sitemap entry; give them a crawl path or `noindex` and drop them. A sitewide footer link is not a crawl path on a low-authority domain: Google discounts footer boilerplate, and a section linked only from the footer can sit at "URL is unknown to Google" for months while nav-linked pages beside it are crawled daily

Report in the same order: what blocks indexation first, then on-page and schema gaps, then the single next fix. Twenty equal-weight findings get none of them done. Report length follows what was found, not the section list. Treat third-party audit tools as leads, not findings: several flag Next.js output falsely (a missing `<meta name="title">` when `<title>` is present, "duplicate" `theme-color` tags that are light and dark variants, repeated `article:tag`), and at least one agent-skill auditor silently skips modules unless given an explicit module list and passes a `FAQPage` whose answers never render. Verify each lead against the served HTML before it goes in the report.

## Gotchas

- Under `cacheComponents: true` (the `scaffold-nextjs` default) the `dynamic`, `dynamicParams`, `revalidate`, and `fetchCache` segment configs are build errors: `export const revalidate` on `app/sitemap.ts`, `dynamic = 'force-static'` on a `robots.txt/route.ts` handler, `fetchCache` anywhere (`runtime`, `maxDuration`, `instant`, and `prefetch` remain valid). Move the content fetch into a separate `'use cache'` helper with `cacheLife('hours')` and `cacheTag`, called from the page or `GET` (the directive cannot sit on the `GET` export itself), and `revalidateTag(tag, 'max')` from the publish webhook. Revalidation does not purge a CDN in front of the app; purge it in the same webhook. - `generateSitemaps()` writes `/product/sitemap/0.xml`, `/product/sitemap/1.xml`, and no index file. List each generated file in `robots.ts` (`sitemap` takes an array) or hand-write the index at `app/sitemap.xml/route.ts`; otherwise Google sees only the ones it stumbles on. - `lastModified: new Date()` on every sitemap row marks the whole site as changed on every deploy; Google then ignores `lastmod` sitewide. `priority` and `changeFrequency` are ignored outright, so emitting them buys nothing and clutters the diff. - No `Host:` line in `robots.txt`. Yandex dropped the directive in 2018 in favour of a 301 to the preferred host, which `next.config.ts` already serves, and auditors flag the leftover line as an unknown directive. Leave `host` out of the `robots.ts` return value. - Next.js replaces object-valued metadata (`openGraph`, `twitter`, `robots`, `alternates`) wholesale when a child segment declares the key; it does not deep-merge. A page that adds `openGraph: { title }` loses the inherited `siteName`, `images`, and `locale`, and `images: undefined` counts as a declaration. When you add one of these blocks to a route, every field the parent supplied becomes yours to restate. Scalars (`title`, `description`, `authors`, `creator`) merge normally. - `title.template` applies to `<title>` only. `og:title` resolves against `openGraph.title.template`, a separate mechanism, so stripping a hand-rolled suffix from a page title silently strips the product name from the share card. Set both templates in the layout, or neither. A bare string `title` in a nested layout also resets the parent template to null for that subtree; return `{ default, template }` instead. - Never set `openGraph.url` in a root layout. It is not per-page, every route inherits it, and every share collapses onto the home URL. A child cannot fix it without declaring `openGraph` and losing the rest of the block. Absent beats wrong: consumers fall back to the URL they fetched, and `alternates.canonical` is already per-page. - `middleware.ts` is deprecated in Next.js 16. New request-time logic (slug checks, CSP nonces, `Accept` negotiation) goes in `proxy.ts` with an exported `proxy` function; `npx @next/codemod@canary middleware-to-proxy .` moves an existing file. Proxy is Node.js only; `runtime` is not a valid export there. - `next.config.ts` redirects match in array order and `:path*` swallows everything beneath it, so a specific source placed after a catch-all is dead code, and a bare `/old` must sit above `/old/:path*` or it lands on `/new/`. Vercel caps the list at 1,024 rules; past that, move the map into `proxy.ts`. - Next owns the `Vary` header on App Router HTML responses and replaces it with its RSC list (`rsc, next-router-state-tree, ...`). A `Vary: Accept` set in `proxy.ts` on `NextResponse.next()`, or in `headers()`, never reaches the client. It survives only on a `Response` you construct yourself, so content-negotiated Markdown must be returned directly, not rewritten to. - `notFound()` after a Suspense boundary or `loading.tsx` has flushed returns 200 with a `noindex` meta, not 404. Without Cache Components, call it before the first suspending `await`; with Cache Components the docs are explicit that every dynamic route streams a static shell first, so the existence check moves to `proxy.ts`. Analytics and uptime monitors otherwise see a health