Ghost Sitemaps and Robots — Built-in Sitemap, robots.txt and Crawler Control

In this tutorial, you'll learn how Ghost handles XML sitemaps and robots.txt — automatically generated search engine tools that help crawlers discover and index your content efficiently.

What You'll Learn

• The Ghost sitemap structure at /sitemap.xml
• How Ghost organizes sitemaps by content type
• robots.txt and what it controls
• Customizing robots.txt behavior
• Submitting sitemaps to Google Search Console
• Managing noindex directives for specific pages
• Crawl budget optimization
• Common sitemap and robots issues

Why It Matters

Sitemaps and robots.txt are the first things search engine crawlers look at when they visit your site. A well-structured sitemap ensures every piece of content is discovered. A properly configured robots.txt tells crawlers what to index and what to skip. Ghost generates both automatically, but understanding how they work helps you diagnose indexing issues and optimize crawler efficiency.

Real-World Use

A site with 500+ posts notices that only 200 are indexed in Google. The site owner checks /sitemap.xml and finds it only contains 300 entries. After investigating, they discover some posts have empty slugs (a rare edge case). Fixing the slugs regenerates the sitemap, and a manual resubmit to Google Search Console results in all 500 posts being indexed within a week.

Learning Path

flowchart LR
  A["SEO Settings"] --> B["Sitemaps & Robots
You are here"]:::current
  B --> C["Structured Data"]
  C --> D["Performance Optimization"]

  classDef current fill:#38bdf8,color:#0f172a,stroke-width:2px

XML Sitemaps

Ghost automatically generates XML sitemaps for your site.

Main Sitemap

Accessible at:

https://yoursite.com/sitemap.xml

The main sitemap is an index file that points to sub-sitemaps for each content type:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://yoursite.com/sitemap-posts.xml</loc>
  </sitemap>
  <sitemap>
    <loc>https://yoursite.com/sitemap-pages.xml</loc>
  </sitemap>
  <sitemap>
    <loc>https://yoursite.com/sitemap-tags.xml</loc>
  </sitemap>
  <sitemap>
    <loc>https://yoursite.com/sitemap-authors.xml</loc>
  </sitemap>
</sitemapindex>

Sub-Sitemaps

Each content type has its own sitemap:

Post Sitemap Entry

<url>
  <loc>https://yoursite.com/my-post-slug/</loc>
  <lastmod>2024-01-15T10:00:00.000Z</lastmod>
  <priority>0.6</priority>
</url>

Sitemap Features

• Automatic updates: Sitemaps regenerate when content is published, updated, or deleted
• Tags and authors included: Tag and author pages are indexed for topic discovery
• Priority set per type: Posts get higher priority than tags
• Lastmod included: Tells crawlers when content was last changed

Exclusions

The sitemap does NOT include:

• Draft or scheduled posts
• Member-only or paid-only content (noindexed)
• Internal tags (prefixed with #)
• Error pages
• Admin URLs (/ghost/)

robots.txt

Ghost automatically generates a robots.txt file at:

https://yoursite.com/robots.txt

Default Content

User-agent: *
Disallow: /ghost/
Disallow: /p/
Disallow: /email/
Disallow: /members/
Disallow: /api/

Sitemap: https://yoursite.com/sitemap.xml

What It Means

Customizing robots.txt

Ghost does not provide a built-in interface to edit robots.txt. For custom rules, use code injection:

1. Go to Settings > Code Injection.
2. Add a custom robots tag in the Site Header:

<meta name="robots" content="noindex, follow">

For comprehensive robots.txt customization, configure it at the server level (Nginx):

location = /robots.txt {
    add_header Content-Type text/plain;
    return 200 "User-agent: *\nDisallow: /ghost/\nDisallow: /p/\nDisallow: /members/\nSitemap: https://yoursite.com/sitemap.xml\n";
}

Noindex Directives

Ghost automatically adds a noindex meta tag to:

• Member-only posts
• Paid-only posts
• Admin pages
• Paginated pages beyond the first page (configurable)

Checking Noindex Status

View the page source in your browser. Look for:

<meta name="robots" content="noindex, nofollow">

If present, search engines will not index this page.

Custom Noindex

Add noindex to specific pages via code injection or per-post:

<meta name="robots" content="noindex">

Submitting Sitemaps to Google Search Console

1. Go to Google Search Console and add your site.
2. Navigate to Sitemaps in the left sidebar.
3. Enter sitemap.xml as the sitemap URL.
4. Click "Submit."

Google crawls the sitemap and starts indexing your pages.

Crawl Budget Optimization

Crawl budget is how many pages a search engine crawls on your site in a given time period. Optimize it by:

Do

• Publish high-quality content regularly
• Update existing content to trigger lastmod changes
• Use the sitemap to highlight your most important content
• Fix broken links that waste crawl budget
• Use internal linking to guide crawlers to important pages

Avoid

• Large numbers of thin or low-value pages
• Infinite pagination (use rel="next" and rel="prev" or consolidate)
• Duplicate content that confuses crawlers
• Orphan pages with no internal links

Sitemap for Tag and Author Pages

Tag and author pages are included in the sitemap by default. This is good for SEO — each tag page can rank for topic-related queries. However, if you have hundreds of tags, consider whether all tag pages provide enough unique value to be indexed.

Common Mistakes

1. Not submitting the sitemap to Search Console: Ghost generates the sitemap automatically, but Google does not know about it until you submit it in Search Console. Submit sitemap.xml as soon as your site is live.

2. Blocking CSS and JS in robots.txt: Some configurations block /assets/ or /css/ in robots.txt. This prevents Google from seeing your page layout and can hurt rankings based on mobile-friendliness and page experience signals.

3. Confusing noindex with removing from sitemap: A noindex page can still appear in the sitemap. Search engines see both directives and typically honor noindex, but they will waste a crawl visiting a page that tells them not to index it.

4. Ignoring sitemap errors in Search Console: If your sitemap has errors (404 URLs, redirect chains, or incorrect dates), Search Console shows warnings. Check these regularly.

5. Assuming the sitemap covers all content: The sitemap only includes published, public posts. Drafts, scheduled posts, and member-only content are excluded. If your post is not appearing in search, check its visibility setting first.

Practice Questions

1. What URLs does the Ghost sitemap include? Answer: The sitemap includes all published, public posts, pages, tags (excluding internal tags), and authors. It excludes drafts, scheduled posts, member-only/paid-only content, admin URLs, error pages, and API endpoints.

2. What does Ghost's default robots.txt disallow? Answer: The default robots.txt disallows crawlers from accessing /ghost/ (admin panel), /p/ (preview URLs), /email/ (email-only pages), /members/ (member pages), and /api/ (API endpoints). It also points crawlers to the sitemap location.

3. How do you submit a Ghost sitemap to Google Search Console? Answer: In Google Search Console, go to the Sitemaps section, enter sitemap.xml as the sitemap URL, and click Submit. Google then fetches and processes the sitemap to discover your content.

4. Challenge: Audit a Ghost site's search engine visibility. Check the sitemap at /sitemap.xml and verify all published posts are included. Check robots.txt at /robots.txt for correctness. Submit the sitemap to Google Search Console (if not already done). Check for any noindex directives on pages that should be indexed. Fix any issues found.

FAQ

Mini Project

Your task: Audit and optimize the sitemap and robots.txt for a Ghost site.

1. Visit /sitemap.xml and document all sub-sitemaps.
2. Verify 5 random posts appear in the sitemap.
3. Visit /robots.txt and verify the rules are correct.
4. Submit the sitemap to Google Search Console.
5. Check Search Console for sitemap errors (if you have access).
6. Create a monitoring checklist for ongoing sitemap health checks.

This exercise ensures your site is fully discoverable by search engines.

What's Next

Now that sitemaps are configured, learn about Google Analytics integration:

Continue to Lesson 32: Google Analytics — Ghost analytics settings and custom code injection for analytics.

Related lessons:

• SEO Settings — Meta tags and social sharing
• Performance Optimization — Site speed and CDN