Laravel Sitemaps and robots.txt: Dynamic Generation Done Properly

Laravel gives you server-rendered HTML by default, which removes most of the crawling problems JavaScript applications suffer. What it does not give you is a sitemap or a robots.txt strategy — both are your responsibility, and both are commonly implemented in ways that break at scale.

Serving robots.txt from Laravel

The static file in public/robots.txt works, but it is the same in every environment — which is how staging sites end up indexed. Serve it through a route instead so it reacts to the environment:

// routes/web.php
Route::get('/robots.txt', function () {
    $lines = app()->environment('production')
        ? [
            'User-agent: *',
            'Disallow: /admin',
            'Disallow: /cart',
            'Disallow: /search',
            '',
            'Sitemap: ' . url('/sitemap.xml'),
        ]
        : [
            'User-agent: *',
            'Disallow: /',
        ];

    return response(implode(PHP_EOL, $lines), 200)
        ->header('Content-Type', 'text/plain');
});

Delete public/robots.txt if it exists, otherwise the web server serves the file and never reaches the route.

A sitemap that scales

Three approaches, in order of catalogue size:

Size	Approach
Under ~500 URLs	Generate on request, cached for a few hours
500–50,000	Scheduled command writing a static file to public/
Over 50,000	Sitemap index plus chunked child sitemaps, written on a schedule

Scheduled generation with chunking

// app/Console/Commands/GenerateSitemap.php
public function handle(): int
{
    $chunkSize = 45000;
    $index = [];
    $part = 1;

    Product::query()
        ->where('is_published', true)
        ->select(['slug', 'updated_at'])
        ->chunkById($chunkSize, function ($products) use (&$index, &$part) {
            $xml = view('sitemaps.urlset', ['items' => $products])->render();
            $file = "sitemap-products-{$part}.xml";
            Storage::disk('public_root')->put($file, $xml);
            $index[] = ['loc' => url($file), 'lastmod' => now()->toAtomString()];
            $part++;
        });

    Storage::disk('public_root')->put(
        'sitemap.xml',
        view('sitemaps.index', ['sitemaps' => $index])->render()
    );

    return self::SUCCESS;
}

Schedule it at a frequency that matches how often content actually changes — hourly for a news site, daily for most catalogues.

Getting lastmod right

lastmod should reflect a meaningful content change, not a touch of the row. If your updated_at changes when stock levels or view counts update, use a dedicated content_updated_at column set only when the indexable content changes.

What belongs in the sitemap

• Only canonical URLs — never a parameterised or redirected variant.
• Only 200-status, indexable pages. A URL that is both in the sitemap and noindexed sends a contradiction.
• Absolute URLs on the canonical host and protocol, with consistent trailing-slash behaviour.
• Separate child sitemaps per content type — products, categories, articles, static pages — so the Search Console sitemap report shows you which type has an indexing problem.

Multilingual sitemaps

If the application serves multiple locales, include hreflang annotations inside the sitemap with xhtml:link rel="alternate" entries rather than duplicating them across thousands of pages. Annotations must be reciprocal: every locale version lists every other, including itself, plus an x-default.

Serving and verifying

1. Confirm /sitemap.xml returns 200 with Content-Type: application/xml.
2. Gzip is supported and worth it for large files; serve .xml.gz with the correct encoding header.
3. Submit the index URL — not each child — in Search Console.
4. Watch the sitemap report for "Couldn't fetch" and for a discovered-versus-indexed gap per content type.
5. Re-run a bulk status check on a sample of sitemap URLs monthly to catch redirects and 404s creeping in.

Common Laravel-specific mistakes

• Generating the sitemap in a controller with an unbounded all() query — memory exhaustion at a few tens of thousands of rows.
• Using url() without a configured APP_URL, producing localhost URLs in production output.
• Including soft-deleted or unpublished records because the global scope was bypassed.
• Leaving the development Disallow: / in production after copying an environment file.
• Sitemap URLs with a trailing slash while routes resolve without one, creating a redirect for every entry.

For the wider Laravel picture — metadata per route, structured data, and performance — see the Laravel SEO best practices guide.