<!-- Canonical: https://gavek.dk/learn/robots-txt/; Language: en -->

[Home](https://gavek.dk/)/[Learning library](https://gavek.dk/learn/)/robots.txt without accidental blocks

Technical & content SEO

# robots.txt without accidental blocks — practical guide

robots.txt gives cooperating crawlers rules about which URL paths they may request. Treat it as crawl management, not access security. A publicly blocked URL may still be known to a search engine, and confidential information needs authentication rather than a line in a publicly readable text file.

Practical guide · English & Danish

### In this guide

[What matters](https://gavek.dk/#understand)[A practical workflow](https://gavek.dk/#workflow)[Apply it to your website](https://gavek.dk/#application)[Mistakes to avoid](https://gavek.dk/#mistakes)[What to measure](https://gavek.dk/#measure)[Common questions](https://gavek.dk/#questions)[Further reading](https://gavek.dk/#sources)

## Start with the decision, not the metric.

robots.txt gives cooperating crawlers rules about which URL paths they may request. Treat it as crawl management, not access security. A publicly blocked URL may still be known to a search engine, and confidential information needs authentication rather than a line in a publicly readable text file.

A single broad rule can unintentionally block a whole product directory or the resources needed to render pages. Configuration mistakes often come from copying staging settings into production. Keep a clear reason for each rule and review the actual public file after deployment, especially after changing a CMS or CDN.

## Turn the idea into a repeatable workflow.

- Open the robots.txt file at the exact origin you want to manage. Check the hostname, scheme, response status, and contents. A file on one subdomain does not automatically configure another subdomain's crawler policy.

- List the crawler groups and the URL patterns each rule affects. Test representative allowed and blocked URLs, including a product page, an asset, a filtered category, and any path that resembles a wildcard boundary.

- Choose the right control for the job. Use authentication for private areas and an accessible noindex directive for public pages you do not want indexed. Do not block access while expecting a crawler to read that page's noindex instruction.

- Store the intended rules with the deployment configuration. Add the sitemap location where appropriate, verify production after release, and inspect logs or Search Console when an important path stops receiving crawler requests unexpectedly.

## Make the next step specific.

Choose one commercially important page or a small, related group of URLs. Record its current state, the user need, the proposed change and the person responsible. Separate evidence from assumptions so you can tell whether the work solved the original problem.

Discuss these checks with Gavek as part of a defined SEO engagement. Agree the investigation, priorities, implementation responsibilities and review criteria using your website and the data available to your team.

[Discuss the SEO work ↗](https://gavek.dk/learn/seo-project-checklist/)

## Avoid the work that creates more work.

- Leaving a staging-wide disallow rule on the production host.

- Using robots.txt as a password or privacy protection mechanism.

- Blocking scripts or styles that are necessary to understand the page.

## Measure useful progress.

Priority paths unintentionally disallowed.

Production robots.txt response and deployment consistency.

Crawler access failures associated with newly changed rules.

Keep the comparison period, location, device and data source consistent. Search results and AI answers vary; a single observation is a diagnostic clue, not proof of long-term growth or causation. Connect visibility changes to relevant enquiries, qualified leads or sales wherever reliable first-party data is available.

## Questions worth asking.

### Does Disallow remove a page from search?

Not reliably. It prevents compliant crawling of matching paths, but it does not itself request deindexing. If removal from results is the goal, select an appropriate indexing or access control and verify the outcome.

### Should small sites restrict crawling heavily?

Usually begin with simple, understandable rules. Complex blocks create maintenance risk. Introduce restrictions for a documented need, then verify they do not hide valuable pages or resources.

## Go to the source.

These independent guides are grounded in primary documentation. Platform behavior and search systems can change; consult the linked documentation when implementing a change.

- [developers.google.com · › search › docs › crawling-indexing › robots › intro](https://developers.google.com/search/docs/crawling-indexing/robots/intro)

- [developers.google.com · › crawling › docs › robots-txt › create-robots-txt](https://developers.google.com/crawling/docs/robots-txt/create-robots-txt)

- [developers.google.com · › search › docs › crawling-indexing › block-indexing](https://developers.google.com/search/docs/crawling-indexing/block-indexing)

Editorial guidance from Gavek. No ranking guarantees, customer-result claims or endorsement by the referenced platforms.

## AI crawler policies: search, training and verified log evidence

Make a purpose-specific policy and verify its effect. Robots rules communicate preferences; they are not authentication or proof of a crawler’s identity.

## Record search and training decisions separately

Keep a policy row for each documented vendor role, intended public paths, business preference and configuration owner. A search-discovery decision is not automatic consent to training. Content-Signal preferences and vendor-specific robots controls are not interchangeable guarantees; inspect the vendor documentation and the actual rule match before editing. Keep private account routes behind authentication regardless of robots.

## Verify traffic without overclaiming

A request labelled GPTBot or Googlebot can be spoofed. Use the vendor’s documented IP verification procedure or trustworthy CDN verification where available. Retain only necessary log fields and avoid collecting user secrets. Count verified requests, status classes and affected paths. These observations describe fetching, not citations, leads or people. A crawler downloading ten URLs is not ten potential customers.

## Test the change narrowly

Save the original rules, check precedence and test both an intended public page and a protected negative control. Compare origin and CDN responses where authorised. If a challenge remains, identify that layer rather than adding contradictory Allow lines. Record the new policy, approval and date; leave uncertainty visible when verified vendor access cannot be observed.

## Continue with the related decision

- [AEO learning paths](https://gavek.dk/aeo/)

- [Structured data for AI search: describe facts, validate meaning](https://gavek.dk/learn/structured-data-for-ai-search/)

- [Discuss the scope of AEO and AI-search work](https://gavek.dk/services/ai-search/)

### Sources, method and review information

Editorial publisher: Gavek. Reviewed 8 November 2026. Practical methods and clearly labelled examples are original editorial guidance, not customer results or a disclosed platform algorithm. Check sources when implementing.

- [OpenAI · crawler roles](https://developers.openai.com/api/docs/bots)

- [Google Search · google](https://developers.google.com/search/docs/appearance/ai-features)

Your next move starts here

## Make search a more useful part of your growth.

Start with your website. Leave with a clearer picture of what to improve, what to measure and where to focus next.

[Discuss your SEO project ↗](https://gavek.dk/contact/)[Explore our services ↗](https://gavek.dk/services/)
