An SEO glossary of terms every marketer should know in 2026
A practical, current glossary covering the essential SEO terminology every marketer should be comfortable with, from foundational concepts to 2026-specific terms.
6 min read
The robots.txt file tells search engine crawlers which parts of a site they are allowed or not allowed to crawl. It is a simple, plain-text file with outsized consequences — a single misplaced line can accidentally block an entire site from being crawled at all.
What robots.txt actually controls, and what it does not
Robots.txt controls crawling — whether a bot is permitted to request a given URL at all. It does not directly control indexing. A page can still be indexed, sometimes with limited information, even if it is blocked from crawling, if other pages link to it — this is a common source of confusion worth understanding clearly.
- Disallow directives prevent crawling of specified paths.
- It does not guarantee a blocked page will not appear in search results at all.
- Use a noindex meta tag, not robots.txt, if you specifically want a page excluded from the index — but note that a noindex tag on a page also blocked by robots.txt can never actually be seen by Google, defeating its purpose.
The single most damaging mistake
A line reading 'Disallow: /' blocks crawling of the entire site. This happens most often when a development or staging environment's robots.txt file, deliberately set to block everything, accidentally gets carried over to the live production site during a launch or migration.
Checking your live robots.txt file after any site launch or migration takes thirty seconds and has saved more sites from accidental deindexing than almost any other single SEO habit.
What is generally worth blocking
Internal search result pages, admin or login areas, staging or duplicate content paths, and certain low-value parameter combinations are reasonable candidates for disallowing, provided they offer no value to search visibility and are not needed for the noindex tag itself to be seen.
- Internal site search result pages, which are typically low-value and can create near-infinite URL variations.
- Admin, login and account areas with no public SEO value.
- Staging or development environments, if genuinely separate from production.
- Low-value parameter combinations that create duplicate content without unique value.
Testing and monitoring
Search Console includes a robots.txt tester that flags syntax errors and lets you check whether a specific URL is blocked before making changes live. Google's robots.txt documentation is the definitive technical reference for correct syntax.
Frequently asked questions
Can robots.txt block a page from ever appearing in search results?
Not reliably on its own — if other pages link to a blocked URL, Google may still index it with limited information. A noindex meta tag is the correct tool for guaranteeing exclusion, and the page must not be blocked from crawling for that tag to be seen.
How do I check my site's current robots.txt file?
Visit yourdomain.com/robots.txt directly in a browser — it is always a plain text file at the root of your domain, publicly visible to anyone.
Should I block CSS and JavaScript files in robots.txt?
No — Google needs to crawl CSS and JavaScript to properly render and understand your page, and blocking these can seriously harm how well Google can evaluate your content and layout.
If you want a second pair of eyes on your robots.txt configuration, contact our team — it is a five-minute check that can prevent a very expensive mistake.
The takeaway
Check your robots.txt file directly, right now, at yourdomain.com/robots.txt — an accidental 'Disallow: /' left over from a development environment is one of the most common, most damaging SEO mistakes.
Written by Fibvo — Bengaluru and Erode, India — working with clients in other time zones. Questions, or think we have it wrong? Tell us.
Keep reading
Related articles
A hybrid SEO strategy for a world with both blue links and AI answers
A practical framework for building one content strategy that serves both traditional search ranking and AI answer engine visibility simultaneously.
6 min readGoogle's Quality Rater Guidelines explained, in plain language
What the Quality Rater Guidelines actually are, how human raters use them, and the practical lessons worth extracting from this lengthy public document.
5 min readNext step
Tell us what you are trying to grow.
Send the problem in a paragraph. We will reply within one working day with an honest view of whether we can help, and what it is likely to cost.