An SEO glossary of terms every marketer should know in 2026
A practical, current glossary covering the essential SEO terminology every marketer should be comfortable with, from foundational concepts to 2026-specific terms.
6 min read
The robots meta tag provides page-level crawling and indexing instructions directly within a page's HTML head, offering more granular control than robots.txt's site-wide or directory-level directives, with a range of specific values worth understanding beyond the commonly known noindex.
The full range of directive values
- noindex: exclude this specific page from the search index.
- nofollow: do not follow (pass ranking credit through) links on this specific page.
- noarchive: do not show a cached version of this page.
- nosnippet: do not show a text snippet or preview for this page in search results.
- noimageindex: do not index images on this specific page.
Combining directives
Multiple directives can be combined in a single meta tag, such as 'noindex, nofollow' for a page you want fully excluded from both indexing and link-passing consideration — a common combination for internal utility pages with no search or link value.
The robots meta tag gives you page-level precision that robots.txt's broader, path-based rules simply cannot match — the two tools solve genuinely different problems and often need to work together.
The X-Robots-Tag HTTP header alternative
For non-HTML files (PDFs, images, other documents) where a meta tag in an HTML head is not applicable, the equivalent instructions can be sent via the X-Robots-Tag HTTP header instead, achieving the same effect for file types that have no HTML head to place a meta tag within.
Common implementation mistakes
Combining noindex with a robots.txt block on the same page, as covered in our noindex versus nofollow guide, prevents Google from ever seeing the meta tag at all — this remains the most common, significant implementation error involving this tag.
Frequently asked questions
Does the robots meta tag replace the need for robots.txt entirely?
No, the two serve complementary purposes — robots.txt controls crawling at a broader, path-based level; the meta tag controls indexing and other behaviours at an individual page level, and often both are used together strategically.
How long does it take for a noindex meta tag to take effect?
It depends on how quickly Google recrawls the specific page — for urgent removal, Search Console's Removals tool provides a faster, if temporary, hiding mechanism while the tag itself propagates.
Can I apply different directives to mobile versus desktop versions of the same page?
Given mobile-first indexing, the mobile version's directives are what Google primarily uses — ensure consistency between versions to avoid unintended, conflicting signals.
If you want help implementing robots directives correctly across your site, talk to our team.
The takeaway
Use the X-Robots-Tag HTTP header, not the meta tag, for non-HTML files like PDFs — the meta tag only works within HTML head content and cannot be applied to other file types at all.
Written by Fibvo — Bengaluru and Erode, India — working with clients in other time zones. Questions, or think we have it wrong? Tell us.
Keep reading
Related articles
A hybrid SEO strategy for a world with both blue links and AI answers
A practical framework for building one content strategy that serves both traditional search ranking and AI answer engine visibility simultaneously.
6 min readGoogle's Quality Rater Guidelines explained, in plain language
What the Quality Rater Guidelines actually are, how human raters use them, and the practical lessons worth extracting from this lengthy public document.
5 min readNext step
Tell us what you are trying to grow.
Send the problem in a paragraph. We will reply within one working day with an honest view of whether we can help, and what it is likely to cost.