Canonical URL
Also used: canonical page
Canonical URL is the representative URL a search engine selects from a set of duplicate or substantially similar resources.
Publishers can signal a preference through redirects, rel="canonical", sitemaps, and consistent internal links, while the search engine reconciles those signals.
Important limit: A declared canonical is a hint rather than an absolute command, and conflicting signals can lead to a different selection.
Related terms: Canonicalization, XML sitemap, Duplicate content, Internal link, Hreflang
See how technical SEO evidence is examined in a human-reviewed audit. →
Canonicalization
Also used: URL canonicalization, duplicate consolidation
Canonicalization is the process of grouping duplicate or very similar URLs and choosing one representative URL for search processing.
Consolidation can reduce duplicate crawling and combine relevant signals when redirects, canonicals, sitemaps, and internal links point consistently to the preferred resource.
Important limit: Canonicalization does not merge materially different intent or guarantee that the publisher’s preferred URL will be selected.
Related terms: Canonical URL, Redirect, Duplicate content, Content consolidation
See how technical SEO evidence is examined in a human-reviewed audit. →
Crawl budget
Also used: crawl allocation, Google crawl budget
Crawl budget is Google’s term for the set of URLs Googlebot can and wants to crawl on a site within a given period.
Google describes it through crawl capacity and crawl demand, making it mainly relevant to large sites or sites with access and efficiency problems.
Important limit: Most small sites do not need crawl-budget optimization, and there is no publisher-controlled fixed quota for every site.
Related terms: Crawling, Crawl capacity limit, Crawl demand, XML sitemap
See how technical SEO evidence is examined in a human-reviewed audit. →
Crawl capacity limit
Also used: crawl capacity, host capacity limit
Crawl capacity limit is Google’s estimate of the maximum simultaneous connections a site can support without harming its servers.
Googlebot adjusts request activity in response to server health, response speed, crawl settings, and signals that the host is overloaded.
Important limit: Higher capacity does not create crawl demand, and raising the limit does not guarantee that more URLs will be fetched.
Related terms: Googlebot, Crawl budget, Crawl demand, HTTP status code
See how technical SEO evidence is examined in a human-reviewed audit. →
Crawl demand
Also used: Google crawl demand
Crawl demand is Google's estimate of how much it wants to crawl a site's URLs based on factors such as popularity and staleness.
Demand helps determine actual crawl activity within the site's capacity, and it can shift as known pages, update patterns, and Google's systems change.
Important limit: Publishers cannot set crawl demand directly, and more demand is not a ranking metric or business outcome.
Related terms: Crawl budget, Crawl capacity limit, URL discovery, Content freshness
See how technical SEO evidence is examined in a human-reviewed audit. →
Crawling
Also used: crawl, web crawling
Crawling is the automated process of requesting discovered URLs and retrieving their responses for further processing.
For search, crawling supplies content and response signals that may later be rendered, interpreted, canonicalized, and considered for indexing.
Important limit: Crawling is distinct from discovery and indexing, and repeated requests do not prove index inclusion.
Related terms: Web crawler, URL discovery, Indexing, Robots.txt
See how technical SEO evidence is examined in a human-reviewed audit. →
Googlebot
Also used: Googlebot crawler
Googlebot is the collective name for Google's crawlers used to discover and retrieve web content for Google products, including Search.
Google documents Googlebot crawler variants and crawl controls, together with reverse and forward DNS checks and published IP ranges for verification.
Important limit: A Googlebot user-agent string alone is not sufficient verification; Google documents DNS and IP-range methods for confirming Google crawler requests.
Related terms: Web crawler, Crawling, URL Inspection tool, Mobile-first indexing
See how technical SEO evidence is examined in a human-reviewed audit. →
Hreflang
Also used: hreflang annotation
Hreflang is an annotation that connects alternate page versions intended for different languages or regional audiences.
Publishers identify language and optional region codes in HTML, HTTP headers, or sitemaps, with reciprocal references helping Google understand the variant set.
Important limit: Hreflang does not translate content, replace canonicalization, or guarantee which version a user will receive.
Related terms: Canonical URL, XML sitemap, Canonicalization, URL structure
See how technical SEO evidence is examined in a human-reviewed audit. →
HTTP status code
Also used: status code, response status
HTTP status code is the three-digit result in an HTTP response that describes how the server handled a request.
The first digit identifies one of five response classes: informational, successful, redirection, client error, or server error.
Important limit: The code does not prove that the returned body is correct; Google can treat a successful response containing an error page as a soft error.
Related terms: Redirect, Soft 404, Crawl capacity limit, Permanent redirect
See how technical SEO evidence is examined in a human-reviewed audit. →
Index
Also used: search index, web index
Index is an organized store of processed information from resources that a search system may use when serving results.
For Google Search, content is analyzed after retrieval and eligible information may be stored with signals about pages, media, language, and canonical relationships.
Important limit: Presence in an index does not guarantee visibility for a query, a particular presentation, or continued inclusion.
Related terms: Crawling, Indexing, Canonical URL, Search ranking system
See how technical SEO evidence is examined in a human-reviewed audit. →
Indexing
Also used: search indexing
Indexing is the process of analyzing a retrieved resource and deciding whether and how its information may be stored in a search index.
The process can include rendering, text and media analysis, duplicate grouping, canonical selection, and interpretation of allowed indexing directives.
Important limit: A crawl does not guarantee indexing, and indexing does not guarantee ranking or search traffic.
Related terms: Crawling, Index, URL Inspection tool, Noindex directive
See how technical SEO evidence is examined in a human-reviewed audit. →
Mobile-first indexing
Also used: mobile indexing, mobile-first
Mobile-first indexing is Google’s use of a site’s mobile version as the primary content source for indexing and search processing.
Content, structured data, metadata, images, and crawl access should remain equivalent enough on mobile for Googlebot Smartphone to retrieve the intended information.
Important limit: Mobile-first indexing is not a separate mobile index and does not mean desktop usability or responsive design can be ignored.
Related terms: Googlebot, Indexing, Client-side rendering, Responsive web design
See how technical SEO evidence is examined in a human-reviewed audit. →
Nofollow
Also used: rel nofollow, nofollow link
Nofollow is a link relationship value used to qualify a link when the publisher does not want to imply normal association or endorsement.
Google treats rel="nofollow" as a hint for search processing and recommends more specific sponsored or user-generated values when those relationships apply.
Important limit: Nofollow is not an access control, does not hide the destination, and does not guarantee that a URL will never be discovered.
Related terms: Sponsored link qualification, User-generated content link qualification, Internal link, Search spam
See how technical SEO evidence is examined in a human-reviewed audit. →
Noindex directive
Also used: noindex
Noindex directive is a supported robots instruction asking a search engine not to include the addressed resource in its index.
It can be delivered through a robots meta tag or X-Robots-Tag and takes effect after a supporting crawler retrieves and processes it.
Important limit: Blocking the URL in robots.txt can prevent discovery of the noindex instruction, and removal may not be immediate.
Related terms: Indexing, Robots.txt, Robots meta tag, X-Robots-Tag
See how technical SEO evidence is examined in a human-reviewed audit. →
Permanent redirect
Permanent redirect is an HTTP redirect indicating that a resource has moved durably to a new location.
Status codes 301 and 308 communicate permanence, with 308 preserving the request method where protocol behavior matters; search systems can use the signal during canonicalization.
Important limit: A permanent redirect should not be used for a temporary experiment, and search consolidation can take time after deployment.
Related terms: Redirect, Temporary redirect, Canonical URL, URL structure
See how technical SEO evidence is examined in a human-reviewed audit. →
Redirect
Also used: URL redirect, HTTP redirection
Redirect is a response or routing mechanism that sends a requester from one URL to another location.
Google documents server-side HTTP 3xx redirects, including permanent and temporary types, as ways to route users and Google Search to another URL.
Important limit: Long chains, loops, irrelevant destinations, and client-only behavior can impede users or processing, so a redirect is not merely a URL substitution.
Related terms: Permanent redirect, Temporary redirect, HTTP status code, Canonicalization
See how technical SEO evidence is examined in a human-reviewed audit. →
Robots meta tag
Also used: meta robots, meta robots tag
Robots meta tag is HTML metadata that gives supported crawlers page-level instructions about indexing and search-result presentation.
The tag is placed in the document head and can carry directives such as noindex or controls for snippets, images, and cached presentation.
Important limit: A crawler must be allowed to fetch the page to read the tag, and unsupported crawlers may ignore its directives.
Related terms: Robots.txt, X-Robots-Tag, Noindex directive, Indexing
See how technical SEO evidence is examined in a human-reviewed audit. →
Robots.txt
Also used: robots.txt file
Robots.txt is a host-level text file that communicates crawler access rules using the Robots Exclusion Protocol.
User-agent groups, allow rules, and disallow rules are matched against URL paths so compliant crawlers can decide whether they may fetch a resource.
Important limit: Robots.txt controls crawling, not guaranteed deindexing, and the file is public rather than a security or confidentiality control.
Related terms: Web crawler, Crawling, Noindex directive, XML sitemap
See how technical SEO evidence is examined in a human-reviewed audit. →
Sitemap index
Also used: sitemap index file, XML sitemap index
Sitemap index is an XML file that lists multiple sitemap files so large or segmented URL sets can be submitted and maintained together.
Each child sitemap retains its own URL inventory, while the index provides one discovery and submission point for the collection.
Important limit: A sitemap index cannot contain page URLs directly, and its inclusion does not validate or canonicalize the child entries.
Related terms: XML sitemap, URL discovery, Canonical URL, Crawl budget
See how technical SEO evidence is examined in a human-reviewed audit. →
Soft 404
Also used: soft not found
Soft 404 is Google’s classification for a response that appears to represent a missing or unavailable page but does not return an appropriate 404 or 410 status.
Google compares the response status and content, so an empty, thin, or error-like page returning 200 can be excluded as though it were not found.
Important limit: The classification is platform-specific and does not replace checking the actual HTTP response and page purpose.
Related terms: HTTP status code, Indexing, Scaled content abuse, URL Inspection tool
See how technical SEO evidence is examined in a human-reviewed audit. →
Sponsored link qualification
Also used: rel sponsored
Sponsored link qualification is the use of rel="sponsored" to identify links created as part of advertising, sponsorship, or another compensated arrangement.
The relationship value is attached to the link so supporting systems can distinguish a paid relationship from an ordinary editorial reference.
Important limit: Sponsored link qualification does not replace a clear disclosure of the compensated relationship; link qualification and disclosure are distinct requirements.
Related terms: Nofollow, User-generated content link qualification, Paid media
See how technical SEO evidence is examined in a human-reviewed audit. →
Temporary redirect
Temporary redirect is an HTTP redirect indicating that the requested resource is available at another location for a limited or provisional period.
Status codes 302 and 307 communicate temporary routing, with 307 preserving the request method; the original URL generally remains the intended long-term address.
Important limit: A temporary redirect is not the right signal for a completed permanent migration, and prolonged use can leave intent unclear.
Related terms: Redirect, Permanent redirect, HTTP status code, Canonical URL
See how technical SEO evidence is examined in a human-reviewed audit. →
URL discovery
Also used: URL finding
URL discovery is the process by which a search system learns that a web address exists.
Google discovers URLs from pages it has already visited, links from known pages, and submitted sitemaps before it may crawl them.
Important limit: A discovered URL may remain uncrawled or unindexed, so discovery should not be reported as search inclusion.
Related terms: Crawling, Indexing, XML sitemap, Internal link
See how technical SEO evidence is examined in a human-reviewed audit. →
URL Inspection tool
Also used: URL Inspection
URL Inspection tool is a Google Search Console diagnostic that reports Google’s indexed information and test observations for a specified URL.
It can show the selected canonical, crawl details, indexing status, detected enhancements, and results from a live test, depending on access and availability.
Important limit: The report reflects Google’s available data, can lag current changes, and does not guarantee indexing after a live test or request.
Related terms: Googlebot, Indexing, Canonical URL, Search Console Performance report
See how technical SEO evidence is examined in a human-reviewed audit. →
User-generated content link qualification
Also used: rel ugc, UGC link attribute
User-generated content link qualification is the use of rel="ugc" for links placed within user-contributed material such as comments or forum posts.
The value labels the relationship at link level, helping supported systems distinguish user placement from the publisher's own editorial linking.
Important limit: User-generated content link qualification identifies a user-contributed link relationship; it is a qualification value and is distinct from a disclosure.
Related terms: Nofollow, Sponsored link qualification, Internal link, Search spam
See how technical SEO evidence is examined in a human-reviewed audit. →
Web crawler
Also used: crawler, spider
Web crawler is a program used to automatically discover and scan websites.
Crawler is a generic term; Google's documentation distinguishes crawlers that act automatically from fetchers that usually make a single request on behalf of a user.
Important limit: Different crawlers and fetchers can have different purposes and rules, so identifying a crawler does not establish how every automated client will behave.
Related terms: Googlebot, Crawling, URL discovery, Robots.txt
See how technical SEO evidence is examined in a human-reviewed audit. →
X-Robots-Tag
Also used: HTTP robots header, X-Robots header
X-Robots-Tag is an HTTP response header that supplies supported robots directives for the returned resource.
Because it is delivered in headers, it can apply indexing controls to non-HTML files such as PDFs as well as to HTML responses.
Important limit: Header rules can be misconfigured at the server or CDN layer, and blocked crawlers cannot use a directive they cannot retrieve.
Related terms: Robots meta tag, Noindex directive, HTTP status code
See how technical SEO evidence is examined in a human-reviewed audit. →
XML sitemap
Also used: XML sitemap file
XML sitemap is an XML file that lists canonical or otherwise intended URLs a site wants supporting search engines to discover.
A sitemap can also carry supported metadata, and submitted or discovered files help crawlers locate URLs that links alone may not expose efficiently.
Important limit: Listing a URL is a discovery signal, not a command to crawl, index, rank, or treat that URL as canonical.
Related terms: URL discovery, Sitemap index, Canonical URL, Internal link
See how technical SEO evidence is examined in a human-reviewed audit. →
Back to glossary navigation