Glossary

What 401 means, why it is about authentication rather than permission, how it differs from 403, and what the WWW-Authenticate header is for.

What 429 means, why Retry-After is the field that makes it usable, and why a crawler receiving one behaves differently from a browser.

What the Retry-After header does on a 429 and a 503, the two formats it accepts, and why omitting it turns every client into a guess.

Every HTTP response opens with a three-digit code. What the 1xx to 5xx classes mean, which codes a crawl surfaces most, and why the code is not the page.

403 means the server understood the request and refuses to fulfil it. How it differs from 401, and why a link checker reporting 403 is often a bot filter, not a dead page.

410 tells clients a resource was deliberately removed and is not coming back. How it differs from 404 in practice, and when the distinction is worth making.

500 means the server hit a fault it could not handle. Why it is deliberately uninformative, how it differs from 502 and 504, and where the real cause is recorded.

503 says the server is temporarily unable to serve. Why it is the right code for maintenance, what Retry-After does, and how crawlers treat a long outage.

A redirect loop is a cycle no client can escape. What causes them, why they are usually a conflict between two correct rules, and how to trace one.

404 means the server has no representation for the requested URL. How it differs from 410 and from a soft 404, and when returning it is the right behaviour.

A MIME type declares what a file actually is. How Content-Type and charset work, what happens when a browser has to guess, and why the declaration matters.

An A record is the entry that turns a hostname into an IPv4 address. How it works, how it differs from AAAA and CNAME, and what multiple A records do.

A CNAME makes one hostname an alias for another. Why it cannot coexist with other records, why the apex is forbidden, and what chains cost.

MX records name the servers that accept mail for a domain, in priority order. How preference values work, and why redundancy matters more than order.

TXT records hold arbitrary text, and by convention carry SPF, DKIM, DMARC and domain verification. Why size limits and duplicates cause real failures.

A CAA record names the certificate authorities allowed to issue for a domain. What authorities must do with it, and what it does not protect against.

An SRV record names the host and port of a specific service, with priority and weight. Where it is used, and why the web does not use it.

A TTL tells resolvers how long to keep a DNS answer. How it governs the cost of a change, why lowering it before a migration works, and its limits.

Nameservers hold and answer for a domain's DNS records. How delegation works, why the registrar's list must match the zone, and what redundancy buys.

Reverse DNS maps an IP address back to a hostname through a PTR record. Who controls it, why mail servers check it, and why websites do not need it.

DNSSEC adds cryptographic signatures to DNS records so a resolver can prove an answer is genuine. What it protects, and what it deliberately does not.

Propagation is the expiry of cached answers, not a distribution process. What actually delays a change, and how to see the state from several resolvers.

IPv6 addresses are 128 bits and reached through AAAA records. What dual-stack means, how clients choose, and what publishing AAAA commits you to.

The apex is the domain itself, without a subdomain. Why it cannot be a CNAME, why it must carry NS and SOA records, and what that means in practice.

A subdomain is a name below the apex. How it differs from a subdirectory for search, what it shares with the parent, and what it does not.

A TLD is the final part of a domain name. The categories, who runs them, which ones carry restrictions, and what a choice of TLD does not determine.

WHOIS reports a domain's registration, expiry and registrar. What privacy redaction removed, what RDAP replaces it with, and what remains reliable.

An IDN contains non-ASCII characters and is encoded as Punycode for DNS. How the two forms relate, and why look-alike characters are a real risk.

An ASN identifies the network that announces a block of IP addresses. What it reveals about hosting, and what it does not say about a site.

A broken link points at a target that does not answer. Which codes mean broken, which are false alarms, and why a 200 does not guarantee correctness.

What the Vary header declares, why omitting it lets a shared cache serve the wrong variant, and why listing too much destroys cache efficiency.

How a server picks a representation from what the client says it accepts, where it is used in practice, and why it makes an audit result client-dependent.

How a browser revalidates a cached response with If-None-Match or If-Modified-Since, why a 304 still costs a round trip, and when to avoid one entirely.

How persistent connections and multiplexing changed the cost of many small resources, and what still makes a high request count expensive.

What a query parameter does, why parameters multiply crawlable URLs, and which ones change content versus which merely track a visit.

Headers carry everything about a request or response that is not the body: caching, security, content type, cookies. How they work and how to read them.

The User-Agent header is how browsers, crawlers and scripts announce what they are. Why it is unreliable, and how robots.txt depends on it anyway.

301 tells clients a resource has permanently moved. Why browsers cache it aggressively, why that makes it hard to undo, and when 308 is the better choice.

302 says a resource is temporarily elsewhere. Why a 302 left in place for years keeps the wrong URL indexed, and when it is genuinely the right code.

A redirect chain is a URL reaching its destination through several hops. What counts as a chain, how it differs from a loop, and why length matters.