Link Checking Your Own Site Without Drowning in False Positives

Broken internal links are the most common defect on a content site and the least visible. Nothing alerts you; a crawler finds them eventually; a reader hits a 404 and leaves.

A link checker is the obvious fix, and the obvious fix fails in a predictable way. External URLs fail for reasons that have nothing to do with you — rate limits, bot protection, transient outages — and a checker that flags all of them produces a report nobody reads. Within a month it is either disabled or ignored, and both are the same outcome.

The two decisions that make it useful

Everything about whether this works comes down to two configuration choices:

Which status codes do you accept? Redirects and bot-protection responses are not failures.

Which links do you check, and when? Internal links should block a deploy. External links should not.

Get those right and the check stays trusted. Get them wrong and it becomes noise.

What the tools document

Tool	Language	Relevant documented behaviour
lychee	Rust	Accept-code ranges, per-host rate limiting, retry policy, cache file
linkinator	JavaScript	Per-code actions (ok/warn/skip/error), retry on 429 honouring retry-after, built-in bot-protection handling
HTMLProofer	Ruby	Internal links, internal anchors, external links, HTTPS, SRI

All three are permissively licensed and all three can run in CI. The difference is in how much of the noise handling is built in versus configured.

lychee: the defaults that matter

lychee’s documented defaults are worth reading before you write a config file, because several are the source of the noise problem.

Accepted status codes. The default accept range is the success codes plus informational ones. This means 403 and 429 fail by default. That is correct behaviour for the tool and wrong behaviour for the internet, where a large share of legitimate URLs return 403 to anything that looks like a bot.

Retries: three by default, with a one-second wait between attempts and a 20-second timeout. That is a reasonable baseline, and on hosts that rate-limit aggressively it is not enough — which is what the accept list is for.

Rate limiting is per-host: a default of 10 concurrent requests per host, with a 50 ms interval between requests that auto-increases when the target throttles you. That adaptive behaviour is the reason lychee is less disruptive to other people’s servers than a naive checker.

Caching. The cache file is only written if you pass the cache flag — the documentation states no data is stored otherwise. The default cache age is one day, and you can configure which status codes are excluded from the cache so that failures are re-tested on every run rather than being remembered.

Exclusions are flexible: a regex exclude list, an include list, a .lycheeignore file, and path-level exclusions.

linkinator: the one that handles bot protection for you

linkinator has the most relevant documented behaviour for this specific problem, and it is worth noting because it does something the others leave to configuration.

It documents handling of Cloudflare’s 403 with a cf-mitigated header and LinkedIn’s 999 status as skipped rather than failed. That is a tool author who has run this against the real web and decided that these responses are not the user’s problem.

It also handles 429 by honouring the retry-after header, and has a separate retry setting for 5xx errors — a distinction worth making, because a 429 means “slow down” and a 500 means “try again,” and treating them identically wastes requests.

Configuration is a JSON file, --skip takes a repeatable regex, and recursion is scoped to the same root domain.

One documented gap: there is no single flag for “internal links only.” The documented approach is to combine recursion with a negative-lookahead skip pattern, which works and is less obvious than it should be.

The classification that makes the report readable

Given those tools, here is the classification I would use, and the reasoning for each row:

Status	Treat as	Why
200	Pass	Obvious
301, 302, 307, 308	Pass, with a warning	The link works, but consider updating it to the destination
403	Pass	Usually bot protection, not a missing page
429	Pass, retry	Rate limiting; try again later
999	Pass	Documented by one tool as a known non-failure
404, 410	Fail	A real broken link
500, 502, 503	Warning, retry	Often transient; re-test before declaring failure
Timeout	Warning, retry	Same
DNS failure	Fail	The hostname is gone

The redirect row deserves a note. A 301 is technically a working link, so failing on it creates noise — but it is also a defect worth knowing about, because every visit through it costs an extra round trip. Warn rather than fail, and clean them up on a slow week.

The scheduled-workflow caveat that will silently disable your check

If you run the external-link check on a schedule in GitHub Actions — which is the right place for it — there is a documented behaviour that will turn it off without telling you:

“In a public repository, scheduled workflows are automatically disabled when no repository activity has occurred in 60 days.”

The failure mode is perfect: the repository is a static site, so no activity is exactly what a healthy month looks like. The check stops running, nothing reports that it stopped, and you find out when a reader emails you about a dead link.

Two other documented details about scheduled workflows:

• They are delayed during periods of high load. The documentation suggests picking a time away from the top of the hour to reduce the chance of a long delay.
• The workflow file must exist on the default branch for the schedule to trigger at all. A schedule defined only on a feature branch does nothing.

If your repository is quiet by nature — which a content site is — the mitigation is a periodic trivial commit, or moving the check to a service that is not GitHub Actions.

Configuration, in the order that matters

1. Internal links block the deploy. They are deterministic, they are your responsibility, and a failure means your site has a defect. Gate on this.

2. Anchor checks too, if your tool supports them. A link to #section that no longer exists is a broken link that returns 200. This is the check most people skip and the one that catches the most real problems on a long-lived content site, because headings get renamed and anchors get orphaned.

3. External links run on a schedule. Not per-commit. The internet is not deterministic and your build should not depend on it.

4. The accept list is a maintenance item. Every genuinely dead external URL that keeps returning 403 teaches you something; every one you add to the ignore list is a small debt. Review the list quarterly rather than growing it forever.

5. Cache in CI. A cache file between runs is the difference between a check that takes a minute and one that hammers every host you have ever linked to.

When not to bother

• A small site with few outbound links. Fifty links checked by hand once a quarter is cheaper than a pipeline.
• Your build already fails on link problems. Some generators do this. Check before adding a second tool that reports the same thing.
• A young site. Run it, fix what it finds, and then only wire it into CI when the failure rate is near zero — otherwise you are just automating a report you will not read.

The metric to watch is not how many problems the checker finds. It is how often you act on what it finds. A check with a 100% action rate on a handful of items is worth more than a check with 40 items you scroll past.