SEO
Technical SEO Checklist: Find Crawl and Indexing Problems
Check crawl access, indexing controls, redirects and canonicals. Use a practical technical SEO checklist to prioritize problems and verify the fixes.
A technical audit can list hundreds of warnings without making the next decision clear. The important question is which finding affects a page the business needs, what the intended behavior is and how a fix will be verified.
Use this checklist to organize an review. It is not a promise that passing every item guarantees search visibility.
Which pages should a technical SEO check cover?
Start with the website's important public pages and a sample of each distinct page type. Include the pages closest to enquiries or sales, then expand the scope when a finding suggests a wider issue. Keep private pages and deliberately excluded URLs in a separate category. An audit is more useful when it explains scope than when it presents every URL as equally important.
For a small service website, begin with the homepage, service pages and contact route. For a shop, add categories, products and relevant navigation paths. Record why each type is included.
Are the important URLs returning the expected response?
Check what the server returns for the actual address, including any redirect destination. Compare that behavior with the page's purpose. A public page, a moved page and a genuinely missing page do not have the same intended response. Keep the status and destination in the record so the person reviewing the audit can understand the problem without recreating every check.
Google's documentation describes how HTTP status codes affect its handling of pages. Use that reference when assessing a technical response rather than assigning meaning from an audit tool's color alone. HTTP status guidance
Are crawl and indexing controls doing what you intend?
Review access rules and indexing instructions as separate controls. A crawl restriction tells a crawler about access, while a noindex instruction concerns indexing and must be seen to be processed. Decide what the business wants for the page before changing either setting. Removing every restriction is not a sensible fix when some resources should remain private or outside search. Robots guidance, noindex guidance
Record the live configuration, not only what a settings screen appears to say. If an environment variable controls indexing, verify the built production response after an approved change.
Are redirects and canonical signals consistent?
Identify which public URL should represent a piece of content, then check whether redirects, internal links and canonical declarations support that choice. A canonical is a signal about the preferred representative, not a command that makes every other URL disappear. Check conflicting signals before adding another rule, and avoid pointing unrelated pages at one destination merely to simplify the audit. Canonical guidance
For a redesign involving URL changes, use the separate migration checklist. That work needs an old-to-new inventory, not just a scan of the current site.
Can search engines access the important page content?
Inspect the public page and use the available diagnostic tools to compare what is delivered with what you intend to publish. Record missing text, failed resources or unexpected account requirements. Do not assume that an attractive browser view proves every request receives the same content. At the same time, avoid declaring all JavaScript a problem without evidence about the actual implementation.
The URL Inspection tool can support a page-level review. Keep the test date and distinguish current live checks from details about a previously crawled version.
How should you prioritize and verify fixes?
Connect each finding to an affected page, business consequence and proposed action. Assign someone to make the change and define a retest before work begins. Fixing a warning is not complete when a setting has merely been saved. Verify the resulting public behavior, check for unintended effects and keep enough evidence for the next person to understand what changed and why.
| Field | Example of a useful entry |
|---|---|
| Affected page | Exact public URL |
| Observation | Response or instruction actually found |
| Intended behavior | What the page should do |
| Owner | Person responsible for the approved fix |
| Retest | The check that demonstrates the intended behavior |
This is a proposed worksheet, not a client audit. Use technical SEO support if you need help reviewing a finding before altering the site.
Frequently asked questions
Is robots.txt the same as a noindex instruction?
No. They serve different purposes, so a change should be based on the intended treatment of the page. Do not assume that blocking access gives a crawler the opportunity to read a noindex instruction on that same page. Ask for an explanation of the current rules before combining them, and verify the public result after any approved technical change.
Does every excluded URL need fixing?
No. Start by deciding whether the URL should be a public search destination. An intentionally private page, a redundant variation and an important service page deserve different treatment. An audit should distinguish those cases rather than count every exclusion as a failure. Review the reason and purpose together, then document which exclusions are expected and which require review.
Can an automated audit prove that a website is technically perfect?
I would not describe a website that way on the basis of one tool. An automated scan has a scope and a set of checks, and its findings still need reading. Use it to locate possible issues, then verify relevant ones against intended behavior. Report what passed, what was not tested and any remaining uncertainty instead of making an absolute claim.
Should private or duplicate pages be added to a sitemap?
Choose sitemap entries according to the public canonical pages you want to present for discovery. Do not add private resources simply to make a count match the number of URLs a crawler found. Review variations and intended exclusions before editing the list. Keep the sitemap consistent with the website's actual publication decisions and verify that its listed destinations work.
What should I record before changing technical settings?
Record the affected URLs, current behavior, intended result and the setting you plan to change. Keep an appropriate backup or recovery path with the person responsible for the website. Decide how the change will be checked and who will review unexpected effects. This makes the work easier to verify and avoids relying on memory if a problem appears later.