SEO
Technical SEO Checklist: Find Crawl and Indexing Problems
Check crawl access, indexing controls, redirects and canonicals. Use a practical technical SEO checklist to prioritize problems and verify the fixes.
A technical audit can list hundreds of warnings without making the next decision clear. The important question is which finding affects a page the business needs, what the intended behavior is and how a fix will be verified.
Use this checklist to organize a review. It is not a promise that passing every item guarantees search visibility.
Which pages should a technical SEO check cover?
Start with the website's important public pages and a sample of each distinct page type. Include the pages closest to enquiries or sales, then expand the scope when a finding suggests a wider issue. Keep private pages and deliberately excluded URLs in a separate category. An audit is more useful when it explains scope than when it presents every URL as equally important.
For a small service website, begin with the homepage, service pages and contact route. For a shop, add categories, products and relevant navigation paths. Record why each type is included.
Simplified overview. Google may revisit or skip stages; access and sitemap inclusion do not guarantee indexing or ranking.
- Publish
A useful public page is available.
- Discover
Links and a sitemap help reveal its address.
- Crawl
A crawler fetches the page.
- Evaluate
Google evaluates content and preferred URLs.
- Search
An indexed page may be shown for a relevant query.
Are the important URLs returning the expected response?
Ask your developer to check the response code for each important page. A 200 normally means the request succeeded; a 404 means the page was not found. A redirect sends the visitor to another address. Record the final destination too: a working response is not enough if a customer lands on the wrong page.
Google's documentation describes how HTTP status codes affect its handling of pages. Use that reference when assessing a technical response rather than assigning meaning from an audit tool's color alone. HTTP status guidance
| Finding | Why the owner should care | Ask the developer to check |
|---|---|---|
| Main service page returns 404 | Customers cannot read the offer | Restore it or identify a relevant replacement |
| Public page carries noindex | It asks search engines to leave it out | Whether that instruction is deliberate |
| Product links lead through several redirects | The path is needlessly complicated | Link directly to the correct final URL |
| Page points to an unrelated canonical | It sends a conflicting signal about the page | The intended preferred URL and visible content |
Are crawl and indexing controls doing what you intend?
Crawling means fetching a page. Indexing means processing it for possible use in search. The robots.txt file gives crawlers access rules; a noindex instruction asks search engines not to index a page. These controls are different. Google must be able to fetch a page to read its noindex instruction. Google robots guidance, noindex guidance
Record the live configuration, not only what a settings screen appears to say. If an environment variable controls indexing, verify the built production response after an approved change.
Are redirects and canonical signals consistent?
A canonical is a signal that identifies the preferred URL when pages are the same or very similar. For example, a shop may have several addresses for one product. Ask your developer which address should appear in search, then check that links and redirects support that choice. Do not point unrelated pages to the homepage. Google canonical guidance
For a redesign involving URL changes, use the separate migration checklist. That work needs an old-to-new inventory, not just a scan of the current site.
Can search engines access the important page content?
Inspect the public page and use the available diagnostic tools to compare what is delivered with what you intend to publish. Record missing text, failed resources or unexpected account requirements. Do not assume that an attractive browser view proves every request receives the same content. At the same time, avoid declaring all JavaScript a problem without evidence about the actual implementation.
The URL Inspection tool can support a page-level review. Keep the test date and distinguish current live checks from details about a previously crawled version.
How should you prioritize and verify fixes?
Connect each finding to an affected page, business consequence and proposed action. Assign someone to make the change and define a retest before work begins. Fixing a warning is not complete when a setting has merely been saved. Verify the resulting public behavior, check for unintended effects and keep enough evidence for the next person to understand what changed and why.
| Field | Example of a useful entry |
|---|---|
| Affected page | Exact public URL |
| Observation | Response or instruction actually found |
| Intended behavior | What the page should do |
| Owner | Person responsible for the approved fix |
| Retest | The check that demonstrates the intended behavior |
This is a proposed worksheet, not a client audit. Use technical SEO support if you need help reviewing a finding before altering the site.
See my mobile PageSpeed audit snapshot for an example of lab performance evidence. A single score is a test result; it should sit alongside crawl, indexing and real-user checks.
Frequently asked questions
Is robots.txt the same as a noindex instruction?
No. They serve different purposes, so a change should be based on the intended treatment of the page. Do not assume that blocking access gives a crawler the opportunity to read a noindex instruction on that same page. Ask for an explanation of the current rules before combining them, and verify the public result after any approved technical change.
Does every excluded URL need fixing?
No. Start by deciding whether the URL should be a public search destination. An intentionally private page, a redundant variation and an important service page deserve different treatment. An audit should distinguish those cases rather than count every exclusion as a failure. Review the reason and purpose together, then document which exclusions are expected and which require review.
Can an automated audit prove that a website is technically perfect?
I would not describe a website that way on the basis of one tool. An automated scan has a scope and a set of checks, and its findings still need interpretation. Use it to locate possible issues, then verify relevant ones against intended behavior. Report what passed, what was not tested and any remaining uncertainty instead of making an absolute claim.
Should private or duplicate pages be added to a sitemap?
Choose sitemap entries according to the public canonical pages you want to present for discovery. Do not add private resources simply to make a count match the number of URLs a crawler found. Review variations and intended exclusions before editing the list. Keep the sitemap consistent with the website's actual publication decisions and verify that its listed destinations work.
What should I record before changing technical settings?
Record the affected URLs, current behavior, intended result and the setting you plan to change. Keep an appropriate backup or recovery path with the person responsible for the website. Decide how the change will be checked and who will review unexpected effects. This makes the work easier to verify and avoids relying on memory if a problem appears later.
Put it into practice
Illustrative example · not a client result
A booking page returns a 404 error after a site update. Restoring the right page matters more to customers than clearing a harmless warning on an unused filter URL.
Your next step
Ask for a finding with an affected URL, a proposed fix and a named person who will retest it.
Key takeaways
- A technical audit should tell you what is broken, why it matters and who will fix it.
- Ask for a finding with an affected URL, a proposed fix and a named person who will retest it.