Here's a question I like to ask teams during QA planning: when was the last time you looked at your own sitemap?
For most products, the answer is "never." Teams maintain hand-written test lists — a shared spreadsheet, a Notion doc, a markdown file that's a year out of date — and they test what they remember. Meanwhile, the production site grows. Marketing adds landing pages. Someone ships a checkout flow that's only linked from a campaign page. A settings page gets a second tab nobody thought to cover.
The gap between "what you think your app looks like" and "what your app actually is" is where bugs live. That's the problem automatic DOM discovery narrows, and it's why I made crawling the foundation of Qalizer rather than an afterthought.
Why hand-written test lists rot
I'm not anti-documentation. I'm anti-fiction. And a hand-maintained test list is a work of fiction the moment it's written, because software changes faster than people update lists.
Consider the classic failure mode:
- You launch with ten pages and write tests for all ten.
- Three months later you add a pricing page. Nobody adds it to the test list.
- Six months later you refactor the nav, which links to a new
/account/billingpage. - The billing page has a bug. Your tests never look at it, because it was never on the list.
- A customer finds the bug. Now it's an incident, not a test failure.
The fix isn't "be more disciplined." The fix is to stop maintaining the list by hand altogether. The pages are right there in your DOM — let a crawler enumerate them.
The three crawl modes
When I built this, I realized "crawl my site" means different things at different moments. So Qalizer has three modes, each doing measurably different work — and each with a page cap, because an unbounded crawl is a good way to get your crawler banned and your credits burned.
First, the numbers, because they are the part people misremember. Each mode has a default page cap, and all three share a hard ceiling of 60 pages; the crawl can also widen its bound toward the site's estimated page count so a larger site still gets covered. A cap means a crawl is a sample, never a guarantee of full coverage.
FastCrawl is the quick pass: it follows links, records what it finds and captures console errors where it can. Its default cap is 10 pages. Use it when you want to know "what changed since last week", or when you're not sure a site is worth a deeper look. It's the cheapest mode, and for simple, mostly-static sites it's usually enough — but ten pages of a fifty-page site is a sample, and you should read it as one.
DeterministicPass is the browser pass without AI. Playwright visits same-domain links, fills forms, clicks buttons, captures console errors and broken elements, and takes DOM snapshots — with a default cap of 25 pages. "Deterministic" here means no AI in the loop: traversal is decided by the link graph and the page's DOM rather than by a model's judgement, which makes it easier to reason about and repeat. The same starting point and the same link graph should produce the same walk. It is not a promise that two runs on a live site produce identical output — a site that changes between runs (a new banner, a rotating carousel, an A/B test) changes what the crawler sees — and page caps mean a large site is always sampled, never exhausted. One distinction worth knowing before you rely on it: this flat pass detects a login wall through its page corpus, producing a needs-credentials gate scenario you can see and act on, but it does not model stateful multi-step gates. That is DeepCrawl's job.
DeepCrawl is the completionist. It runs the stateful explorer: multi-pass element discovery, form interaction, wizard walking, and AI-assisted gate-form analysis. The default is 40 pages, so "deeper" means more interaction per page and more discovery through interaction — not "every route on the internet". Because it does more work, expect it to cost more credits than the other two; that's the complexity pricing doing its job. Use it when you're launching a big release, or when you've inherited a legacy app and want to know how much surface area you're actually shipping.
The point isn't that one is "best." The point is that each mode matches a different moment in your workflow: a quick check, a repeatable browser pass, or the deepest walk you can get inside a page budget.
Why sitemap + DOM-tree beats a test list
A sitemap — whether you generate it from the crawler or read an existing sitemap.xml — tells you what pages should exist, from the site's own perspective. A DOM tree tells you what actually renders.
That's a meaningful difference. I've crawled sites where sitemap.xml was stale and pointed at removed pages. I've also seen new routes appear in a crawl before anyone remembered to update the sitemap. When you look at both signals, you get a more truthful picture than either one alone:
- Sitemap gives intent: "the maintainers believe these pages exist."
- DOM tree gives reality: "these pages rendered and were reachable when the crawler looked."
- The intersection is your confident core.
- The DOM-minus-sitemap difference is your blind spot — pages you're shipping but forgot to document.
To be precise about what "blind spot" means: it's the set of linked or reachable pages the sitemap doesn't mention. Routes that are never linked, are behind auth, or only appear after an interaction the crawl didn't perform won't show up in the diff either. The diff is evidence, not a completeness proof.
Still, blind-spot discovery is the highest-value output of crawling. The pages you forgot are the ones nobody tests.
Page classification drives smarter tests
Discovering a URL is only step one. A URL tells you that a page exists; it doesn't tell you what kind of page it is, and therefore what kind of testing it needs.
That's where page classification comes in, and it's what makes AI-generated tests useful instead of generic. Crawled pages are grouped into risk categories before generation:
- Auth pages — login, sign-up, forgot-password. These need different handling: credential management, positive and negative login attempts, session behavior.
- Forms — pages with input elements. These need data injection, validation testing, empty-submit checks.
- Purchase flows — cart, checkout, payment. The riskiest pages on your site, deserving the deepest negative-test coverage.
- Navigational pages — the rest. Mostly they need to render correctly, so they're good visual-regression candidates.
The grouping is rule-based: URLs and element inventories are matched against keyword sets (login/sign-in/register, payment/checkout/cart, form/submit/input, and so on). It's a heuristic, not a human review, so it will occasionally misfile a page with an ambiguous name. The upside is that you can see why a page landed in a category and correct it, instead of wondering what a model thought.
When you classify pages before generating tests, you stop writing the same generic "it loads" test for everything and instead generate contextually appropriate tests. A purchase page gets edge cases around empty carts and failed payments. A form gets negative scenarios around empty submits and invalid values. A login page gets auth-specific scenarios.
That classification is also what makes crawling feel like a foundation rather than a menu of trees. It's the difference between "here are 40 URLs" and "here are 40 URLs, grouped by risk, with the highest-risk pages already prioritized for testing."
What this surfaced in my own projects
The two examples below are hypothetical illustrations of the pattern, not records of specific incidents. They describe the kind of thing a crawl is designed to look at; treat them as a worked example rather than as a case study.
I've run this flow on my own apps, and the pattern is consistent: I think I know my site, and the crawl shows me something I had forgotten.
Imagine a project where the crawl surfaced a legacy admin route left enabled from an early prototype. It was linked from an old settings menu, wasn't in the test list, and still accepted requests without authentication. It would have stayed live indefinitely had the crawl not walked the DOM and found it. That kind of discovery is what makes the exercise worth doing.
Another common shape: the form group catches pages I hadn't thought of as high-risk. Running generated form tests on them surfaces a broken email-validation edge case. The crawler didn't invent the bug — it just made sure I was actually looking at the page where the bug lived.
Neither of those is a promise that your first crawl finds a security hole. It's the more modest, repeatable effect: the crawl looks at pages a human checklist skipped.
Start by crawling, not by listing
If you take one thing from this: stop starting your QA effort with a spreadsheet. Start it with a crawl. Let the machine enumerate the surface area it can actually reach, classify it, and only then decide what deserves manual, hand-crafted attention.
The pages you forgot are usually still linked somewhere. A crawl finds them and gives the generated tests something concrete to work from — before a customer does. That's the entire philosophy behind Qalizer's crawler-first design, and it's the biggest reason my side projects now start a release with a crawl rather than with hopeful fingers-crossed debugging.
If you've got an app and you're not sure you actually know what's on it, run a crawl and see what it finds — you might be surprised what you've been shipping blind. Crawls cost credits, but a new account starts with 100 signup credits, and a FastCrawl on a small site is priced as the cheapest tier, so a first pass fits comfortably inside that.