Getting a New Website Indexed by Google: Technical Tips to Know

Ten years ago, the usual advice was to put your website online, share a few links on forums, and wait for Google to discover it. That approach has been outdated for a long time. Today, Google still needs to discover your pages, but it must also crawl them and decide whether to index them.
A new domain generally starts with very few signals to encourage search engines: no crawl history, few or no backlinks, and no established popularity. Google may discover its first pages fairly quickly, but that doesn't mean they will be crawled immediately, let alone indexed, especially in recent years. And contrary to what you might think, submitting a sitemap or clicking an indexing request button doesn't guarantee anything.
Discovery, crawling, and indexing: three distinct stages
Before appearing in search results, a page must go through several stages.
The first is discovery. Google needs to know that the URL exists. It can find it by following a link from another known page, checking an XML sitemap, or using information submitted through its own tools.
Next comes crawling. Googlebot sends an HTTP request to retrieve the page and its resources. The crawler analyzes the HTML, links, indexing directives, and any elements required to render the page.
Finally, the page goes through the indexing process. Google needs to extract its content, identify possible canonical versions, detect duplicates, and determine whether the document is valuable enough to be included in its index.
The process can be represented in a simplified way:
URL discovery
|
v
Queued for crawling
|
v
HTTP request by Googlebot
|
v
Content analysis and rendering
|
v
Canonical version selection
|
v
Indexing decision
|
v
Possible appearance in search resultsHowever, this diagram should not be interpreted as a strictly linear process. Google can revisit a URL, crawl it again, change its canonical selection, or reassess its indexing status. A crawled page is therefore not necessarily indexed, and an indexed page can subsequently be removed from the index.
This distinction already makes it easier to diagnose a problem: a URL that Google doesn't know about faces a different issue from a page that has been crawled but considered a duplicate or insufficient in terms of content.
Google Search Console: the first tool to set up
Google Search Console is essential for any website owner! When launching a site, you should create a property in Google Search Console and verify that you own it.
This allows you to submit an XML sitemap and check indexing reports. The sitemap should contain the canonical URLs you want to appear in Google, not every URL accessible on the server.
Here's a minimal example:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://pythonium.net/</loc>
<lastmod>2026-11-14</lastmod>
</url>
<url>
<loc>https://pythonium.net/pyrexp</loc>
<lastmod>2026-11-14</lastmod>
</url>
</urlset>You can also declare the sitemap directly in your `robots.txt` file:
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlThe sitemap helps Google discover URLs, but it doesn't force Google to crawl or index them immediately. It's a discovery signal, not a command.
URL Inspection and its API
Search Console also provides a URL Inspection tool. It lets you check what Google knows about a page, including its last crawl date, whether it's in the index, the selected canonical URL, and any detected issues.
To automate these checks, Google provides the URL Inspection API. It allows you to retrieve the indexing status of a URL through an HTTP POST request:
POST https://searchconsole.googleapis.com/v1/urlInspection/index:inspectWith a JSON body like this:
{
"inspectionUrl": "https://example.com/documentation",
"siteUrl": "https://example.com/",
"languageCode": "fr-FR"
}The response contains an `inspectionResult` object with various details about the URL's status in Google's index.
This API does not allow you to request indexing for a page. It is only used to check its status in the index. Manual indexing requests are a separate feature in the Search Console interface, accessible through the URL Inspection tool.
The API also has fairly generous quotas for a small project: 2,000 requests per day per property, with a limit of 600 requests per minute. This makes it useful for building an automated monitoring tool, but not for forcing Google to crawl a list of URLs.
Crawl budget: why Google doesn't crawl every page immediately
Google doesn't have unlimited resources to crawl the Web. It has to distribute its crawlers across a huge number of websites and URLs. This is what is known as the *crawl budget*.
Google describes this budget as the result of two main factors: the crawl capacity limit and crawl demand.
The crawl capacity limit refers to the number of requests Google can make to a website without risking overloading its server. If the server responds slowly, regularly encounters errors, or becomes unavailable, Google may reduce its crawling frequency.
Crawl demand depends on factors such as URL popularity, update frequency, perceived page quality, and the level of interest Google assigns to them. A page that is regularly updated and referenced by other websites may therefore be crawled more often than an isolated page that rarely changes.
On some of my websites that are still relatively unpopular in Google's eyes, I sometimes have to wait several weeks before a new page gets indexed. In these cases, submitting a manual indexing request through Google Search Console can save a lot of time 🙂
Crawl budget is mainly an issue for large or frequently updated websites. Google points out that smaller websites, with relatively few pages that rarely change, generally don't need a specific crawl budget optimization strategy.
For a new project with a few dozen or a few hundred pages, the goal should therefore not be to artificially optimize the crawl budget, but to make Google's job easier.
Technical points to check
A technically sound website doesn't guarantee more frequent crawling, but it prevents wasting Googlebot's resources and creating unnecessary obstacles. Here are the main things to check:
- HTTP response time: avoid unnecessarily slow server requests. An overloaded or unstable server can slow down crawling.
- HTTP status codes: accessible pages should return an appropriate status, usually `200`. Deleted pages should return `404` or `410`, rather than an error page that still returns `200`.
- **Redirects:** avoid redirect chains and use direct redirects to the final URL whenever possible.
- Internal links: every important page should be accessible through at least one other page on the website. A URL that only appears in a sitemap, with no internal links, is harder to discover through the site's navigation structure.
- Canonical URLs: correctly declare the preferred version of a page using `rel="canonical"` when multiple URLs can display identical or very similar content.
- Robots.txt: make sure the file doesn't accidentally block pages or resources required for rendering.
- Indexing directives: check `noindex` tags and `X-Robots-Tag` HTTP headers, which can prevent pages from being indexed.
- XML sitemap: only include relevant, accessible URLs that are intended to be indexed.
JavaScript, rendering, and accessible content
Another point worth paying attention to is JavaScript rendering. Googlebot can execute JavaScript and index applications built with frameworks such as React, Vue, or Angular. This doesn't mean that an SPA is automatically bad for SEO.
However, the process can be more complex than with an HTML page that contains its content directly. Google first needs to retrieve the document and may then have to execute the scripts required to obtain the final content. This rendering can be delayed, and some JavaScript errors can prevent content from being extracted correctly.
Let's take a very simple example:
<div id="app"></div>
<script src="/assets/app.js"></script>If the page content is entirely injected on the client side, the initial HTML contains practically no useful information about the subject of the page. If the script fails to load, a remote API doesn't respond, or a JavaScript error occurs, the expected content may never be rendered.
A more robust approach is to use server-side rendering (SSR), static site generation (SSG), or pre-rendering when it fits the project's architecture.
The goal is to provide initial HTML that already contains the essential elements: title, main content, internal links, and metadata. JavaScript can then enhance the user experience without being essential to understanding the page.
To check the result, simply opening the website in your browser isn't enough. You should also inspect the HTML returned by the server using a command such as:
curl -L https://example.com/documentationThis lets you check whether the main content is actually present in the HTTP response, independently of JavaScript execution.
External links still matter for a new domain
A new domain generally has no history or established popularity. Getting a few links from pages already known to search engines can therefore help them discover its URLs.
A link published in an article, technical documentation, a GitHub repository, or a relevant discussion can allow a crawler to discover a page it didn't know about before.
However, discovery and ranking should not be confused. A backlink can help Google find a URL and contribute to its understanding of a page's popularity. It doesn't guarantee indexing or an immediate improvement in search rankings.
Link quality and relevance matter more than simply counting links. Building artificial links in directories unrelated to the project or posting comments solely to get backlinks is not a particularly useful strategy. This hasn't been relevant for at least ten years.
For a tool aimed at developers, a few contextual links from technical articles, documentation, or community resources are often much more relevant than dozens of generic links from unrelated blogs.
How to diagnose a page that isn't indexed
Search Console provides several useful statuses to help understand why a page isn't appearing in Google. Not all of them indicate a technical problem.
Some of the most common cases include:
| Status | Meaning | Suggested action |
|---|---|---|
| Discovered – currently not indexed | Google knows the URL but hasn't crawled it yet | Check internal links and wait for another crawl |
| Crawled – currently not indexed | Google retrieved the page but didn't include it in its index | Review the content, duplicates, and added value |
| Page with redirect | The URL redirects to another address | Check the destination and canonical URLs |
Excluded by a noindex tag | A directive prevents indexing | Check whether this exclusion is intentional |
| Duplicate, Google chose a different canonical URL | Google considers another URL to be the preferred version | Check canonical URLs and similar content |
| Soft 404 | The page appears empty or non-existent despite returning HTTP 200 | Fix the content or return a genuine 404 |
The distinction between "Discovered, currently not indexed" and "Crawled, currently not indexed" is particularly interesting.
In the first case, Google knows the URL but hasn't yet made a request to retrieve its content. In the second, the crawler has already visited the page, but it wasn't selected for the index. Submitting another indexing request won't necessarily fix this second problem: you first need to understand why the page wasn't selected.
It's also best not to rely solely on a `site:example.com/my-page` search in Google. This command can provide an indication, but it isn't an exhaustive diagnostic tool. Some pages that are indexed may not appear in these results.
Conclusion
There you go, we've covered the basics of Google indexing 🙂
But today, we also have to deal with competition from LLMs and, apparently, get into GEO (Generative Engine Optimization). But do these AI platforms actually bring traffic? For now, AI platforms send me 4% of the traffic I get from Google.
We'll see how all this evolves over the next few years.





