Indexing control in bulk site creation
Indexing control shows whether search engines have accepted the sites produced by bulk site creation. Best Boost Partners generates sitemaps, submits pages, monitors how many pages of each site are in the index, finds the sites that dropped out and fixes the cause. The work suits owners of networks, affiliate projects and regional sites. You receive an indexing table for every site and a log of problems with a description of the measures taken.
What exactly do we check and do in indexing control?
Sitemaps and robots.txt
The generator creates a sitemap and a robots.txt file for every site and updates them at each build. We check that the sitemap contains only pages with a normal server response that are open for indexing.
Submitting pages
The sites are added to Google Search Console, and the sitemaps are submitted there. New and changed pages are sent for crawling in portions, within the limits of the search engine.
Monitoring the index
For every site we record on a schedule how many pages were submitted and how many are in the index. The data comes from Google Search Console and from checks of the search results.
Server logs
We use server logs to see whether search engine bots visit a site and which pages they request. A site that bots do not visit is handled differently from a site that is crawled and still left out of the index.
Finding sites that dropped out
The monitoring flags a site when the number of its indexed pages falls or stays at zero. Flagged sites go into a separate list with the date of the change.
Fixing the cause
For each flagged site we look for the cause: a technical block, duplicates or thin content, a problematic domain history, a manual action. The fix depends on the cause: from a config correction to rewritten content or a replaced domain.
What do you get from indexing control?
- An indexing table: submitted and indexed pages for every site, with dates.
- Google Search Console properties for the sites, with access for your team.
- A list of sites that dropped out of the index or never entered it, with the cause found for each.
- A log of fixes: what was changed on which site and how the index reacted.
- A monitoring script or schedule that your team keeps using after handover.
When is indexing control needed?
- A batch of sites has been launched, and you do not know how many of them are in the index.
- Some sites of a network have stopped having an effect, and you suspect that they dropped out of the index.
- New pages of your sites reach search slowly or do not reach it at all.
- You are preparing the next batch and want to know what went wrong with the previous one.
Frequently asked questions
Can you guarantee that every page of every site will be indexed?
No. The search engine decides what to index, and with sites built in bulk some pages often stay outside the index. Indexing control shows which sites and pages are missing from the index and why, so that the cause can be removed where that is possible.
Why do sites built in bulk drop out of the index?
The most frequent cause is thin or near-identical content across the sites, technical errors repeated by the template, or a bad domain history. Search engines also act against whole groups of sites when they see scaled content abuse. We determine the cause for each site separately, because the fix will differ.
What do you do with a site that cannot be returned to the index?
A site that has not returned to the index after the causes were removed is taken out of the working set. We mark this in the register, and you decide whether to rebuild the site on another domain with new content or to close it. Leaving such a site connected to the others is a risk for the network.
Tell us about the project
Fill in what you already know. Blank fields are fine: we will clarify the rest in conversation.
Brief sent
Thank you. We will reply within one working day.
Preview mode: the form handler is not connected yet, so nothing was sent.