Preload translations with the scraper
Clonable translates a page through its reverse proxy when someone opens that page. On the first visit, translations still need to be generated, which can make the page load more slowly.
The scraper lets you preload translations. It opens the pages listed in your clone's sitemap, triggering translation before visitors open those pages.
Before you start
- Exclude any pages you do not want to include first. Excluded pages are also left out of the total number of pages to scrape.
- Make sure your clone is not set to no index. Disable no index before using the scraper.
Using the scraper
- Enter your clone's sitemap URL as the input for the scraper.
- Wait for Clonable to validate the sitemap and count its pages. Large sitemaps may take a while.
- Once the status is Ready to scrape, you can start the scraper.
- Follow the scrape job's status. While it is running, refresh the page to see the updated state.
How long does scraping take?
The scraper opens one page every 30 seconds. Allow approximately 50 minutes for 100 pages, or 8 hours and 20 minutes for 1,000 pages. Scanning the sitemap and starting the scraper take additional time.
Scrape job statuses
| Status | What it means |
|---|---|
| New | The sitemap has been validated and is waiting for counting to start. |
| Scanning sitemap | Clonable is counting the pages in your sitemap. Larger sitemaps may take longer. |
| Ready to scrape | Counting is complete. Excluded pages are not included in the total. The job is ready to start. |
| Sitemap error | An error with the sitemap prevented the scraper from starting. Check that you entered the correct clone sitemap URL and that the sitemap is accessible. |
| Starting scraper | The scraper is being started and will begin opening pages soon. |
| Scraping | The scraper is opening your pages to preload translations. Refresh the page to see the updated state. |
| Completed | The scrape job has finished. |
Scraping again
You can start a new scrape job once the previous one has finished. This is useful when you have added many new pages and want to preload their translations too. Make sure those pages are included in your clone's sitemap.
User agent
The scraper uses the following user agent when opening URLs from your sitemap:
Mozilla/5.0 (compatible; ClonableSearchBot/1.0; +https://www.clonable.net)
If your website uses security rules that restrict bots, make sure the scraper can access the pages you want to preload.