Scraper keeps stopping - (Twitter / X - List scrape)

[BWhitman]

Trying to scrape a Twitter/X.com List

Scraper starts off fine but then stops at random points. Normally around 35 records.
(Although on once occasion it did actually manage to pull 648)

Here is the play book - Shared Playbook Template

[Bardeeni]

Hello Fergus,

The most common reason for this issue is that the results may not be fully loaded, causing the scraper to consider the task complete prematurely.

We highly recommend the following best practices to avoid some of the issues you are facing:

1. Add a custom delay per page, so the scraping is more human like and also allows the page to load before scraping. This could happen if the page takes a long time for results to load, Bardeen will then think there are no more results. Could you please go into the playbook builder, look for the scraper action and add a custom delay of about 5 seconds? Adding the custom delay tells Bardeen to wait for 5 seconds every time a new set of results are created when it scrolls down.

Here’s an example:

2. We’ve added a new setting to your scraper models that would allow the scraper to run in a normal browser window, but behind the currently opened web pages. Previously, the scraper would try to get the data from a minimized window and, in some cases, would fail to do so because of limitations on some websites - like getting a list of reviews from Google maps and so on. Now, you can disable this so that the browser window doesn’t get minimized, but it also doesn’t get in your way because it’s behind your current windows. This setting is available for both new and existing scraper models - you can easily modify your existing scraper models by opening the scraper settings and disabling the “Use minimized window for background scraping” switch.

3. Scrape in smaller chunks than you are currently doing
4. Using an app to help keep your computer awake. This is a great one for Mac https://apps.apple.com/us/app/jolt-of-caffeine/id1437130425?mt=12

Hope that helps!
Victoria

Customer Support - bardeen.ai
Knowledge Base https://support.bardeen.ai/hc/en-us
Explore | @bardeenai | Bardeen Community