Hi, I’m using the template “Export Instagram Followers/Following to Google Sheet” and it’s only giving me a limited number of results compared to the true number expected.
Yes, I’m on the /Followers page, and yes it does successfully pull some data. But for example, this account has ~4,000 followers, and my first scrape pulled 19, then 50, then 100, then 300, then 400, so it’s not loading the full pop-up window of followers to scrape.
I’m deleting the spreadsheet and re-running it, but this can’t be the best solution to keep running the scraper to load the pop-up window further to get +100 more each time when I’m trying to get to 4k?
We highly recommend the following best practices to avoid some of the issues you are facing:
Add a custom delay per page, so the scraping is more human like and also allows the page to load before scraping. This could happen if the page takes a long time for results to load, Bardeen will then think there are no more results. Could you please go into the playbook builder, look for the scraper action and add a custom delay of about 5 seconds? Adding the custom delay tells Bardeen to wait for 5 seconds every time a new set of results are created when it scrolls down.
Here’s an example:
We’ve added a new setting to your scraper models that would allow the scraper to run in a normal browser window, but behind the currently opened web pages. Previously, the scraper would try to get the data from a minimized window and, in some cases, would fail to do so because of limitations on some websites - like getting a list of reviews from Google maps and so on. Now, you can disable this so that the browser window doesn’t get minimized, but it also doesn’t get in your way because it’s behind your current windows. This setting is available for both new and existing scraper models - you can easily modify your existing scraper models by opening the scraper settings and disabling the “Use minimized window for background scraping” switch.
Scrape in smaller chunks than you are currently doing
Hey Lucy! I’m fine to build my own custom scraper and add a delay, but I was using the prebuilt Bardeen template, so are you saying I need to modify the existing template? I guess I was expecting the pre-built template to work as-is?
re 2: How can I scrape the list in smaller chunks? If I re-run it in starts from the top again.
re 3: The computer awakeness doesn’t seem to be an issue for this.
I looks like the issue may be about your connection speed, so I would also recommend adding a delay to action settings. I presume you can see an increase in the amount of pulled profiles because every next time you restart the scraper, it already did scroll down the list more and more and those records are loaded. With the next few scrolls it will face a loading issue hence scraping interrupted.
You do not need to modify the built-in scraper, a delay can be added like this:
Please try with a delay of at least 2-3 seconds. Also make sure that your device does not go to a sleep mode. Alternatively you can try with a different connection (if faster is available) and/or device.
Just to make sure that the scraper does not have an obvious issue, I tested it on my end and it pulled more than 2K profiles right away:
