I would really like to get some support for one automation that I just cannot seem to get to work properly. Im scraping a list of concerts from this site.
Im not getting back all the concerts on the list. It just does stops collecting at some point. No clue what is going on. I a similar automation/scraper on other sites and there it works fine.
This is this automation in question:
And this is the google sheet where the results are dumped in.
I know you must be really busy working on the migration to 3.0 but would really appreciate the support .
Good to hear from you. As a first step, can you try adding a custom delay per page, so the scraping is more human like and also allows the page to load before scraping. This could happen if the page takes a long time for results to load, Bardeen will then think there are no more results. Could you please go into the playbook builder, look for the scraper action and add a custom delay of about 5 seconds? Adding the custom delay tells Bardeen to wait for 5 seconds every time a new set of results are created when it scrolls down.
Here’s an example:
There is a setting to your scraper models that would allow the scraper to run in a normal browser window, but behind the currently opened web pages. Previously, the scraper would try to get the data from a minimized window and, in some cases, would fail to do so because of limitations on some websites - like getting a list of reviews from Google maps and so on. Now, you can disable this so that the browser window doesn’t get minimized, but it also doesn’t get in your way because it’s behind your current windows. This setting is available for both new and existing scraper models - you can easily modify your existing scraper models by opening the scraper settings and disabling the “Use minimized window for background scraping” switch.
I tried this on my end and it seems to work. Please let me know if the issue still persists and I can escalate.
Thank you Lucy, i really appreciate the extensive response and time you put into testing.
Both solutions didnt work for me. I rebuilt the scrapers and the automation but it is still not scraping all the list results. Only parts of the list items were scraped. Did it scrape all the concerts when you tried it?
This venue is quite critical for my concert page. If there is any other thing that you can think of that would be really really appreciated.
The new scraper automation below. The previous one deleted as it gave error messages after disabling the background scraping.
Bardeen Magic again. Thank you Victoria, you truly made my day. The success of this scrape was pretty essential to the go/no-go of my project. Let me know if I ever can return the favor somehow.
–Bob
.
Let me know if I ever can return the favor somehow.