Scraper in the background missing out the selected element item to scrape

šŸ Getting started
, ,
1

Hello Iā€™m using Bardeen to scrape this website
https://salesweb.civilview.com/.
Iā€™m scraping different counties into one spreadsheet but one tab for each county. I built a manual scraping template for each county and the first action is to scrape active tab which then feed the link found in the active tab into the next action which is scraping the background. Iā€™m finding that the second action in background is not picking up all the ā€œApprox. upsetā€ element (aka ā€˜Good Fatih Upsetā€™) found on the second page. Sometimes it picks it and sometimes it doesnt. See images below showing red arrow as missing element. Also linked to each scraper template

Scraper template for Passaic county . red arrow missing here but, the first entry here is missing the approx. upset element but the automation goes on to pick it up for the other 2 entries

for bardeen
for bardeen1236Ɨ360 44.7 KB

Scraper template for Monmouth county

for bardeen 2
for bardeen 21358Ɨ362 55.4 KB

Scraper template for Union county

for bardeen 3
for bardeen 31250Ɨ279 37.5 KB

Can someone help please?

2

Hi pro1710,

Sorry your are running into issues. We highly recommend the following best practices to avoid some of the issues you are facing which could be caused by loading issues:

  1. Increasing the custom delay per page, so the scraping is more human like and also allows the page to load before scraping. This could happen if the page takes a long time for results to load, Bardeen will then think there are no more results. Could you please go into the playbook builder, look for the scraper action and change the custom delay of about 5 seconds? Adding the custom delay tells Bardeen to wait for 5 seconds every time a new set of results are created when it scrolls down.

Hereā€™s an example:

  1. Weā€™ve added a new setting to your scraper models that would allow the scraper to run in a normal browser window, but behind the currently opened web pages. Previously, the scraper would try to get the data from a minimized window and, in some cases, would fail to do so because of limitations on some websites - like getting a list of reviews from Google maps and so on. Now, you can disable this so that the browser window doesnā€™t get minimized, but it also doesnā€™t get in your way because itā€™s behind your current windows. This setting is available for both new and existing scraper models - you can easily modify your existing scraper models by opening the scraper settings and disabling the ā€œUse minimized window for background scrapingā€ switch.

Please let us know if that resolves the issue.

Thanks!
Lucy

Customer Support - bardeen.ai
Explore | @bardeenai | Bardeen Community