My scraper is working fine , however, the website randomly display a popup window from time to time.
when is happen, scraper will break and display an error msg: âScraper returned no results. Scraper hasnât returned any data. Make sure that you are using the correct scraper template on a web page with dataâ
is there anyway to overcome this? my workaround is to reduce the number of pages to scrape so that it is not long enough to encounter the popup. which is frustrating as I have to keep restarting the scraperâŚ
Hey @alandeyong thatâs probably due to the page structure changes when website shows that popup, one way to overcome this will be to create a custom scraper selectors.
Here is a tutorial on basics: https://youtu.be/26Gt_9kFVok?t=1794
But creating custom scraper model may require some knowledge around CSS.
If you can share the website with us you are trying to run scraper on, we will be able to help you building custom model that we will ensure works when the popup is visible on the page. Unless the popup is a access wall and it prevents from data on the page to be accessed.
I did attempt to clean up the selectors using custom selectors as it was not working well by default when they were auto detected by Bardeen and got rid of the erroneous ones.
The popup Google ad applied an overlay over the page I am scraping when it shows up. I need to click the âcloseâ button to close it. However, it seems that even if I close it in time before the delays (I added) runs out, the scraper will break. I have attached a screen grab of the popup.
An idea came to mind but I have gotten the change to check whether itâs feasible:
Is there a error handling trigger I could use, so that when the automation detected unsuccessful scraping (due to the error) it can be restarted from the last page?
From what I see the ad popup page is actually completely different, when you click close a new page content is being loaded. Most probably this is due to the fact that page also clears up Bardeen related content from it causing the issue. I can see 2 solutions here:
Use click action in Bardeen before any action you have and follow it with the delay action before scraping fields.
Alternatively you can use Adblockers which suppose to block ads on the page.
