Background scraping stoping

🐞 Report an issue
1

### Description of the issue:

I have created a workflow where Bardeen get a table from G-sheet, a colomn with Google urls (google search for pharmacies) and I apply a simple scrape which get the google card and the meta description of the 3 first results.

But the background scraping stopq by itslef juster after a few research. I have tested the scraping with 3 lines on the g-sheet (3 differents URLS ) and it worked fine. As soon as I put all my list of URLs, the background scrapper stops after 4/5 pages scrapped.

Error message (if aplicable):

No error message. On bardeen the progress bar continues to run, but the google windows where the pages to scrape are uploaded stops on a page and doesn’t upload the following URLs




##### Bardeen version: 2.37.1

##### Link to Playbook or Autobook (if applicable):
[chrome-extension://ihhkmalpkhkoedlmcnilbbhhbhnicjga/landing.html](https://www.bardeen.ai/playbook/community/PHARMA_TEL-X6U55cZ1310pvPyB5I)

##### 🎥Video recording or screenshots (optional, but recommended):

![image|690x137](upload://v5SbkhHvwDBi3gkTbXhGYrmABHv.png)
2

Hi @bauge.maxime ,

Sorry you’re facing issues getting the data scraped. There are a few initial troubleshooting steps that you could try :

  1. We have now released a new version of Barden (2.37.2) which addresses the issue you were having. You can check your current version of Bardeen in Google Chrome by following the steps in my screenshot below (the version number will appear next to the Bardeen extension):

image
image457×507 17.3 KB

Here is a quick guide on how to update your version of Bardeen: How to Update Bardeen

  1. In your playbook , I see you’ve set a custom delay of 3 seconds. If you’re internet connection isn’t very fast, you might need to set a slightly longer delay to give the page time to load before scraping.

If these two steps above don’t work, could you please DM me with a link to the google sheet you are getting the URLs from ? I’ll then be able to test it and see what the issue is.

3

hi @vin_bardeen,

Thanks for the reactivity and the answer. Indeed, I updated the extension and now it works.

New Trouble for me :

While permforming the background scraping after few hundreds pages, google asked me to verify Im human via the reCaptcha.

→ How can I adress it ? Cause of course the pages don’t load and I can’t scrape these ones.

thanks in advance

4

Hi Maxime,

Great to hear it’s all working for you! Unfortunately, verifying Captcha is one of actions the Bardeen is unable to automate. There are a few ways you could try to avoid having a captcha pop up:

  1. Add a custom delay per page, so the scraping is more human like and also allows the page to load before scraping
  2. Scrape in smaller chunks than you are currently doing

Hope this helps !

5

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.