Unable to get Scraper running in background after premium subscription. Will not open additional windows from google sheet with url links

🐞 Report an issue
1

Hi I’m trying to scrap from a list of urls however when I set the automation to run in the background it will not open up multiple windows. I’m following this guide almost exactly without any luck: https://www.youtube.com/watch?v=c9IDz8E_nek

Please advise the best way to troubleshoot the issue. My playbooks will not run to completion.

1 Like
2

Hi @will1, Welcome to the Bardeen Community :slight_smile:

  • This is actually how the “Scrape data in the background” action is designed to behave as it allows you to still continue working in other various applications while it scrape behind the scenes.

If the issue is something different, please let us know. Asa friendly reminder, it’s always helpful to share the automation you are working with to assist on our end to troubleshoot for you.

Thank you,
Jess

3

Hi Jess,

Thank you for the reply. Here is one of the test playbooks that I’m working on, I have several iterations of this.

https://www.bardeen.ai/playbook/community/Test-Full-List-hJFWQiLF3sM9FXaaL4

I’ve yet to get a successful output from this automation. I need to deep dive back into it today and see where I am hung up but perhaps it would be helpful to zoom later to go over the issue? I will follow up with this email with a few screenshots as well as soon as I get caught up on a few things.

Thanks!
Will

4

Hi @will1,

Thank you for sharing your playbook - after a quick peek, here’s my recommendations:

  1. Remove the “Create Google Sheet” Action
    image
    image1445×900 57.3 KB
  2. The “Add rows to Google Sheet” will supply everything you need by using the command to create a google sheet within the input here:
    image
    image1290×904 38 KB
  3. I’m unable to run it on my end to further troubleshooting because you need to share the first google sheet called “D2 Full List” publicly with edit rights.

Thank you,
Jess

5

Hey Jess,

Here is the D2 Full List with edit rights, let me know if you have any issues accessing it. I’ll make those updates based on your suggestions as well.

https://docs.google.com/spreadsheets/d/1XwLO-oHXr5MCeLae9S7dTIEFKtlkx2J7OLLDr5ipNws/edit?usp=sharing

Thanks,

Will

1 Like
6

Thank you @will1 !

It appears to be working on my end - believe the issue was the name of the “Test List” tab header for the URLs. Let me know if you have any questions.

Thank you,
Jess

7

Hi Jess,

One other thing I’m noticing, when I run the playbook for the first time it seems to be adding in this step to the playbook as I’m not initially setting this up. Is that correct behaviour of the system?

image.png
image.png2120×646 75.7 KB

Thanks,

Will

On Tue, Oct 31, 2023 at 12:05 PM Will Shoemaker will@popacta.com wrote:

Hey Jess,

Here is the D2 Full List with edit rights, let me know if you have any issues accessing it. I’ll make those updates based on your suggestions as well.

D2 Full List - Google Sheets

Thanks,

Will

8

Hey Jess,

The Test Full List is returning no results still. I did set up another playbook today using the same sequence which will not run to completion. Link below. Perhaps I need to change my output to something besides a googlesheet?

https://www.bardeen.ai/playbook/community/1031-Test-List-zMG69YDuIYc9hXSMe7

Thank you!

Will

9

Hi Will, this is not normal behavior if I’m understanding correctly.

Okay, I’m trying to navigate to the GSheet “D2 Full List” But it gives me this screen:

image
image1507×865 54.1 KB

Please make sure the name of the “Test List” tab header for the URLs is the exact same as the input for the scraping action in the workflow:

image
image838×732 40 KB

I hope this helps!
Thank you,
Jessp

10

Hey Jess,

Yes I am available to zoom any time today, I can send you a meeting invite if you let me know a time that works well for you. Thank you for all of your help with this!

Will

11

Sorry, I’m not available today (in office).

12

Hey Jess,

Thats okay just let me know when you have some availability, I’m still working through some issues with the playbook now that I straightened out the version control issues with the google doc.

Thanks again for all your help!
Will

1 Like
13

Okay sounds good, I’ll likely have more time tomorrow or Friday. I’ll keep you posted.

Thank you,
Jess

14

Thanks for the tips @Jess

Welcome to the community @will1 :slight_smile:

I’ve noticed the different URLs have slightly different formats, so I’m checking with the team on whether one scraper template can be used across the different sites and I’m also checking why you haven’t been having any luck getting some output.

Are you able to share a quick recording on what kind of error you get when running the playbook please?

15

Hey Vin,

Let me know what you find out about the url formats, they all come from the same hosting platform, Learfield so I hope that they are consistent enough to be able to scrape in the full bundle.

https://www.loom.com/share/b278536f962e4d2db2b83ae48c50beb9?sid=4b9657d6-8f3e-4abd-a8dd-7a3ffe7816fe

Here is a video of running the playbook. As you can see I run it and it begins to scrap the first url then gets hung up there, it never completes the playbook providing a .csv output. However, one time I was able to get it to scrape 3 sites from a google doc and spit the output into the same google doc, I was never able to recreate it though. Let me know if I can send you any more videos or if you would like to zoom today to go over the issues.

Thank you,
Will

16

Thanks for that @will1 , our engineers are working on a fix for this issue. I’ll be in touch when it is released. Thanks for your patience :slight_smile:

17

Hey Vin,

Checking in on this. I saw you or possibly Jess created a playbook that used scraping the full table then extracting only the text in email format as a possible work around? I haven’t had a chance to play with that approach yet but is this probably the best way to achieve what we’re trying to do here?

Thank you!
Will

18

Hey Will,

Could I confirm that you’d only like to scrape the email address from these websites ? I’ve given it a test and a basic scraper template built off one of the websites won’t work on the other websites since the table columns are not in the same order across the sites. We’ll need to make some tweaks to the CSS selectors to match the different ordering of the columns across the different sites.

With regards to the bug I was referring too, that is affecting background scraping and we’re rolling out a fix for it soon. For now, scraping in the active tab is still possible.

19

Hey Vin,

Yes just email is fine. If we could pull the name column as well that would be helpful but not necessary. Let me me know if you have any other questions.

Thank you!
Will

20

Hi Will,

First and foremost, I want to extend our sincerest apologies for all the inconvenience you may have experienced due to the recent bug in our software. We understand how frustrating such issues can be, and we appreciate your patience and understanding as we worked diligently to address it.

I am pleased to inform you that our development team has successfully identified and resolved the bug that was causing the issues you reported. The fix is now available with a new version of Bardeen (2.37.2) which addresses the issue an issue with background scraping in the previous version. You can check your current version of Bardeen in Google Chrome by following the steps in my screenshot below (the version number will appear next to the Bardeen extension) as you might still be on the old one:

Here is a quick guide on how to update your version of Bardeen: How to Update Bardeen

Thank you for your patience, and if you need assistance, email me directly lucy@bardeen.ai.

Customer Support - bardeen.ai
Explore | @bardeenai | Bardeen Community

Customer Support - bardeen.ai
Explore | @bardeenai | Bardeen Community

Bardeen