Recursive scrapping of website with each post

❓Help and questions
1

I want to scrape a QandA website with a URL with a sequential number for each post.
Example
https://www.healthtap.com/questions/1366327

How do get Bardeen to sequentially go through each number e.g., healthtap.com/question/10010, and then go to the next post healthtap.com/ question/10011, and pull the questions and answers from each page and put them into a spreadsheet

10010 question answer1, answer2, etc…

Thanks for your help
Nik

  • Take 5 minutes to search among the forum for existing questions. The answer might already be there!

  • If you’re trying to build automation, try describing your workflow into Bardeen’s Magic Box (main search bar on the extension), Bardeen’s AI will try to build it for you :magic_wand:

  • Consider we also have YouTube tutorials that can help answer your query!
    https://www.youtube.com/c/Bardeenai

Tips for posting (and getting faster answers⚡)

  • Start by sharing some context about your use-case or how you’re using Bardeen

  • If you have a question related to your automation, share the automation link

  • Are you trying to scrape a particular website? Share the link so we can give it a check.

Did you run into a bug or issue with Bardeen?

Check our :lady_beetle: Report an issue category and follow this tutorial: READ ME: How to report a bug in Bardeen

2

Hi Nik,

I tested your logic of increasing the number on the URL but that doesn’t work. e.g Post 136627 exists but post 136628 doesn’t

Anyway, I think the best way to do this is in google sheets to create sequential links. I’ve created a playbook where you can then tell the playbook where to get these links from and it will then scrape the question and the answers from the link and then adds the results back to the google sheet (you can choose to put the results in a different tab). You can access the playbook here

Customer Support - bardeen.ai
Explore | @bardeenai | Bardeen Community

3

That sounds like a great solution. Let me test it!

4

Can you repost link? There does not appear to be a hyperlink.

5

Thanks. I am not familiar with how to use a playbook. Do you have a video to share? or instructions on how to use this playbook?

6

https://www.bardeen.ai/playbook/community/Scraping-www.healthtap.com-uT5zu8KO7INvoNMq6K

You can click on the link above, then click Pin It and the playbook will be saved in your version of Bardeen. By clicking on the playbook within Bardeen, you can then launch the playbook and it will ask you for inputs on which Google Sheet to get the links you want to scrape

Customer Support - bardeen.ai
Explore | @bardeenai | Bardeen Community