Scraping Only Taking First Item of Each Page

🐞 Report an issue
, ,
1

Description of the issue:

I configured a scraper to scrape the data from this website: https://www.ncaa.com/stats/basketball-men/d1/current/team/474 and enter the information into a Google Sheet.

The issue is that it is only collecting the data of the first row, one each page. For example, it takes UConn on Page 1, San Francisco on Page 2, and so on. I need all of the data in a spreadsheet, not just the first item.

I tried scraping the page as an auto-configuration table, but there is no paganation option afterwards, so I would essentially be back to just copy and pasting.

I just upgraded my membership and need this to work.

Can someone please help?

Bardeen version: 2.45.0
Link to Playbook or Autobook (if applicable):

Shared Playbook Template

2

Hi Patrick,

We highly recommend the following best practices to avoid some of the issues you are facing:

  1. Add a custom delay per page, so the scraping is more human like and also allows the page to load before scraping. This could happen if the page takes a long time for results to load, Bardeen will then think there are no more results. Could you please go into the playbook builder, look for the scraper action and add a custom delay of about 5 seconds? Adding the custom delay tells Bardeen to wait for 5 seconds every time a new set of results are created when it scrolls down.

Here’s an example:

  1. Scrape in smaller chunks than you are currently doing
  2. Using an app to help keep your computer awake. This is a great one for Mac https://apps.apple.com/us/app/jolt-of-caffeine/id1437130425?mt=12

Hope that helps!
Lucy

Customer Support - bardeen.ai
Explore | @bardeenai | Bardeen Community

3

I re-did the scraper with a 5-second wait step but it is still only bringing in the 1st data point per page as you can see in this image:

image

It also is only 7 pages of data, so there shouldn’t be a reason why this would need to be broken up.

4

Hi Patrick,

Thanks for reaching out. To help us troubleshoot this issue for you, could you provide the following details:

  1. The link to the playbook you are trying to use - here is how you can share your playbook link
  2. A screenshot of your scraper model or tell us what data you want to scrape

Thanks!

Customer Support - bardeen.ai
Explore | @bardeenai | Bardeen Community