Must be a pagination problem

but I don’t know how to solve it. I just get the first 24 items on this page
Berklee Faculty | Berklee College of Music. It seems like a job for the advanced pagination dialog … is there something weird about this Next button.

Hi David,

You should just select the More button for your “next” button in scraper settings:

As I understand it, the list of scraped items is not complete for you. In that case I see several reasons for a failure. First thing to try is adding a delay in scraping (in case if list items are simply not loaded at the moment of scraping):

The second possible reason is misconfiguration of a scraper list. If you can share your playbook here, I will be happy to take a look.

Thank you,
Victoria

Customer Support - bardeen.ai
Knowledge Base https://support.bardeen.ai/hc/en-us
Explore | @bardeenai | Bardeen Community

Hi Support,

Sorry for my late reply. I tried your suggestion, to add a scrape delay. I picked 20 seconds. It still just pulls the first page’s 24 items. I’ve definitely selected the More tab at the bottom of the page for pagination. Here’s what the selectors were chosen

BardeenBerklee.png

And here’s what the page content looks like:

page1

  • <a class="button" href="?page=1" title="Go to next page" rel="next">More</a>
    

page2

  • <a class="button" href="?page=2" title="Go to next page" rel="next">More</a>
    

I know how to get the button selector from the console “document.querySelector(‘.js-pager__items.pager .pager__item a.button’);” works pretty well. I don’t understand how to put that into the selector dialog. I entered “.js-pager__items.pager .pager__item a.button” and it wouldn’t save.

On Mon, Jul 1, 2024 at 5:33 AM Victoria from Bardeen (Support) support@bardeen.ai wrote:

Is there documentation on how the selector dialog works?

On Sat, Jul 6, 2024 at 1:00 PM CMS IT Director itdirector@music.org wrote:

Hi Support,

Sorry for my late reply. I tried your suggestion, to add a scrape delay. I picked 20 seconds. It still just pulls the first page’s 24 items. I’ve definitely selected the More tab at the bottom of the page for pagination. Here’s what the selectors were chosen

BardeenBerklee.png

And here’s what the page content looks like:

page1

  • <a class="button" href="?page=1" title="Go to next page" rel="next">More</a>
    

page2

  • <a class="button" href="?page=2" title="Go to next page" rel="next">More</a>
    

I know how to get the button selector from the console “document.querySelector(‘.js-pager__items.pager .pager__item a.button’);” works pretty well. I don’t understand how to put that into the selector dialog. I entered “.js-pager__items.pager .pager__item a.button” and it wouldn’t save.

Hi David,

Unfortunately we do not have much documentation on CSS selectors, but I checked your case and found a selector for you. Please note that CSS selectors can only be adjusted for a list container and for list items, there’s no way to adjust button selector with CSS.

Please try adjusting a list container for your scraper as shown below:

Use the next code for the container instead of existing:

DIV.featured-profile-set > UL:first-of-type > LI

I tried on my end and I am now able to fetch as many items as requested.

Hope this helps!
Victoria

Customer Support - bardeen.ai
Knowledge Base https://support.bardeen.ai/hc/en-us
Explore | @bardeenai | Bardeen Community