Must be a pagination problem

šŸ Getting started.
1

but I donā€™t know how to solve it. I just get the first 24 items on this page
Berklee Faculty | Berklee College of Music. It seems like a job for the advanced pagination dialog ā€¦ is there something weird about this Next button.

2

Hi David,

You should just select the More button for your ā€œnextā€ button in scraper settings:

As I understand it, the list of scraped items is not complete for you. In that case I see several reasons for a failure. First thing to try is adding a delay in scraping (in case if list items are simply not loaded at the moment of scraping):

The second possible reason is misconfiguration of a scraper list. If you can share your playbook here, I will be happy to take a look.

Thank you,
Victoria

Customer Support - bardeen.ai
Knowledge Base https://support.bardeen.ai/hc/en-us
Explore | @bardeenai | Bardeen Community

3

Hi Support,

Sorry for my late reply. I tried your suggestion, to add a scrape delay. I picked 20 seconds. It still just pulls the first pageā€™s 24 items. Iā€™ve definitely selected the More tab at the bottom of the page for pagination. Hereā€™s what the selectors were chosen

BardeenBerklee.png

And hereā€™s what the page content looks like:

page1

  • <a class="button" href="?page=1" title="Go to next page" rel="next">More</a>

page2

  • <a class="button" href="?page=2" title="Go to next page" rel="next">More</a>

I know how to get the button selector from the console ā€œdocument.querySelector(ā€˜.js-pager__items.pager .pager__item a.buttonā€™);ā€ works pretty well. I donā€™t understand how to put that into the selector dialog. I entered ā€œ.js-pager__items.pager .pager__item a.buttonā€ and it wouldnā€™t save.

On Mon, Jul 1, 2024 at 5:33ā€ÆAM Victoria from Bardeen (Support) support@bardeen.ai wrote:

4

Is there documentation on how the selector dialog works?

On Sat, Jul 6, 2024 at 1:00ā€ÆPM CMS IT Director itdirector@music.org wrote:

Hi Support,

Sorry for my late reply. I tried your suggestion, to add a scrape delay. I picked 20 seconds. It still just pulls the first pageā€™s 24 items. Iā€™ve definitely selected the More tab at the bottom of the page for pagination. Hereā€™s what the selectors were chosen

BardeenBerklee.png

And hereā€™s what the page content looks like:

page1

  • <a class="button" href="?page=1" title="Go to next page" rel="next">More</a>

page2

  • <a class="button" href="?page=2" title="Go to next page" rel="next">More</a>

I know how to get the button selector from the console ā€œdocument.querySelector(ā€˜.js-pager__items.pager .pager__item a.buttonā€™);ā€ works pretty well. I donā€™t understand how to put that into the selector dialog. I entered ā€œ.js-pager__items.pager .pager__item a.buttonā€ and it wouldnā€™t save.

5

Hi David,

Unfortunately we do not have much documentation on CSS selectors, but I checked your case and found a selector for you. Please note that CSS selectors can only be adjusted for a list container and for list items, thereā€™s no way to adjust button selector with CSS.

Please try adjusting a list container for your scraper as shown below:

Use the next code for the container instead of existing:

DIV.featured-profile-set > UL:first-of-type > LI

I tried on my end and I am now able to fetch as many items as requested.

Hope this helps!
Victoria

Customer Support - bardeen.ai
Knowledge Base https://support.bardeen.ai/hc/en-us
Explore | @bardeenai | Bardeen Community