Can’t think of a way to approach this either.
Since the URL doesn’t change and there isn’t any way to trigger the pagination, this website is an outlier case, don’t think it’s possible to scrape in bulk.
Yet, we are creating a web agent that’s able to extract data on prompt, it’s currently on demo. I think cases like these will be possible to approach with the agent in the near future.
@michael.lutz thoughts?