Scraping Indeed Issue

I am trying to scrape indeed search results and when I scrape a small number of results, my scraper works fine. When I try to scrape a large number of results, my scraper does not pull in the full job description. How can I result this?

It also doesnt listen to my limitations for the number of results to scrape

When building a custom scraping template it seems to work up until I try to run the automation then only half of it works

That image shows the fields I want my scraper to grab and it looks like it is successfully grabbing them when building the template. Then, when I run the template it is only able to successfully grab the job title and indeed job link column. I have been trouble shooting this for a long time with no success

Hi Skye,

It sounds like Bardeen is scraping the page before it is fully loaded and isn’t able to capture all the data. Could you check if you have a custom delay set in the action that does the background scraping ? If you don’t, we typically advise to set a delay of 3-5 s to allow the page to load (this also depends on your internet speed)

In terms of your limitations on number of results to scrape, is it scraping too many or too few ? I’ve got an example below of how you fill in these two

If the tips above don’t work, please share your playbook so I can take a look at it. Thanks !

Customer Support -
Explore | @bardeenai | Bardeen Community

I figured out how to get the custom elements to work it was because bardeen was unable to detect the elements on the page and I needed to make a custom selector but I am still running into a lot of issues I am unsure of how to solve. When I upgraded my account it mentioned premium support, how can I access that?

The biggest thing I am running into is that when I run the scraper it only works on my current browser and won’t run in the background correctly (it doesn’t scrape the same items as it does when it scrapes in the active tab.)

The other issue is that even when I scrape in the active tab, if I try to do more than 30 results, it bugs out and the pagination doesn’t work properly. It will accidentally click into a job instead of the next page and the automation basically dies from there.

How can I resolve these issues?

Hi Skye,

You can reach out premium support team via this page :

Customer Support -
Explore | @bardeenai | Bardeen Community

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.