I can’t seem to automate a deep scrape of a database. I can scrape individual page details or rows of data but not all together and from the master list. The data on each page changes as well and requires clicks. Xpath does not work for me. Since it’s a database it’s protected using JS instead of links. I know it can be done, I just don’t know how to do it.
Master List > Profile Data > Email List (Hidden Behind “Click to view”) > Back to Master list > Next List Item > REPEAT
From what you are sharing, it looks like a complex setup is needed. For the flow “Master List > Profile Data” a deep scraping would be recommended and it involves background loading of Profile pages. In your case those links can not be fetched, so background scraping is not an option.
The other solution theoretically could be setting up a sequence of “click > get data” steps, but it may be not reliable since it is not designed for deep multi-level scraping. Did you have a chance to try this way?
Victoria,
Customer Support - bardeen.ai
Explore | @bardeenai | Bardeen Community