Hi,
I’m trying to build a database of vehicle parts and part numbers. Of course, it’s possible to buy any and all parts online and so, all the information I need is out there. That said, vehicle part webistes tend to be organised with multiple layers due to the complexity of vehicels and thier parts.
I want Bardeen to go in to each of the vehicle categories, click through each of the sub categories of vehicle and be able to scrape item descriptions, part numbers and images from the sub categories below that.
I’ve watched numerous turorials and seen a few posts, but nothing seems to deal with this level of complexity. I did use another scraper to do something similar however, I really like the way Bardeen lets me customise and name the output as well as the field mapping function to give a tidy output in sheets.
This seems to be indeed a complex task. But not impossible as long as categories have the same structure. I would like to clarify the deepest level of scraping. I was trying to follow your instructions when clicking the categories, but the actual website seems to have more levels.
As far as I understand, we start scraping from car model which has a sub-level. The sub-level has a category and each of them a sub-category and those are these:
You intend to scrape the parts which reside inside of these sub-categories, is this correct?
I checked the sub-categories and appears that they do not have a similar structure, so there may be no way to automate the scrapping in a way that I am describing above. For example, Brakes sub-category has more categories inside while Tyres sub-category has not. In this case I would recommend to first come up with the structure for your database in google sheets. Then we can try to split the scraping into smaller parts and precisely fill your database. Would you like to scrape the website for all available cars and models or for a specific one?
Victoria,
Customer Support - bardeen.ai
Explore | @bardeenai | Bardeen Community
Thanks so much for your reply. I carried on playing around and watching tutorials and pretty much found out what you’re saying about the categories being non-standard and that causing issues.
I moved to a different website hoping for a better arrangement of parts/catalogue I’ve made an automated right click scraper for this website and have been getting data that way.
If you look at this site, there’s a standard list on the left side. I’ve tried to build a list scraper but I’m having trouble getting Bardeen to go through the list properly.
If you look at this part, there are filter tick boxes in some of the list categories. Once I’ve chosen the master category (brakes, engine etc) is Bardeen able to scrape, clikc those boxes and then scrape again? (this is what I’ve been doing manually)
Failing that, I’m happy for it to scrape the whole list and I’ll sort the data manually/find some programme or AI that will do it!
I am having trouble with the list scraping functionality. It works perfectly on something like LinkedIn, but not so well on other lists I’ve found online!
This website appears to be not good to work with as well. As far as I can tell, the list of sub-categories placed into columns are not being recognized as a single list. I’ve made several attempts to scrape that, but even this minor part of the automation you need becomes way too cumbersome.
I am thinking to go back to the previous website and try to find a solution for that. I will let you know when I have some progress.
Victoria,
Customer Support - bardeen.ai
Explore | @bardeenai | Bardeen Community
