Trying to crawl specific site, not sure how I can get it done

❓Help and questions
1

I am trying to crawl this public data, which displays major companies in South Korea.

Tricky part is, I want Bardeen to…

  1. Click the name of the companies in the list, which will display a modal that contains more information on them
  2. Collect all the information and pipe it into a google spread sheet, ideally.

I tried Bardeen for hours and it is what I came with, which seems to failing at what it supposed to do.

So, how can I fix the issue? I am willing to buy Pro plan if it is required.
Thanks.

2

Hi Steven,

I see you have tried building this automation with AI prompt. It seems to me that this scraping case is rather complex for AI and requires manual scraper configuration.

Let me share some information that will help you:

  1. If every company has their own link, you can employ a “deep scraping” technique. This involves first scraping a list of company links (level 1) and then extracting the detailed information (level 2) from those links.

To assist you further, here are some resources that might be helpful:

  • General guide on scraping: YouTube Guide
  • Deep scraping tutorial with live examples: YouTube Tutorial
  • Tutorial on deep scraping: Bardeen Scraper Guide
  1. If links are not available and all the information is provided in modals, you can try building a scraper that clicks every link and scrapes information from there. Following this method, it is important to set proper delays in between “click” and other actions. Here is our general guide that I suggest you to look through: https://support.bardeen.ai/hc/en-us/articles/23925200748173-How-to-use-Bardeen-scraper

By the way, our team can build a custom polished playbook for you, we offer this as part of our Business plan. Just let us know, and we’d be happy to connect you with our Sales Team to discuss details.

Victoria

Customer Support - bardeen.ai
Knowledge Base https://support.bardeen.ai/hc/en-us
Explore | @bardeenai | Bardeen Community

3

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.