Limited scraping with Facebook Group Members Scraper

I have created a scraper automation which scrapes facebook group memebers, their profile url, their name and additional info and saves it in a google sheet . It’s an active tab scraper and I am using infinite scroll as the profiles loads on scrolling in facebook group.

The problem is that I am trying to scrape all 21K members of the group but the scraper stops after scraping some profiles. Like in my firt try I got 1200 profiles, then in second I got 2K profiles and in my third try I got 1900 profiles. How do I solve this issue and scrape as many profiles/rows as I want?

here is the link to my playbook: Shared Playbook Template

Please help.

Hi there @ma336ahsan, it’s best practice to add delays inside of both the scraper action and inside of the scraper. I hope this helps!

Hi Muhammad,

To add here, I would like to emphasize that for successful scraping of such a big load, you should make sure that your computer does not lose internet connection and stays awake all the time. Also the bigger delay you use the better, an automation should simulate human scrolling pace.

Hope this helps!
Victoria

Customer Support - bardeen.ai
Knowledge Base https://support.bardeen.ai/hc/en-us
Explore | @bardeenai | Bardeen Community

@Jess How much delay should I add to solve this issue? Also, outside of the scraper action where should I add delay? before adding the rows in google sheet step?

In the first action of your playbook, add delay of 3 seconds.

The one in your scraper, try .3.

1 Like

Hey @Jess I tried again and I tried to scrape 3K profiles this time, for the first try it stopped at 1432 profiles and for the second try it stpped at 1702 profiles. still not working :frowning:

here is the updated playbook: Shared Playbook Template

@Bardeeni Yes I made sure my internet was connected all the time with a steady speed and my laptop was awaks too in the multiple iteertions of my test. How big of a delay do you think I should add?

Hi Muhammad,

I’ve seen the same behavior before when scraping Instagram. The case is the platform is sending hidden human checks time to time. I don’t think it is possible to overcome this with increasing delay only since it is checking if the window is in focus and if there are any mouse moves. Personally I was able to scrape up to 10K records only when I was controlling the process of scraping all the time and performing some mouse moves time to time when I saw the window is getting stuck on scrolling or something else (hidden check is working).

Also I heard from developers that such huge lists of followers or members are not designed to be all shown on a page, they are there to display the count and to allow searching for specific members.

I hope this helps.
Victoria

Customer Support - bardeen.ai
Knowledge Base https://support.bardeen.ai/hc/en-us
Explore | @bardeenai | Bardeen Community

1 Like

@Bardeeni is there any way for me to use the scraper in continuity repeatedly? For example if I want to scrape 10K profiles, I could scrape 2K profiles 5 times but each time the scraper should continue from the last data/row scraped?

You could try configuring the Container’s CSS Selector inside of the scraper (more technical/hacky way) to scrape in batches of 1000 at a time and then update the Container CSS Selector to get the next 1000 and so forth.

I think the problem here is that FB does not provide pagination for the list of members, so if the “hacky” way won’t work, there won’t be an alternative. In fact, “how to scrape large lists of members or followers” is a painful question for a long time already and I didn’t see any reliable solution so far.

Victoria

Customer Support - bardeen.ai
Knowledge Base https://support.bardeen.ai/hc/en-us
Explore | @bardeenai | Bardeen Community

Hey @Jess I don’t know how to apply this hack to the scraper template. If you could provide a step by step guide then it would be a big help :pray:

This would require my services for a fee as it’s more advanced and time consuming. Is this something you’d be interested in moving forward with @ma336ahsan?