I have created a scraper automation which scrapes facebook group memebers, their profile url, their name and additional info and saves it in a google sheet . Itâs an active tab scraper and I am using infinite scroll as the profiles loads on scrolling in facebook group.
The problem is that I am trying to scrape all 21K members of the group but the scraper stops after scraping some profiles. Like in my firt try I got 1200 profiles, then in second I got 2K profiles and in my third try I got 1900 profiles. How do I solve this issue and scrape as many profiles/rows as I want?
To add here, I would like to emphasize that for successful scraping of such a big load, you should make sure that your computer does not lose internet connection and stays awake all the time. Also the bigger delay you use the better, an automation should simulate human scrolling pace.
@Jess How much delay should I add to solve this issue? Also, outside of the scraper action where should I add delay? before adding the rows in google sheet step?
Hey @Jess I tried again and I tried to scrape 3K profiles this time, for the first try it stopped at 1432 profiles and for the second try it stpped at 1702 profiles. still not working
@Bardeeni Yes I made sure my internet was connected all the time with a steady speed and my laptop was awaks too in the multiple iteertions of my test. How big of a delay do you think I should add?
Iâve seen the same behavior before when scraping Instagram. The case is the platform is sending hidden human checks time to time. I donât think it is possible to overcome this with increasing delay only since it is checking if the window is in focus and if there are any mouse moves. Personally I was able to scrape up to 10K records only when I was controlling the process of scraping all the time and performing some mouse moves time to time when I saw the window is getting stuck on scrolling or something else (hidden check is working).
Also I heard from developers that such huge lists of followers or members are not designed to be all shown on a page, they are there to display the count and to allow searching for specific members.
@Bardeeni is there any way for me to use the scraper in continuity repeatedly? For example if I want to scrape 10K profiles, I could scrape 2K profiles 5 times but each time the scraper should continue from the last data/row scraped?
You could try configuring the Containerâs CSS Selector inside of the scraper (more technical/hacky way) to scrape in batches of 1000 at a time and then update the Container CSS Selector to get the next 1000 and so forth.
I think the problem here is that FB does not provide pagination for the list of members, so if the âhackyâ way wonât work, there wonât be an alternative. In fact, âhow to scrape large lists of members or followersâ is a painful question for a long time already and I didnât see any reliable solution so far.
