Problem with scraper not pulling all Tweets

❓Help and questions
1

Hello! I am trying to scrape Tweets that mention a specific phrase between two set dates, and compile them into a Google Sheet, for some simple research. I am using the scrape from active tab, to eliminate additional inputs/steps, in order to make it as streamlined as possible. So, the tab is loaded with the Tweets that I want. In my playbook, I have allocated a 5 second load wait time. When I run the playbook, I can see it scrolling through the page and loading more Tweets.

However, my problem is that it doesn’t end up pulling all of the Tweets. I have tried different load wait times, making different scrapers, as well as utilizing a premium playbook that runs in a background tab. None of these have been working to scrape all the Tweets. At most, it has returned 1356 Tweets in one run. I do not get any error messages when executing the playbook.

I have seen some similar questions on the forum, but none relating to Twitter. Am I unable to pull all of the Tweets because I don’t have a Premium subscription? I could run the playbook on multiple search result pages, but I think that would be defeating the point of building it in the first place, plus I wouldn’t know if it actually pulled all of the Tweets.

This is the page I am trying to scrape Tweets from: https://twitter.com/search?q="gerardo%20ortiz"%20until%3A2016-04-21%20since%3A2016-03-20%20-filter%3Areplies&src=typed_query&f=live

This is my playbook: Shared Playbook Template

I’m not a super techy person, so I find Bardeen to be hugely beneficial. Any help is appreciated!

2

Hello ! Thanks for submitting this request

I’ve taken a look at the scraper you are using and there might be a bug with it. I’ve escalated this with our scraper team to look into.

While they fix that, I’ve created a new custom scraper for you that scrapes twitter pages (along with a video on how I created it). The scraper I built is in this playbook which you can then build on to add the google sheets action to output the results. I’ve also limited the output to 100 results for the purposes of this video but I don’t see why you won’t be able to scrape more with it (you can edit that number in the playbook).

Here’s a video of how I built the playbook:

Hope this helps !

Customer Support - bardeen.ai
Explore | @bardeenai | Bardeen Community

3

Thank you! This is working so much better! Could you please resend the video link?

4

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.