Hello! I am trying to scrape Tweets that mention a specific phrase between two set dates, and compile them into a Google Sheet, for some simple research. I am using the scrape from active tab, to eliminate additional inputs/steps, in order to make it as streamlined as possible. So, the tab is loaded with the Tweets that I want. In my playbook, I have allocated a 5 second load wait time. When I run the playbook, I can see it scrolling through the page and loading more Tweets.
However, my problem is that it doesn’t end up pulling all of the Tweets. I have tried different load wait times, making different scrapers, as well as utilizing a premium playbook that runs in a background tab. None of these have been working to scrape all the Tweets. At most, it has returned 1356 Tweets in one run. I do not get any error messages when executing the playbook.
I have seen some similar questions on the forum, but none relating to Twitter. Am I unable to pull all of the Tweets because I don’t have a Premium subscription? I could run the playbook on multiple search result pages, but I think that would be defeating the point of building it in the first place, plus I wouldn’t know if it actually pulled all of the Tweets.
This is the page I am trying to scrape Tweets from: https://twitter.com/search?q="gerardo%20ortiz"%20until%3A2016-04-21%20since%3A2016-03-20%20-filter%3Areplies&src=typed_query&f=live
This is my playbook: Shared Playbook Template
I’m not a super techy person, so I find Bardeen to be hugely beneficial. Any help is appreciated!