Bulk Twitter URL Search Results scraping — Is it possible?

Hey Bardeen friends.

I’m new to your uniquely fantastic service, but I’ve pre-emptively looked through prior Q&A to educate myself on the following:

Presently, one can insert a Twitter (X.com) URL to scrape search results, like so:

1. This in-browser redirects to the Twitter web-page in question
2 Conducts the scraping process while on-page
3 And works flawlessly at
collecting the required data, subsequently exporting it to excel. Fantastic.

• What I’ve scoured far and wide (perhaps in having a blindspot)
Is it possible to undergo the same process by bulk queuing up urls?
For example:

https://twitter .com/search?q=Bardeen%20until:2024-06-17%20since:2024-06-16&src=typed_query&f=live
https://twitter. com/search?q=Bardeen%20until:2024-06-16%20since:2024-06-15&src=typed_query&f=live
https://twitter. com/search?q=Bardeen%20until:2024-06-15%20since:2024-06-14&src=typed_query&f=live

Having the bardeen service automatically open up the next URL in queue, and conducting the scraping process, while holistically generating a single excel file with all URL scrape results?
I’ve tried inserting the urls separated by space/comma, but it only processes the first.

Thank you profusely for your attention and patience with me.

Hi there,

One of the options could be to store twitter links in a column of a google sheet. Then you could set up an automation like this:

  1. Get Table from google sheet or excel
  2. Scrape in the background (connected to a column of the google sheet)
  3. Export to a google sheet or excel online

Here is how a playbook would look like: https://www.bardeen.ai/playbook/community/get-table-from-google-sheet-and-scrape-all-search-result-twitter-links-ej9A5J0b8vFqp4TdKb

Let us know how it goes or if you have any other questions.
Victoria

Customer Support - bardeen.ai
Knowledge Base https://www.bardeen.ai/tutorials
Explore | @bardeenai | Bardeen Community

Thank you so so much.
1. That looks fantastic and you’ve already helped me greatly in learning the solution first part of the equation: inserting bulk URLS (from a table in Google Sheets).

If you’d be so kind as to allow me to abuse your patience in looking to educate myself with second part of the equation.

2. You’ve communicated “Scrape in the background”, as it pertains to the twitter urls.
Presently, on a url by url basis,
chrome_2024-06-20_08-33-40
the incredible extension detects the tab currently opened ( a list of urls you’ve thought me I could derive from my Google Sheets list) migrates to it with the Bardeen extension, scrolls toward the end of the results, and collects the vital data.

All of this is done in browser. Which is a huge plus.
3. I fear “scrape in background” might entail logging into Twitter and conducting back-end data collection (if I’m not mistaken).

My final question then would be:
Is the same process you’ve described above, is it possible to undergo it while conducting it with in-browser data collection (as opposed to scraping it in the background)?
Much like the single URL example, but now at scale?

Thank you so much for your time, I am infinitely grateful for your patience.

Hi again!

“Scrape in the background” will work exactly the same as on an active tab while giving you an option to connect it to links in your google sheet. In terms of logging in to your Twitter, there is no difference as well. Bardeen will not login for you nor collect your login details, but use your browser’s background tab for scraping. Hope this makes sense.

I hope this also answers your final question :slightly_smiling_face: But let me know if not.

Thank you,
Victoria

Customer Support - bardeen.ai
Knowledge Base https://support.bardeen.ai/hc/en-us
Explore | @bardeenai | Bardeen Community

1 Like

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.