New to Bardeen, testing it as possible ongoing solution for our ongoing daily scraping needs.
I have the first pass of scrapes to look for any new table listings (links to new entries), and for all new listings, queue them to scrape the secondary pages, across a number of websites.
For this I setup a unique URL column setup in each tab, so that when I do the initial scrape, it only adds records that are new. I’ve added a UNIQUE constraint to reject new entries that arent unique.
However, because Bardeen connects via API to Google Sheets, it bypasses the unique constraint, repeating new rows, which would then require new background lookukps.
What is the simplest method for automatically enforcing the unique constraint, so that I’m not duplicating efforts or needing to intervene with macros manually?
I am also having a hard time with this forum in my Brave browser. It keeps freezing up for some weird reason and I have to switch to Chrome, which doesn’t let me login as the same user.
Anyways, so far I’ve attempted this both as a UNIQUE constraint and using the COUNT method, without success.
any help on this yet? ive been digging around for videos and such and found nothing. The closest Im seeing is the Autobook for Google Sheets and Product Hunt, though i havent been able to comprehend the steps and if its a fit
We don’t currently have a way to check if results already exist in a sheet before inputting them in a sheet ( I hope I understood you problem correctly).
I’ve created a playbook (https://www.bardeen.ai/playbook/community/Delete-duplicates-U2wExHuPnIlnIHSbzv) that deletes duplicates in a sheet. Hopefully that helps as a guide on how to delete duplicate rows in a sheet. You can add these actions to an existing playbook to remove duplicates in a sheet once a scrape is completed
Regards,
Vin
Customer Support - bardeen.ai
Explore | @bardeenai | Bardeen Community
Thanks Vin, this does help me get closer. It generates a new table within Bardeen that I could theoretically use to overwrite the one containing duplicates. However, I need to scale this across a fairly large number of different websites, so i need to be able to replace that google sheet with the new data without making a lot of new tabs. How would I be able to accomplish that? Does Bardeen support deleting the rows or deleting an entire sheet so i can add it back?
Glad to know we’re one step closer. We have an action to Clear a Google Sheet which does what you’ve described above. Before you populate the google sheet with the scraped results, you can clear a specific tab and then repopulate it with the new results
Customer Support - bardeen.ai
Explore | @bardeenai | Bardeen Community
Hi Vin, thank you for the helpful script. It works for writing unique rows to another new sheet after clearing out the new sheet. However, when the unique rows are written, there are blank rows in between unique rows. I think the row number is being preserved so the rows are written to the same row number. Is there a way to not have blank rows in the middle of the unique rows?
