I am currently using your service to scrape data and send it to Airtable, but I need some clarification on how to properly map dynamic scraped data fields to my Airtable table.
Problem:
In my current setup, I scrape several lines of data (e.g., part names, part numbers, prices, etc.) and want to send each of these lines to a new record in my Airtable base. However, the number of lines may vary, and I need to ensure that each piece of scraped data is dynamically mapped to the correct Airtable field.
Mapping Fields:
Here are the mappings I would like to establish:
Airtable Field
Scarped Values
(Ad) Amount
Part amount
(Ad) Unit cost
Part cost
(Ad) Part name
Part name
(Ad) Part number Ad
Part number
(Ad) Unit price
Part price
(Ad) Parts Order Number
Parts Order Number
Additional Context:
In Builder v2, there is no option to map the dynamic data fields, which makes it difficult to properly map the scraped data to the respective Airtable fields.
In Builder v1, there is an option for mapping, but unfortunately, it does not seem to work as expected. The data does not get mapped correctly and JSON is not being created at all with error (HttpRequest failed to post: [object Object] (422))
I have tested a basic JSON setup which works as expected for static values:
{
"records": [
{
"fields": {
"(Ad) Part name": "MANUAL VALUE FOR TEST"
}
}
]
}
Request:
How can I dynamically map scraped data from multiple rows to Airtable records?
Should I loop through each scraped data entry (line) to create a separate Airtable record for each one?
Can you provide an example of the JSON body that would work in this case, where the number of scraped lines may vary from time to time?
I would greatly appreciate your assistance in guiding me on how to configure this integration correctly, as it seems like a critical function that is currently not working as expected.
p.s. some Playbooks open in builder v1 and some in builder v2 - despite builder v2 future turned on or off - it has no impact…
The “Send http POST request” action is not needed for this as Airtable is supported as a connector in Bardeen. First connect and provide access to the airtable table you are looking to add new items to. Then in your Bardeen automation, use the “Add data to Airtable” to be able to map your fields easily:
Unfortunately, the Airtable connection isn’t a viable solution for us since the team members using Bardeen only have access to Interfaces. I couldn’t find a way to restrict access to just one table within the Airtable database connected to Bardeen, which would allow them to receive information and process it via automation inside Airtable. I cannot grant access to the full Airtable system but would like to enable access only to a specific table. However, I couldn’t find a way to do this, so I’m looking for alternative solutions.
Please note that I can’t use Google Sheets as an intermediary because, while Bardeen does insert data into Google Sheets which then triggers the Airtable table update, this method is too slow for us. The data is delayed by a few minutes, but we need near-instant data transfer.
Before writing here a request regarding Airtable API, I also tested a direct connection between Bardeen and Airtable which you are recommending, and the data transfer was almost instant, but this works only for my account which has access to all Airtable base. Since I won’t be the one using it daily, this setup isn’t practical for our use case when other users have no access to Airtable base.
Please help me by advising to my first post about connecting Bardee with Airtable via API or maybe there is another easier work around instead of using API - please answer to my initial request where I have described situation in details.
Thank you for your reply. I appreciate your suggestion regarding the Airtable connector. However, I need to insist on using the API for this integration due to the following critical reasons:
Limited Access Requirements
The team members using Bardeen only have access to Interfaces (frontend) in Airtable and do not have full access to the Airtable base (backend). In Bardeen, each user must set up a connection with the Airtable base (backend), but since these users do not have access to the full Airtable base (backend), they cannot complete this setup. Granting full access is not an option because our Airtable base contains a highly sensitive ERP system with critical settings that could be disrupted accidentally. For security and operational reasons, users (managers) are restricted to the Interface (frontend) only, which they use daily to manage business processes.
I hope this explanation clarifies how our Airtable setup functions and highlights the limitations of using the Bardeen+Airtable integration in our specific scenario—one that I believe is quite common.
By the way, Airtable has launched beta testing for Business users, allowing external users to access the backend, with a planned rollout expected sometime next year. This indicates that use cases like ours will likely become more prevalent in the future.
No Workaround with Google Sheets
As I mentioned earlier, using Google Sheets as an intermediary is not a viable option because the delay in data transfer (even a few minutes) disrupts our workflow. We need near-instant updates, which makes the direct API integration the only practical solution.
API Offers Full Control and Flexibility
The API allows us to map dynamic scraped data fields to Airtable records programmatically, which is essential since the number of scraped rows varies. This level of flexibility is not achievable with the connector or existing built-in actions.
Current Testing Challenges with Builder
While the connector may work in certain scenarios (for users with Airtable base level access like me who is admin but not actually final user) , the issues I’ve outlined (dynamic field mapping, variable row handling, and JSON creation) are not addressed by the connector and require API-level control to resolve effectively. I need help with the API setup to ensure we can handle these challenges.
I kindly ask for your support in addressing my original request regarding the proper JSON structure and method for dynamically mapping scraped data to Airtable records using the API. This will allow me to ensure a seamless and secure integration that fits our use case.
I hope this clarifies why the API is the preferred solution in our scenario. I look forward to your guidance.
I’d suggest creating a Team workspace in Bardeen that allows team members to access the automation via the shared connector to Airtable. This keeps your security concerns addressed and will allow you to easily map the fields.
Thank you for your suggestion regarding the Team Workspace. I want to clarify that we already have a shared Team Workspace in Bardeen, where all team members can access the same Playbooks. However, as mentioned earlier, the issue lies in the Airtable connection.
Even within the shared Team Workspace, each team member is required to create their own connection to the Airtable base. This is not feasible in our case since these users do not have access to the full Airtable base. Without this access, they cannot establish the necessary connection, and granting them such access is not an option due to the sensitivity of our data.
This limitation is why I am insisting on an API-based solution for this integration. I understand the ease of using the Airtable connector, but it does not resolve our use case, which involves restricted access for users.
Could you please revisit my initial request and assist me with guidance on how to achieve the integration using the Airtable API? Specifically, I am looking for help with:
Dynamically mapping scraped data fields to Airtable records via the API.
Handling scenarios where the number of scraped rows varies.
A proper JSON body example that supports this functionality.
Your expertise on the API setup will be crucial in resolving this issue. I kindly ask for your support in addressing my original question, as the connector solution, unfortunately, does not meet our requirements.
Thank you so much for the detailed explanation and the examples—this looks very promising! My apologies as well for not clarifying things better initially.
I’ll give your suggestion a try soon, as it seems like exactly what I’ve been looking for. If I run into any issues during the setup, I might reach out again for further guidance.
All works great!
There is a minor issue - when on first step I am asking user to enter data manually, in settings it is impossible to save Title (body) - after inserting new Title it stays ‘body’.
Report an issue