Once you have confirmed that the source platform supports sending events to BigQuery Stream, follow these steps:
From your RudderStack dashboard, add the source. Then, from the list of destinations, select BigQuery Stream.
Assign a name to the destination and click Continue.
Connection settings
To successfully configure BigQuery Stream as a destination, you will need to configure the following settings:
Project ID: Enter your BigQuery project ID.
Dataset ID: This is the ID of the project dataset for which you specified the Project ID above.
Table ID: Provide the ID of the table into which you want to stream the event data.
Insert ID: This is an optional field. Enter the insertId used by Google to deduplicate the data sent to BigQuery. For more information on this setting, refer to the Deduplicating data section.
RudderStack supports sending only track events to BigQuery Stream. Make sure your track event payload format matches the table schema corresponding to Table ID specified in the dashboard settings.
RudderStack currently does not support the templateSupportSuffix feature which creates a table schema during a streaming insert action.
Suppose you want to stream the events from your web source to BigQuery and the table schema in your BigQuery dataset is as shown:
To successfully stream the events, the event tracked from your JavaScript SDK should look like the following:
Note that the track properties in the above payload match with the fields specified in your table schema. Once streamed, you can view this event in your BigQuery console by running the following SQL command :
Deduplicating data
Google leverages the insertId to deduplicate the data sent to BigQuery. insertId is essentially an event property that uniquely identifies an event.
RudderStack currently supports only numeric or string values as insertId.
For more information on the deduplication process in BigQuery, refer to the BigQuery documentation.
Use case
Consider the following table schema:
When sending an Insert Product event to BigQuery, you can use the productId field to uniquely identify the product. Upon setting productId as the insertId, BigQuery uses it to deduplicate the data.
To dynamically configure insertId via the event payload, make sure that insertId is the column name present in your schema (or in the properties object in the event payload) used to uniquely identify an event.
Consider the following schema:
Suppose you have a dynamic configuration like {{ message.uniqueId || "productId" }} for the above schema. There are three cases to consider here:
Case 1: Unique ID is sent as a value which is not a key in the event properties
In this case, deduplication is applicable as RudderStack sends the productId value (212) as the insertId to Google.
If you use the dynamic destination configuration for insertId by passing a random value (e.g. 1234) in the above payload, deduplication will not be applicable as the properties object does not contain the value 1234.
In the left sidebar, go to APIs & Services > Credentials.
Then, click CREATE CREDENTIALS > Service account:
Enter the service account details and click CREATE AND CONTINUE.
In the Select a role field, search and select the BigQuery Data Editor role and click CONTINUE.
Click DONE to finish the setup.
Next, you need the service account credentials JSON required for RudderStack to send the data to BigQuery. To obtain this JSON, go to your service account.
Then, go to KEYS > ADD KEY > Create new key.
Select the Key type as JSON and click CREATE.
Your JSON key will be automatically downloaded. Copy and paste the contents of this JSON key in the Credentials field while configuring BigQuery Stream as a destination in RudderStack.
Troubleshooting
For the different error messages you might get when sending your event data to BigQuery Stream and their troubleshooting steps, refer to the BigQuery documentation.
This site uses cookies to improve your experience while you navigate through the website. Out of
these
cookies, the cookies that are categorized as necessary are stored on your browser as they are as
essential
for the working of basic functionalities of the website. We also use third-party cookies that
help
us
analyze and understand how you use this website. These cookies will be stored in your browser
only
with
your
consent. You also have the option to opt-out of these cookies. But opting out of some of these
cookies
may
have an effect on your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. This
category only includes cookies that ensures basic functionalities and security
features of the website. These cookies do not store any personal information.
This site uses cookies to improve your experience. If you want to
learn more about cookies and why we use them, visit our cookie
policy. We'll assume you're ok with this, but you can opt-out if you wish Cookie Settings.