Set Up Reverse ETL Connection

Connect your Reverse ETL sources to destinations in RudderStack.

Follow these steps to set up a Reverse ETL connection in RudderStack:

  1. Create a Reverse ETL source
  2. Connect it to a destination
  3. Specify the data mappings
  4. Define a sync schedule
  5. Configure advanced settings
  6. Activate the connection
info
You can set up to 10 connections in the RudderStack Cloud Free and Starter plans and unlimited connections in the Growth and Enterprise plans.

Add source

info
Reverse ETL connections are source-driven. You must create a source first and then connect it to a destination.
  1. Sign in to your RudderStack dashboard and click + Add source in the default Connections view.
Add source
  1. Under Sources, click Reverse ETL and select your warehouse source.
Select source
  1. Configure your source. See the source-specific documentation for configuration details.
warning

RudderStack requires some warehouse-specific permissions to sync data from it. These permissions are listed in the source documentation, for example, see BigQuery permissions.

Make sure to grant these permissions before you set up the source.

Connect destination

Once you have created a Reverse ETL source, connect it to an existing or new RudderStack destination.

info

RudderStack supports connecting multiple Reverse ETL sources to a destination.

Note that this feature is available only for the Facebook Custom Audience destination currently. Support for more destinations is coming soon.

To connect a destination:

  1. Click Use existing destination or Set up a new destination depending on your requirement.
warning
You can connect an Audience source only to Audience and List destinations.
Connect destination options
  1. Enter the connection settings for the destination. See the destination-specific documentation for more details on these settings.

Specify data mappings

RudderStack provides two options to map your warehouse columns to specific destination fields before syncing your data:

info

RudderStack supports the Map with Visualizer option only for selected destinations.

Other destinations that do not support Visual Data Mapper have the Map with JSON option by default, with the Map with Visualizer option greyed out.

The steps to set the data mappings (using JSON) are listed below:

  1. Select the Sync mode to specify how RudderStack syncs each record to the destination. RudderStack supports two sync modes:
Sync modeDescription
UpsertIn this mode, RudderStack inserts new records and updates the existing records in the destination while syncing the data.
MirrorIn this mode, RudderStack ‘mirrors’ the source by keeping the destination data the same as the source data. It performs insertion, deletion, and updates to the records while syncing the data.

Note that:
  • Only select destinations support this mode.
  • RudderStack does not support track event type for this mode.
    1. Select the Event Type - RudderStack supports sending the synced data to your downstream destinations as identify or track events.

    If you select track as the Event Type, you also need to set the event name that is sent to the downstream destination. This is because every track event requires a name. You can also set the column name as the event name by turning on the Lookup event name by column toggle and selecting the column from the dropdown:

    Schema tab options in RudderStack
    1. Set at least one user identifier from user_id or anonymous_id in the Choose user identifier setting.
    Map with JSON feature
    1. Preview the resulting data and make the necessary changes before proceeding.

    Schedule syncs

    You can configure the schedule settings and sync frequency at the connection level. This is because a source might be connected to multiple destinations and the sync interval and frequency might need to be configured differently for each connection.

    RudderStack determines how and when to run a sync based on the sync schedule you set for your Reverse ETL connection.

    Schedule typeDescription
    BasicRun syncs at a given time interval and specified time (in UTC).
    CRONRun syncs based on a specified CRON expression (in UTC).
    ManualRun syncs manually.
    Sync schedule

    Configure advanced settings

    warning
    This option is available only if you have set up a new destination.

    Once you create a connection successfully, you can configure the optional advanced settings to receive the data correctly in your destination.

    Advanced destination configuration
    1. Click Configure your destination.
    2. Specify the advanced settings and click Save.

    Activate connection

    Stop data syncs for connection

    Go to the connection page and turn off the connection toggle to stop data syncs for the connection.

    Note that the Sync Now button is greyed out and the connection status is Paused until you turn on the connection again.

    Turn off connection

    Update mapping configuration

    Go to the Schema tab of your connection and click Update to change the mapping configuration and update your column selection. Then, click Save to update and save the configuration.

    warning
    The Object (for VDM mapping), Sync mode, Event type, and User identifier fields are not editable.
    Update connection configuration

    Update connection settings

    Go to the Settings tab of your connection to:

    • Get the Connection ID.
    • Change the sync schedule and frequency.
    • Specify the settings to retain sync logs in your warehouse and retry syncing failed records. See Syncs Observability for more information.
    • Delete the connection permanently.
    Update connection settings

    View all connections for Reverse ETL source

    Go to the Overview tab of your Reverse ETL source. Here, you will see all the destinations connected to that source. Click the required destination or click View > Connection details to go to the connection’s page.

    Connection details

    RudderStack automatically redirects you to the Syncs tab where you can view the latest sync details or check past syncs.

    View sync details

    Go to the Syncs tab to see detailed metrics on the latest sync. You can also view details of the past syncs (up to 1 month). These details include:

    MetricDetails
    StatusThe sync status. It can be one of the following:

    • Aborted
    • Completed, with failures
    • Completed, no failures
    Sync modeDefines how RudderStack syncs data to the destination. It can be one of the following:

    • Upsert
    • Mirror
    Sync typeDetermines the scope of the sync. It can be one of the following:

    • Incremental: RudderStack syncs only the newly added data in the warehouse since the last sync.
    • Full: RudderStack syncs all the data irrespective of whether it was synced to the destination previously.
    TriggerDetermines how the sync was triggered. It can be one of the following:

    Sync start timeDetermines when the sync was triggered.
    DurationThe sync duration.

    You can also see the following source and destination-specific metrics:

    Sync details

    View invalid records

    Click View invalid records and click a row to see all the invalid records that RudderStack did not sync to the destination.

    You also see the error message and sample erroneous rows in this view.

    View invalid records

    View sync graph

    warning

    This graph is only available for:

    RudderStack provides an intuitive visual graph detailing the records sent to the destination over time. The X-axis represents the time of the day and the Y-axis represents the number of records classified as:

    • Successfully synced records
    • Dropped records
    • Records that failed to sync
    View sync graph

    Switch to the Retries tab to get a graphical view of the retries over time. The X-axis represents the retry time of the day and the Y-axis represents the number of retry attempts.

    Sync retries graph


    Questions? Contact us by email or on Slack