Use Transformations in Cloud, Device, and Hybrid Mode

This guide explains how to use transformations with destinations connected in different RudderStack connection modes.

Transform a single event

While using a transformation, RudderStack applies the transformEvent function on each event that takes the following two arguments:

After the transformation is complete, the transformEvent function returns the final event to be sent to the destination.

Access event metadata

RudderStack injects a function metadata(event) into your transformations as an argument. This lets you access the event metadata variables to customize your transformations.

metadata() takes the event as the input and returns the metadata of the event.

The following properties, if available, are present in the metadata response:

Example transformations using the metadata data method are shown below:

export function transformEvent(event, metadata) {
  const meta = metadata(event);
  event.sourceId = meta.sourceId;

  return event;
}

def transformEvent(event, metadata):
    meta = metadata(event)
    event['sourceId'] = meta['sourceId']
    return event

Make external API requests

You can make any number of external API requests in your transformation functions and use the fetched responses to enrich your events.

JavaScript

RudderStack injects an asynchronous fetch function in your transformations. It makes an API call to the given URL and returns the response in the JSON format.

You can use the fetch function in your transformations:

export async function transformEvent(event, metadata) {
  const res = await fetch("post_url", {
    method: "POST",  // POST, PUT, DELETE, GET, etc.
    headers: {
      "Content-Type": "application/json;charset=UTF-8",
      Authorization: "Bearer <authorization_token>"
    },
    body: JSON.stringify(event)
  });
  event.response = JSON.stringify(res);
  return event;
}

To see the fetch function in action, refer to the Clearbit enrichment example.

For improved performance, it is highly recommended to use the batch API requests wherever possible instead of a separate API request for each event.

Fetch v2

fetchv2 is a wrapper for the fetch call. It enables you to fetch the response properties more efficiently while making the external API calls.

The fetchv2 response contains the following properties:

The following example highlights the use of the fetchV2 function in a transformation to capture failure due to a timeout:

export async function transformEvent(event) {
  try {
    const res = await fetchV2("url", { timeout: 1000});
    if (res.status == 200) {
      event.response = JSON.stringify(res.body);
    }
  } catch (err) {
    log(err.message);
  }
  return event;
}

Python

You can use Python’s requests package to fetch response properties while making the external API calls:

import requests

def transformEvent(event, metadata):
    res = requests.get("url")
    if res.status_code == 200:
        event["response"] = res.json();
    return event

Transform batch of events

You can perform any aggregation or roll-up operation on a batch of events using the transformBatch function instead of transformEvent function:

export function transformBatch(events, metadata) {
    return events;
}

def transformBatch(events, metadata):
    return events

To ensure event ordering when using the transformBatch function, make sure you pass the messageId from the input event to the output event. Without the messageId, RudderStack does not guarantee event ordering.

It is highly recommended to use transformEvent as much as possible, as it ensures event ordering.

Cloud mode

When you add a transformation and connect it to a destination in cloud mode, RudderStack does the following:

1. Tracks and collects events at the source.
2. Applies the user transformation logic to your events.
3. Transforms the events in the destination-specific format. This is done internally and requires no user intervention.
4. Forwards the transformed events to your destination.

Connect cloud mode destination

You can connect only one transformation to a destination. However, one transformation can be used by multiple destinations.

There are two ways to connect a transformation to a destination:

From transformation

1. Click the Connections tab of your transformation and click Connect Destination. You will see a list of all the destinations and the transformations connected to them.
2. Scroll to the destination you want to connect to the transformation and click Connect.

3. Enable the Connect to transformation toggle and click Save.

From destination

1. Go to the destination in the dashboard. Click the Transformation tab and click Add a transformation:

2. Select the transformation to connect to the destination and click Choose.

Device mode

Device mode transformations is a beta feature and available for the Enterprise plan users only.

You can use transformations with destinations supporting device mode, like Firebase, Hotjar, and so on.

Device mode transformations leverage our Client Transformation service to transform events, then send the transformed events directly to the device mode destination through their native SDK.

Learn more about the architecture in the Client Transformations Service architecture section below.

Note that:

• Only the JavaScript, Android, and iOS SDKs support device mode transformations.
• You can write device mode transformations only in JavaScript. Python is not supported currently.
• The transformations you use in device mode must adhere to the RudderStack message schema. Otherwise, the event will be dropped.

Connect device mode destination

You can enable device mode transformations only from the Transformations tab.

Prerequisites

Before you set up a device mode transformation:

• Choose the connection mode as Device mode while configuring your destination.

• In some cases, setting up a device mode destination involves enabling the Use device mode to send events toggle while configuring the destination.

• You can also change the connection mode after the destination is set up. Go to the destination’s Configuration tab and click Edit configuration.

• Some destinations support only device mode. See the destination-specific documentation for the supported connection modes.

Connect transformation to device mode destination

1. Write the transformation to connect to your device mode destination.
2. Set up your device mode destination in RudderStack.
3. In the RudderStack dashboard, go to Collect > Transformations in the left sidebar. Then, select the transformation to connect to this destination.
4. Click the Connections tab . If you don’t have any destinations connected to it, click Connect Destination. If the transformation already has connected destinations, click Manage destinations.

Depending on whether the destination you set up (in Step 1) is already connected to some transformation, there are two ways to add a device mode transformation to it:

• If your destination is not connected to any transformation click Connect. You will see a fly window with these options:

• If your destination is already connected to some other transformation, click the Edit connections button to switch your transformation. See Switch transformation for more information. Once you switch the transformation, you will see the above options to enable the device mode transformation.

5. Configure the following settings under Device Mode:

• Enable the Connect to transformation toggle.
• Enable the Propagate errors toggle depending on your requirement.

Note that:

• Under Cloud Mode, the Connect to transformation toggle is enabled by default. Do not disable this toggle as it will not allow you to configure any device mode transformation settings.
• If Propagate errors is enabled, RudderStack sends the events to the destination without transforming the events, for any transformation errors. This is helpful in preventing data loss in case of transformation code or runtime errors.
• If your transformation involves hashing PII, enabling this setting will send the untransformed event data in case of any transformation errors.

6. Click Save to confirm the settings and for the changes to take effect.

To confirm that the device mode transformation is connected, go to the Connections tab of your transformation and check the Connection Mode column:

SDK setup

After adding a transformation and connecting it to a device mode destination, you can connect your SDK source to it.

Follow the below steps for setting up the required SDK:

implementation 'com.rudderstack.android.sdk:core:1.16.0'

pod 'Rudder', '1.17.0'

Tokenization

RudderStack provides the tokenization feature in device mode transformations to add an extra layer of security. The token is available as a metadata in the transformation and you can validate it before attaching with the RudderStack SDK.

A sample code to access the token in the transformation:

export function transformEvent(event, metadata) {
  const dmtToken = metadata(event)["Custom-Authorization"];
  //verify the token
  ...
  return event;
}

You can use the following methods to attach the token (in the string format) to the SDK:

rudderanalytics.setAuthToken("my-token");

RudderClient.putAuthToken("my-token")

[RSClient putAuthToken:@"my-token"];

You can call these methods multiple times.

RudderStack clears the token when the user logs out, that is, when the reset() API is called.

Client Transformation service architecture

RudderStack provides the Client Transformation service to facilitate transformations for device mode destinations. It ensures that the event ingestion is unaffected, ensuring minimum response time from the RudderStack backend.

Workflow

When you add a transformation and connect it to a device mode destination supporting the device mode:

1. RudderStack SDK sends the event to the Client Transformation service. It multiplexes the event for different destinations and connects with the transformations attached to these device mode destinations.
2. Client Transformation service sends the event to the RudderStack Transformation service to transform the event according to the functions defined in the attached transformations.
3. After the transformation is applied, the transformed event is returned to the Client Transformation service.
4. Client Transformation service responds to the SDK with the transformed events for all the destinations.
5. Finally, the SDK parses the transformed events and forwards them to the specified device-mode destinations.

If the request to the Client Transformation service fails, the RudderStack SDK makes three retry attempts. If it fails, the SDK forwards the untransformed events to the destination or drops them - based on the Propagate errors toggle while configuring the device mode transformation (see Step 6).

Debugging

Check the Network tab in the browser’s developer tools for any transform network requests to debug errors in your device mode transformations. These requests can help you determine if:

• The transformation has run as expected.
• The responses are as expected, based on the sent batch or otherwise.

Performance considerations

RudderStack makes a network call for each event (in case of JavaScript SDK) or a batch of events (in case of mobile SDKs) when you use transformations with device mode destinations. Hence, there can be an added performance cost.

Hybrid mode

While using transformations with destinations connected in hybrid mode like GA4, Braze, Leanplum, or Rockerbox, you can choose to transform:

• Events sent in both cloud and device mode: By default, the transformation is applied to both the cloud and device mode events.
• Only device mode events: Access the mode field in your transformation’s event metadata and check if its value is deviceMode. Then, specify the transformation logic to be applied to the device mode events:

export function transformEvent(event, metadata) {

  const mode = metadata(event)["mode"];
  if(mode === "deviceMode") {
      //some transformation
      return event;
    }

    return event;
}

• Only cloud mode events: Return the device mode events as it is using the mode field and specify the transformation logic for the rest of the events (cloud mode events), as shown:

export function transformEvent(event, metadata) {

  const mode = metadata(event)["mode"];
  if(mode === "deviceMode") {
      return event;
    }

    ... //some transformation
    return event;
}

Limitations

A transformation connected in either cloud mode or device mode must follow the below memory and time limits. It fails if these limits are exceeded:

The following limitations are applicable when invoking a transformation in device mode:

• The native destination SDKs might persist some captured data - mainly identifiers and traits. So, in device/hybrid mode, you cannot transform these fields using device mode transformations.
• Network unavailability can lead to higher latency than expected while sending the events to the destinations. RudderStack doesn’t lose data due to the network loss and stores the events in the client device until they are successfully delivered to the destination.
• The iOS SDK does not support the background processing of an event when the app is closed. However, it sends the pending events the next time the app is opened.

Questions? Contact us by email or on Slack