Congratulations Transformations Challenge Winners!

Data teams use Transformations to ship data projects faster, secure and build data trust, and quickly adapt to changing business needs. Last month, we launched several powerful new Transformations features that expand its flexibility and usability.

We also held our first ever RudderStack Transformations Challenge, where users competed for prizes by submitting their own transformations to our open source library.

This post will highlight the winning transformations, why we love them, and give you a sneak peek of even more Transformations features coming soon.

🏆 Winning Transformations

To highlight the flexibility of Transformations and the new features available in the Transformations suite, we launched our first ever RudderStack Transformations Challenge.

First, we'd like to thank everyone who participated in this challenge. By submitting your use cases and code, you've provided useful knowledge to the community, helped improve our Transformations feature, and inspired other data professionals with creative solutions to common challenges.

While the steps were simple, providing a useful, innovative, and easy to understand transformation for users to apply to their own projects was no easy feat. Each transformation was evaluated using the following criteria:

Usability
Innovation
Clarity/Documentation

To decide the winners, 4 judges (3 external and one from RudderStack), ranked each submission based on those three criteria. Many thanks to our judges, Max, Satyam, Steve and Emil for their expertise and perspective that brought us to our final list of winners.

We are excited to announce the top 3 winners of our Transformations Challenge, their winning transformations, and the key elements that stood out to the judges and the RudderStack team.

🥉 3rd place: Convert UTC Timestamp to Local Date and Time

by pgiannone33

JAVASCRIPT
export function transformEvent(event) {

    const timeZone = event.context?.timezone;
    const originalTimestamp = event.originalTimestamp;
    const locale = event.context?.locale || "en-US";

    if (timeZone && originalTimestamp) {
        const date = new Date(originalTimestamp);
        const localDateTime = date.toLocaleString(locale, {
            ...options,
            timeZone
        });
        event.localTimestamp = localDateTime;
    }

    return event;
}

View the submission in GitHub here

Category: Data Processing and Enrichment

What it does:

This transformation can be used to dynamically convert an ISO 8601 timestamp to a user's local date and time based on their time zone.

Why we love it:

Same reason we love Transformations - flexibility! Many businesses offer services to users in various geographic regions. Personalizing the experience by adjusting to the local timezone makes it easier for users to schedule actions, such as emails, social media posts, or other interactions. This transformation saves time and effort by avoiding the need to manually convert timestamps from GMT time to the client's local time at each destination.

🥈 2nd place - Filter Bot Traffic

by amiryselim

PYTHON
def transformEvent(event, metadata):
    lower_user_agent = event["context"]["userAgent"]
    known_bot_filter_list = ['bot', 'crawler', 'spider', 'feedfetcher-google', 'mediapartners-google', 'apis-google', 'slurp', 'python-urllib',’seostar.co',....]
    if any([bot_name in lower_user_agent for bot_name in known_bot_filter_list]):
        return
    else:
        return event

View the submission in GitHub here

Category: Data Security & Governance

What it does:

This transformation helps filter events that are generated by known bot agents.

Why we love it:

According to Cloudflare, bot traffic accounts for 40% of overall internet traffic. However, many destinations, such as data warehouses, lack the ability to filter out bot traffic. This transformation simplifies the process of obtaining higher quality trusted data, allowing data engineers to spend less time cleaning the data by filtering out those events downstream in the warehouse, data lake or other analytics tools.

🥇 1st place - Properties Schema Validation

by Ioannis Foukarakis, Senior Data Engineer at Mattermost

JAVASCRIPT
export async function transformBatch(events, metadata) {
    // Create a registry for schemas
    const contracts = new Contracts();
    // Register schema for event "Add To Cart" from a URL
    await contracts.registerSchemaFromURL("Add To Cart", "https://raw.githubusercontent.com/ifoukarakis/tests/main/product.json");
    // Register a different schema for event "User Registered" from a URL
    await contracts.registerSchemaFromURL("User Registered", "https://raw.githubusercontent.com/ifoukarakis/tests/main/person.json");
    // Register more events here.

    return events.filter(event => contracts.validateProperties(event))
}

View the submission in GitHub here

Category: Security & Governance

What it does:

This transformation enables users to create a data contract that serves as an agreement between engineering teams and the teams responsible for instrumenting events. This contract defines what the event schema should look like.

Why we love it:

One of the biggest pain points for data engineering teams is downstream data quality issues. These issues often arise when the format of the collected data differs from the format of data imported into the data warehouse or cloud destinations. To ensure data consistency across different destinations, users can plan their events and specify what they should look like using data contracts. Transformations provide the flexibility to allow teams to create their own data contracts that define their event schema.

RudderStack users can either build data contracts within transformations or use tracking plans and the data governance API to define event properties and enforce their own imported schema.

Coming soon to Transformations

Over the course of this next year, we will continue to release new Transformations features that are focused on enhancing observability and flexibility.

Here’s a sneak peek of just a few new Transformations releases coming soon:

Real-time error alerting: Identify and debug transformations errors quickly by viewing errors real-time and enrolling in error notifications

Additional Transformations UI improvements: Users will be able to connect to destinations from the Transformations tab in the UI, and see the difference between input vs output payload.

Getting started with Transformations

Over 75% of RudderStack customers use Transformations to simplify their data enrichment, processing, governance, privacy, and custom integration use cases.

To get started, all you need is a RudderStack account. Get started for free or check out our pricing page to find the plan that's right for you.

April 13, 2023