How to send data from apache kafka to braze

Introduction

In today's rapidly evolving tech landscape, the need for real-time data integration has become paramount. Developers and data engineers find themselves at the forefront of this change, challenged to keep up with the mounting demands of delivering timely insights for smarter decision-making. This is where tools like Apache Kafka and Braze come into play. Apache Kafka, an open-source stream processing platform, empowers businesses to handle massive streams of real-time data. On the other side, Braze, a leading customer engagement platform, thrives on these real-time insights to fuel personalized marketing campaigns and push notifications, ensuring that marketers reach their end-users at just the right moment.

Imagine you're a data engineer at a bustling e-commerce platform. As midnight strikes, launching a worldwide flash sale, millions of users flood your website. At such crucial moments, the integration of real-time event data from Apache Kafka directly into Braze can be the difference between a seamless customer experience and missed opportunities. This guide aims to bridge that gap, ensuring that businesses can harness the power of real-time data for unparalleled customer engagement.

Understanding Apache Kafka and Braze

What is Apache Kafka?

Apache Kafka, at its essence, is a distributed event streaming platform. What does this mean? Picture a high-capacity conveyor belt, continuously transporting information from multiple sources to multiple destinations. That's Kafka in the realm of data. Designed by the team at LinkedIn and later open-sourced, Kafka has since taken the world of real-time data processing by storm.

Let’s understand what are the core functionalities provided by Kafka.

The core functionalities of Apache Kafka include:

  • Publish and Subscribe: Data is published to topics by producers and read from topics by subscribers, ensuring a system where data sources and recipients remain decoupled.
  • Data Storage: Beyond real-time transmission, Kafka retains large datasets for a set duration, allowing repeated access by multiple applications.
  • Stream Processing: With Kafka Streams, data can be processed and transformed in real-time during transit.
  • Fault Tolerance and Scalability: Kafka's design ensures resilience and high availability. It can expand by adding more nodes, meeting growing data demands.

Now that you understand the key features, let’s understand what Kafka can be used for.

Some of the notable use cases for Kafka include:

  • Centralizing data across company divisions
  • Foundation for data platforms and event-driven architectures
  • Instant response to customer interactions in retail and travel
  • Real-time processing of financial transactions in sectors like banking
  • Real-time tracking of vehicles and shipments in logistics
  • Analyzing sensor data in real-time for factories and renewable energy
  • Monitoring hospital patients for immediate emergency care

What is Braze?

Braze is a sophisticated customer engagement platform that allows businesses to communicate with their users in a more personalized and timely manner. Using Braze, companies can send targeted messages based on user behavior, preferences, and real-time data. Built with scalability in mind, it's designed to handle vast amounts of data and reach users across various channels such as in-app messages, emails, push notifications, and more.

Key Functionalities of Braze:

  • Segmentation: Using a plethora of criteria, companies can segment their user base, ensuring that each message is tailored to its recipient.
  • Multichannel Messaging: From SMS to push notifications and in-app messages, Braze supports a multitude of channels, ensuring users are reached wherever they are.
  • Campaign Automation: Automated campaigns based on user behavior or other triggers can be set up, ensuring timely engagement.
  • Analytics and Reporting: Post-campaign, Braze provides in-depth metrics and insights into how users interacted with the messages, aiding in future campaign optimizations.

Use Cases:

  • E-commerce: Real-time recommendations based on browsing history, abandoned cart reminders, and personalized sale notifications.
  • Finance: Transaction alerts, account updates, or personalized financial tips based on spending patterns.
  • Gaming: Player milestones, in-app purchase incentives, or updates about new game features.
  • Travel & Hospitality: Last-minute deals, travel itinerary updates, or loyalty program promotions.

In essence, Braze emerges as a powerhouse for businesses aiming to foster strong, lasting relationships with their customers by harnessing the might of real-time data and automation. Whether you're in the retail industry aiming to boost sales or a content platform striving for higher retention, Braze offers the tools necessary to achieve these goals.

Why send data from Apache Kafka to Braze?

Integrating Apache Kafka with Braze offers businesses a unique opportunity to leverage real-time data in enhancing customer experiences. From the perspective of a user or a business, here are the primary benefits of this integration:

1. Tailored Experiences: Real-time data synchronization means that as a user interacts with a platform or product, their experiences can be immediately personalized based on their actions. For instance, if a user browses a specific product category, they can receive targeted promotions or content related to that category in near real-time.

2. Instant Feedback Loop: Businesses can immediately gauge user reactions to new features or campaigns. If a new product launch isn’t resonating, the integration allows for quick pivots, ensuring that the user experience remains optimal.

3. Consistency Across Channels: With data streaming into Braze in real-time from various sources via Kafka, users experience consistent messaging and branding across all touchpoints, be it emails, in-app notifications, or other communication channels.

4. Enhanced Engagement: Real-time data means more timely and relevant push notifications, emails, or in-app messages, increasing the likelihood of user engagement.

5. Operational Efficiency: Businesses benefit from reduced latency in their data pipelines. Instead of batch processing, which might introduce delays, real-time processing ensures that marketing and operational decisions are based on the freshest data available.

While integrating Apache Kafka with Braze can supercharge personalized marketing, businesses may face challenges like increased complexity in setup and potential data privacy concerns. Additionally, there's a misconception that immediate data availability will guarantee instant campaign success, overlooking the need for strategic planning and iteration. So keeping those in mind, let’s start with the integration.

Sending Data from Apache Kafka to Braze

There are different ways of integrating Kafka with Braze such as:

  • Kafka Connect with Braze Connectors: Kafka Connect is a tool provided by Kafka to connect Kafka with various systems. The Confluent Hub and other open-source communities offer pre-built Kafka connectors. They provide both source connector as well as sink connector (the one we are interested in to send data to Braze). As of now, we didn’t find any Sink connector for Braze. So if you want to utilize this method, you will have to create your own custom Kafka Connector.
  • Indirect Integration: We can utilize an intermediary storage solution and then programmatically, we will send data from that storage to Braze. Data warehouses could be a good choice as an intermediary storage. For example, Amazon S3, it also has a Kafka connector available, so sending data in real-time with low latency to S3 will be easy.
  • Middleware Solutions: There are third-party middleware tools that can act as intermediaries, handling the data transformation and flow between Kafka and Braze.

As there’s no Kafka Connector for Braze destination, we will choose the indirect integration using AWS S3 as intermediary data warehouse and then we will use Python programming to send data from S3 to Braze.

Let’s get started with the steps of this Kafka to S3 to Braze integration:

1. Prerequisites:

Before embarking on this integration journey, ensure you're equipped with:

  • Braze account: Make sure to set up a Braze account and generate the API keys. We will need the API keys in the later steps.
  • Amazon S3: Make sure to set up an Amazon S3 account. Get the AWS credentials and bucket details, we will need them later.

2. Setting up Apache Kafka:

Ensure you've installed Kafka. Understanding the nuances of Kafka clusters and the Braze API is crucial, so make sure to read the introduction.

Ensure that your Kafka cluster is running and healthy. This includes the Kafka brokers, ZooKeeper instances, and any other necessary infrastructure.

Also, you should have a pre-defined topic from which data will be fetched. Data pushed to this topic will be what's sent to Braze. Then decide on a unique consumer group name for this integration, ensuring that offsets are properly managed.

a. Dedicate Kafka topics for Braze:

SH
/path/to/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic BrazeDataTopic

b. Setup producers

To get the customer data to Kafka, you may use the appropriate Kafka Source Connector depending upon where you are collecting this data from. If a connector is not available, you may send the data to the producer programmatically. And then, we can go ahead with the next steps of transforming and consuming that data.

Read Kafka documentation to understand various concepts related to Kafka.

3. Preparing data for Braze

Braze primarily accepts JSON. If you’re using Python, you can use `json.dumps` to convert the data to `json`.

PYTHON
import json








data = {


'user_id': '12345',


'event': 'purchase',


'value': 29.99


}








json_data = json.dumps(data)

Make sure to transform your json data to meet the schema requirements of Braze.

4. Configure intermediary storage

Since the direct connectors between Kafka and Braze aren't available, we'll utilize an intermediary storage solution. Here, we'll demonstrate using AWS S3:

a. Send Data from Kafka to AWS S3:

First, you'd need a tool to export data from Kafka to S3. A popular choice is Kafka Connect with the S3 sink connector. Follow this guide to set it up. It involves setting AWS credentials and updating `kafka-connect-s3/quickstart-s3.properties` as following:

SH
confluent local services connect connector load s3-sink








{


"name": "s3-sink",


"config": {


"connector.class": "io.confluent.connect.s3.S3SinkConnector",


"tasks.max": "1",


"topics": "s3_topic",


"s3.region": "us-west-2",


"s3.bucket.name": "confluent-kafka-connect-s3-testing",


"s3.part.size": "5242880",


"flush.size": "3",


"storage.class": "io.confluent.connect.s3.storage.S3Storage",


"format.class": "io.confluent.connect.s3.format.avro.AvroFormat",


"schema.generator.class": "io.confluent.connect.storage.hive.schema.DefaultSchemaGenerator",


"partitioner.class": "io.confluent.connect.storage.partitioner.DefaultPartitioner",


"schema.compatibility": "NONE",


"name": "s3-sink"


},


"tasks": []


}

b. Upload Data from S3 to Braze:

Using Python, you can extract the data from S3 and push it to Braze. We will use AWS SDK for Python (`boto3`) to make it easy.

PYTHON
import boto3


import requests


import json








# Extract data from S3


s3 = boto3.client('s3')


data = s3.get_object(Bucket='your-bucket-name', Key='your-data-key')


data_content = data['Body'].read().decode('utf-8')


data_json = json.loads(data_content)








# Push to Braze


braze_url = "https://YOUR_BRAZE_URL_ENDPOINT/users/track"


headers = {


"Content-Type": "application/json",


"Authorization": "Bearer YOUR_BRAZE_API_KEY"


}


response = requests.post(braze_url, headers=headers, data=json.dumps(data_json))


//TODO: Handle request success/failures appropriately

Ensure you replace placeholders (`YOUR_BRAZE_URL_ENDPOINT`, `YOUR_BRAZE_API_KEY`) with appropriate values.

5. Monitoring & Troubleshooting:

Leverage tools like Grafana for monitoring and set alerts for anomalies.

Further Reading:

Note, this is just a simple example. For production systems, you should also think about:

  • Data Mapping: Ensure Kafka messages fields match Braze object fields.
  • Security and Privacy: Ensure secure data transmission using HTTPS for Braze and SSL for Kafka. Implement necessary mechanisms for data handling to ensure data privacy according to applicable privacy regulations such as GDPR, CCPA, etc.
  • Braze API Limits: Remember, Braze has API request limits based on your license. Be mindful of this when setting up your data streams.

By following this guide, you can seamlessly funnel real-time Kafka data into Braze. Always test in a non-production environment first to ensure data integrity and proper integration.

Conclusion

We explained a method to send data from Kafka to Braze and provided references to help with integration. We showcased how you can send customer data from Kafka to AWS S3 as an intermediary storage and then programmatically send that data to Braze. We used Python for the code but you can use any programming language of your choice.

With this integration, we leveraged the potency of Apache Kafka's real-time stream processing with Braze's advanced customer engagement capabilities. By integrating Kafka with Braze, you unlock the door to real-time personalized marketing, creating enriched user experiences. We strongly urge you to commence this integration journey.

Don't want to go through the pain of direct integration? RudderStack's Apache Kafka Source integration makes it easy to send data from Apache Kafka Source to Braze.