How to load data from Mixpanel to Snowflake
Extract Mixpanel Data
Mixpanel is an analytics-as-a-service application, so naturally, it requires data to offer its analytics features. We usually think of it as a consumer of data and not a place where we would get data to perform analysis. But Mixpanel collects a lot of data related to how your customers use your product, and in the case where you would like to do anything that also involves data coming from other sources, you have two choices.
The first one is to enrich data coming from Mixpanel with data coming from other sources. The second one is to extract any data Mixpanel holds for you and load it on a data warehousing repository for further analysis. This post will consider the second case.
Mixpanel is evolving into a platform where apart from the analytics services that it offers, you will also be able to build integrated applications. In this post, we will work only with the Export API to export our data stored in Mixpanel.
As a web API, you can access it using tools like CURL or Postman or your favorite http client for the language or framework of your choice. Some options are the following:
- Apache HttpClient for Java
- Spray-client for Scala
- Hyper for Rust
- Ruby rest-client
- Python http-client
Or you can use the libraries/SDKs that Mixpanel offers for the following languages:
- Python
- PHP
- Ruby
- Javascript
As a RESTful API it offers the following resources that you can interact with:
Annotations
- annotations– list the annotations for a specified date range.
- create– create an annotation
- update– update an annotation
- delete– delete an annotation
Export
- export– get a “raw dump” of tracked events over a period of time
Events
- events– get total, unique, or average data for a set of events over a time period
- top– get the top events from the last day
- names– get the top event names for a period of time
Event Properties
- properties– get total, unique, or average data by a single event property
- top– get the top properties for an event
- values– get the top values for a single event property
Funnels
- funnels– get data for a set of funnels over a time period
- list– get a list of the names of all the funnels
Segmentation
- segmentation– get data for an event, segmented and filtered by properties over a time period
- numeric– get numeric data, divided up into buckets for an event segmented and filtered by properties over a time period
- sum– get the sum of a segment’s values per time unit
- average– get the average of a segment’s values per time unit
- Segmentation Expressions– a detailed overview of what a segmentation expression consists of
Retention
- retention– get data about how often people are coming back (cohort analysis)
- addiction– get data about how frequently people are performing events
People Analytics
- engage– get People Analytics data
Let’s assume that we want to export our raw data coming from Mixpanel. To do so we’ll need to execute requests to the export endpoint. An example of a request that would get us back raw events from Mixpanel looks like this:
JAVASCRIPT
The returned result is always in JSON serialization with one event per line sorted by increasing timestamp. It looks like the following sample:
JAVASCRIPT
Mixpanel Data Preparation for Snowflake
Before you start ingesting your data into a Snowflake data warehouse instance, the first step is to have a well-defined schema of your data.
Data in Snowflake is organized around tables with a well-defined set of columns, with each one having a specific data type.
Snowflake supports a rich set of data types. It is worth mentioning that a number of semi-structured data types are also supported. It is possible to load data directly in JSON, Avro, ORC, Parquet, or XML format with Snowflake. Hierarchical data is treated as a first-class citizen, similar to what Google BigQuery offers.
There is also one notable common data type that Snowflake does not support. LOB or large object data type is not supported. Instead, you should use a BINARY or VARCHAR type. But these types are not that useful for data warehouse use cases.
A typical strategy for loading data from Mixpanel to Snowflake is to create a schema where you will map each API endpoint to a table.
Each key inside the Mixpanel API endpoint response should be mapped to a column of that table, and you should ensure the right conversion to a Snowflake data type.
Of course, you will have to ensure that as data types from the Mixpanel API might change, you will adapt your database tables accordingly. There’s no such thing as automatic data type casting.
After you have a complete and well-defined data model or schema for Snowflake, you can move forward and start loading your data into the database.
Load data from Mixpanel to Snowflake
Usually, data is loaded into Snowflake in a bulk way, using the COPY INTO command. In JSON format, files containing data are stored in a local file system or in Amazon S3 buckets. Then a COPY INTO command is invoked on the Snowflake instance, and data is copied into a data warehouse.
The files can be pushed into Snowflake using the PUT command into a staging environment before the COPY command is invoked.
Another alternative is to upload data directly into a service like Amazon S3, from where Snowflake can access data directly.
Finally, Snowflake offers a web interface as a data loading wizard where someone can visually set up and copy every data into the data warehouse. Just keep in mind that the functionality of this wizard is limited compared to the rest of the methods.
In contrast to other technologies like Redshift, Snowflake does not require a data schema to be packed together with the data that will be copied. Instead, the schema is part of the query that will copy all data into a data warehouse. This simplifies the data loading process and offers more flexibility on data type management.
Updating your Mixpanel data on Snowflake
As you will be generating more data on Mixpanel, you will need to update your older data on Snowflake. This includes new records together with updates to older records that, for any reason, have been updated on Mixpanel.
You ought to periodically check Mixpanel for new data and repeat the previously described process while updating your currently available data if needed. Updating an already existing row on a Snowflake table is achieved by creating UPDATE statements.
Snowflake has a great tutorial on the different ways of handling updates, especially using primary keys.
Another issue that you need to take care of is identifying and removing any duplicate records on your database. Either because Mixpanel does not have a mechanism to identify new and updated records or because of errors on your data pipelines, duplicate records might be introduced to your database.
In general, ensuring the quality of data that is inserted in your database is a big and difficult issue.
The best way to load data from Mixpanel to Snowflake
So far, we just scraped the surface of what you can do with Snowflake and how to load data into it. Things can get even more complicated if you want to integrate data coming from different sources.
Are you striving to achieve results right now?
Instead of writing, hosting, and maintaining a flexible data infrastructure, RudderStack can automatically handle everything for you.
RudderStack, with one click, integrates with sources or services, creates analytics-ready data, and syncs your Mixpanel to Snowflake right away. Don't want to go through the pain of direct integration? RudderStack’s Mixpanel to Snowflake integration makes it easy to send data from Mixpanel to Snowflake.