RudderStack Dagster Integration

Today, we are excited to announce the launch of the dagster-rudderstack Python library, which integrates RudderStack with Dagster. This integration allows teams to seamlessly orchestrate RudderStack Profiles and Reverse ETL jobs within their existing Dagster workflows. You can also define cross-system dependencies, ensuring your warehouse operations are synchronized and optimized beyond the boundaries of RudderStack.

Overview

As organizations integrate more data sources into their workflows, the processes of cleaning, transforming, and activating data become increasingly complex.

Orchestration tools like Dagster simplify these processes by enabling teams to schedule jobs, define dependencies, and maintain reliable, maintainable, and observable data pipelines. Its modular design and support for modern data practices empower teams to streamline workflows, reduce errors, and focus on delivering actionable insights.

Feature walkthrough

Workflow

1. Install the dagster-rudderstack Python package.
2. Use the RudderStack-specific operations to define Profiles runs and Reverse ETL syncs within jobs.
3. Define RudderStack-specific resources like connection details (access tokens, connection IDs, etc.)
4. Define a schedule.

See the RudderStack Dagster integration docs for more information on using this tool.

Questions? Contact us by email or on Slack