Amazon S3 Reverse ETL Source

Amazon S3 is a cloud-based object storage service that lets businesses securely store their data at scale.

RudderStack supports S3 as a data source from which you can ingest data and route it to your desired downstream destinations.

Set up S3 source in RudderStack

1. Log in to your RudderStack dashboard.
2. On the Connections page, click Add source.
3. Under Sources, click Reverse ETL and select Snowflake.

Connection credentials

Configure the following settings to authenticate RudderStack to access your S3 account:

• Connection Mode: RudderStack provides the following options to connect to S3:
  • Cross-Account Role (recommended): This option lets you connect to S3 through an IAM access role. To do so, you need to first create an IAM role for RudderStack with the required permissions to access your S3 account. See Creating the RudderStack IAM Role for S3 for the detailed steps.
  • Access Key: This option lets you connect to S3 using your AWS access key ID and secret access key.

Note the following:

• It is highly recommended to use the Cross-Account Role method for connecting to S3 as the Access Key method will be deprecated soon.
• See S3 permissions for the minimum permissions that need to be attached to IAM role or the access keys (depending on your connection method).

• Account Name: Specify a name that will be used to identify the connection account.
• Role ARN: If you select the Cross-Account Role (recommended) connection mode, specify the ARN after creating the RudderStack IAM role.
• AWS Access Key ID: If you select the Access Key connection mode for authenticating RudderStack, specify your AWS access key ID. For more information on obtaining your access key ID and secret access key, refer to the FAQ section below.
• AWS Secret Access Key: Enter the corresponding secret access key.

Specify name, bucket, and prefix

• Source name: Assign a name to uniquely identify the source in the RudderStack dashboard.
• S3 Bucket Name: Enter the S3 bucket name.
• Prefix: Prefix refers to the path within your S3 bucket from where RudderStack imports the data. For example, if Prefix is set to RUDDER, then RudderStack imports the data stored in the location <your_s3_bucket>/RUDDER.

Your S3 bucket (with the prefix, if specified above) should only consist of Apache Parquet files as RudderStack can extract only the Parquet files. Also, the first row of the Parquet file should not have a null value (empty strings are allowed) for any column. It helps RudderStack to determine the correct schema of the file.

Review and complete setup

To make any changes to the warehouse credentials or source configuration, click the edit icon present next to those sections.

Review your configuration and click Create source to complete the setup.

S3 permissions

The minimum S3 permissions that need to be attached to IAM role or the access keys (depending on your connection method) are listed below:

"Action": [
  "s3:GetObject",
  "s3:ListBucket"
],

Create RudderStack IAM role

Follow the steps in this section to create a RudderStack IAM role and obtain the role ARN.

Create policy

To create a managed policy defining the permissions for the RudderStack IAM role, follow these steps:

1. Sign in to your AWS Management Console and open the IAM console.
2. In the left navigation pane, click Policies followed by Create policy.
3. In the JSON tab, paste the following policy:

{
  "Version": "2012-10-17",
  "Statement": [{
      "Effect": "Allow",
      "Action": "s3:ListAllMyBuckets",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": "*"
    }
  ]
}

4. Click Review policy. On the Review page, enter read-write-app-bucket.

Create IAM role

1. In the left navigation pane, click Roles and go to Create role.
2. Under Trusted entity type, select AWS account:

3. Select Another AWS account and under Account ID, enter 422074288268, the account ID associated with RudderStack.
4. Under Options check Require external ID and enter your workspace ID as the External ID.

5. Review all settings carefully and click Next to proceed.
6. In the Permissions window, select the check box next to the policy you created in the Creating the policy section above.
7. Review all settings carefully and click Next to proceed.
8. Enter a unique name for your role. Note that this name is case-insensitive. For example, you cannot create a role named RUDDERSTACK if rudderstack already exists.

You cannot edit the name of the role after it has been created.

9. Optional: Enter the description for this role.
10. Click Create role to complete the setup.
11. Finally, copy the ARN of this newly created role and paste it in the Role ARN field in the dashboard settings.

See AWS IAM tutorial for more information on delegating access across AWS account using IAM roles.

Update source configuration and settings

Go to the Configuration tab of your S3 source to update the configuration settings. Here, you can update the S3 bucket name and prefix.

Go to the Settings tab to:

• Get your source ID.
• Change your warehouse credentials.
• Set up custom alerts for your Reverse ETL source.
• Delete the source permanently.

You cannot delete a source that is connected to any destination.

Troubleshooting

Failing syncs with large row groups

Note that the Reverse ETL syncs will fail if your files contain large row groups with sizes more than 512 MB. This is because S3 cannot process Parquet files with row groups larger than 512 MB.

Make sure that:

• The maximum record length in the input or result is 1 MB.
• The maximum uncompressed row group size is 512 MB.

See S3 documentation for more information on these limits.

FAQ

Where can I obtain the AWS Access Key ID and the AWS Secret Access Key?

1. Sign in to your AWS Management Console as the root user.
2. From the upper right corner, click your account and go to Security Credentials. You can find your access key ID listed here. You can also create a new access key by clicking the Create access key button:

See the AWS documentation for more information on these credentials.

See S3 permissions for more information on the actions must be attached to your access keys required for setting up the S3 source.

Questions? Contact us by email or on Slack