How to load data from Aftership to Google BigQuery
Access your data on Aftership
The first step in loading your Aftership data to any data warehouse solution is to access them and start extracting it.
Using the REST API that Aftership offers, you can programmatically interact with your account in order to gain access to your order tracking data. By doing so, you can:
- Get the list of all supported couriers.
- Retrieve tracking results
- Get tracking information of the last checkpoint of a tracking
- Gain access to contacts (sms or email) to be notified when the status of a tracking has changed.
You can also retrieve some basic aggregated metrics for any user-defined time period, such as the average score of all your surveys or of a specific trend or client.
In addition to the above, the things that you have to keep in mind when dealing with the Aftership API, are:
- Rate limits. In order to guarantee a high quality of service to all users of the API, Aftership applies certain rate limits. Currently, users are limited to 600 requests per minute per account.
- Authentication. You can authenticate to Aftership using a private API key that is linked to your account.
- Pagination. API endpoints that return a collection of items are always paginated.
Transform and prepare your Aftership data for Google BigQuery
After you have accessed data on Aftership, you will have to transform it based on two main factors,
- The limitations of the database that is going to be used
- The type of analysis that you plan to perform
Each system has specific limitations on the data types and data structures that it supports. If for example, you want to push data into BigQuery, then you can send nested data like JSON directly.
Also, you have to choose the right data types. Again, depending on the system that you will send data to and the data types that the API exposes to you, you will have to make the right choices. These choices are important because they can limit the expressivity of your queries and limit your analysts on what they can do directly out of the database.
Also, you have to consider that the reports you’ll get from Aftership are like CSV files in terms of their structure, and you need to somehow identify what and how to map to a table into your database.
Load data from Aftership to Google BigQuery
If you want to load any data from Aftership to BigQuery, you have to use one of the following supported data sources.
- Google Cloud Storage
- Sent data directly to BigQuery with a POST request
- Google Cloud Datastore Backup
- Streaming insert
- App Engine log files
- Cloud Storage logs
From the above list of sources, 5 and 6 are not applicable in our case.
For Google Cloud Storage, you first have to load some data into it. There are a few options on how to do this; for example, you can use the console directly as described here, and do not forget to follow the best practices.
Another option is to post data through the JSON API. APIs play an important role in both the extraction but also the loading of data into our data warehouse. In its simplest case, it’s just a matter of one HTTP POST request using a tool like CURL or Postman.
After you have loaded your data into Google Cloud Storage, you have to create a Load Job for BigQuery actually to load any data into it. This Job should point to the source data in Cloud Storage that have to be imported, and it happens by providing source URIs that point to the appropriate objects.
The best way to load data from Aftership to Google BigQuery
So far, we just scraped the surface of what you can do with BigQuery and how to load any data into it. Things can get even more complicated if you want to integrate data coming from different sources.
Are you striving to achieve results right now?
Instead of writing, hosting, and maintaining a flexible data infrastructure, use RudderStack to handle everything automatically for you.
RudderStack, with one click, integrates with sources or services, creates analytics-ready data, and syncs your Aftership to BigQuery right away.