How to load data from Hubspot to Snowflake

Extract your data from Hubspot

Hubspot APIs are following the REST architecture that can be accessed through HTTP. As a RESTful API, interacting with it can be achieved by using tools like CURL or Postman or by using HTTP clients for your favorite language or framework.

A few suggestions:

  • Apache HttpClient for Java
  • Spray-client for Scala
  • Hyper for Rust
  • Ruby rest-client
  • Python http-client

Responses of the API are all in JSON, including errors although HTTP codes are also returned to indicate errors. Hubspot maintains some official SDKs for their APIs, like the hapipy client for python, also unofficial clients can be found.

Hubspot API Authentication

Hubspot API allows two types of authentication. OAuth and API keys. Hubspot encourages the use of oAuth for any serious integration and suggests using basic authentication with API keys only for testing and rapid prototyping purposes.

Hubspot API rate limiting

HubSpot public endpoints are powered by the same underlying technology that powers the core HubSpot application. As a result, HubSpot engineering closely monitors usage of the public APIs to ensure a quality experience for users of every HubSpot application.

Below, you’ll find the limits by which a single integration (as identified by an access token) can consume HubSpot public APIs.

  1. Integrations cannot consume a HubSpot API at a rate greater than 10 requests/second.
  2. Polling of HubSpot APIs should occur at intervals of 5 minutes or more.
  3. Total requests to each HubSpot APIs should not exceed 10,000 in a 24 hour period.
  4. Failed requests to any HubSpot APIs may not exceed 5% of total syncs.
  5. All data passed to HubSpot must be properly encoded, and use application/json formatting.
  6. Integrations should use HubSpot’s OAuth protocol.
  7. Integrators must store time-to-live (TTL) data for OAuth access tokens as well as refresh tokens. Unauthorized (401) requests are not a valid indicator that a new access token must be retrieved.
  8. Integrators should use their own public and documented APIs when working with HubSpot APIs.
  9. We reserve the right to change or deprecate the APIs over time – we will provide developers ample notification in those cases.

Endpoints and available Resources

Hubspot API is actually a collection of a large number of APIs. Something that makes sense if we consider that Hubspot is actually more than one product and each one of these products is quite complex on its own. The APIs are the following:

  • Calendar API. Anything that has to do with calendars in Hubspot and their possible operations.
  • Companies APIs. When we talk about business and customers we talk about companies, so here is your API for interacting with companies.
  • Companies properties APIs. Companies are important for Hubspot and there’s a lot of functionality around them so there’s an auxiliary API just for working with properties of companies.
  • Contacts APIs. Contacts are the fundamental building block to HubSpot – they store lead-specific data that makes it possible to leverage much of the functionality in HubSpot, from marketing automation, to lead scoring to smart content.
  • Contact Lists APIs. API for managing the lists of your contacts.
  • Contact properties APIs. Similar to companies, this API allows you to interact with the properties of your contacts.
  • COS Blog API. Interact with Blogs throughs the Hubspot platform.
  • COS Blog Authors API. API for interacting with the authors of your blogs.
  • COS Blog Comments API. Exposes functionality about the comments of your APIs.
  • COS Blog Posts API. Anything concerning the posts of your Blog.
  • COS Blog Topics API. Manage the topics of your Blog.
  • COS Domains API. Manage your Domains through this API.
  • COS Files API. Operations for file management inside Hubspot.
  • COS Layouts API. Manage the layouts of your pages through this API.
  • COS Page Publishing API. Operations about publishing content through the Hubspot platform.
  • COS Sitemaps API. Operations concerning the management of sitemaps for the sites you create through the Hubspot platform.
  • COS Templates API. API for managing the templates of your sites.
  • COS URL Mappings API. Operations for URL mappings for the sites inside Hubspot.
  • Deals API. Anything that has to do with deals inside your CRM.
  • Deal Pipelines API. Manage the sales pipelines through this API.
  • Deals Properties API. Again, manage the properties of your deals.
  • Email API. Anything related to emailing from within Hubspot platform.
  • Email Events API. Track and interact with events that happen inside emails.
  • Engagements API. Anything related to customer engagement inside the hubspot platform.
  • Events API. Event handling for Hubspot.
  • Forms API. Manage custom forms that you create inside the platform.
  • Keywords API. Operations about keywords for SEO.
  • Owners API. Anything related to Owner.
  • Social Media API. API for interacting with Social media through the Hubspot platform.
  • Transactional Email API. The transactional email functionality of Hubspot.
  • Workflows API. Define and manage sales and marketing workflows.

From all the above endpoints we can pull data out of the platform, so it is easy to understand the richness of data we can get from a platform like Hubspot. Let’s assume, as an example, that we want to get all the Deals data. By executing a GET request like this GET /deals/v1/deal/recent/modified we can get all the recently modified deals. The parameters that we can pass to the call are the following:

  • count: for specifying the number of results per page of the response.
  • offset: for paginating through all available results.
  • since: a timestamp for defining from which exact time you would like to fetch data from.

As we said earlier, results from Hubspot API are always in JSON, so if we successfully execute the above query we’ll get the following results back:

JAVASCRIPT
{
   "results": [
       {
           "portalId": 62515,
           "dealId": 1030663,
           "isDeleted": false,
           "associations": {
               "associatedVids": [
                   27316
               ],
               "associatedCompanyIds": [
 
               ],
               "associatedDealIds": [
 
               ]
           },
           "properties": {
               "dealstage": {
                   "value": "closedwon",
                   "timestamp": 1417686612442,
                   "source": "API",
                   "sourceId": null,

The API offers you the opportunity to get very granular data about your accounting activities and use it for analytic and reporting purposes.

Hubspot Data Preparation for Snowflake

The first step, before you start ingesting your own data into a Snowflake data warehouse instance, is to have a well-defined schema of your data.

Data in Snowflake is organized around tables with a well-defined set of columns with each one having a specific data type.

Snowflake supports a rich set of data types. It is worth mentioning that a number of semi-structured data types are also supported. With Snowflake, is possible to load directly data in JSON, Avro, ORC, Parquet, or XML format. Hierarchical data is treated as a first-class citizen, similar to what Google BigQuery offers.

There is also one notable common data type that is not supported by Snowflake. LOB or large object data type is not supported, instead, you should use a BINARY or VARCHAR type instead. But these types are not that useful for data warehouse use cases.

A typical strategy for loading data from Hubspot to Snowflake is to create a schema where you will map each API endpoint to a table.

Each key inside a Hubspot API endpoint response should be mapped to a column of that table and you should ensure the right conversion to a Snowflake data type.

Of course, you will need to ensure that as data types from a Hubspot API might change, you will adapt any database tables accordingly, there’s no such thing as automatic data type casting.

After you have a complete and well-defined data model or schema for Snowflake, you can move forward and start loading every data into the database.

Load data from Hubspot to Snowflake

Usually, data is loaded into Snowflake in a bulk way, using the COPY INTO command. Files containing data, usually in JSON format, are stored in a local file system or in Amazon S3 buckets. Then a COPY INTO command is invoked on the Snowflake instance and data is copied into a data warehouse.

The files can be pushed into Snowflake using the PUT command, into a staging environment before the COPY command is invoked.

Another alternative is to upload every data directly into a service like Amazon S3 from where Snowflake can access data directly.

Finally, Snowflake offers a web interface as a data loading wizard where someone can visually set up and copy the data into the data warehouse. Just keep in mind, that the functionality of this wizard is limited compared to the rest of the methods.

Snowflake in contrast to other technologies like Redshift, does not require a data schema to be packed together with the data that will be copied. Instead, the schema is part of the query that will copy the data into a data warehouse. This simplifies the data loading process and offers more flexibility on data type management.

Updating your Hubspot data on Snowflake

As you will be generating more data on Hubspot, you will need to update your older data on Snowflake. This includes new records together with updates to older records that for any reason have been updated on Hubspot.

You will need to periodically check Hubspot for new data and repeat the process that has been described previously while updating your currently available data if needed. Updating an already existing row on a Snowflake table is achieved by creating UPDATE statements.

Snowflake has a great tutorial on the different ways of handling updates, especially using primary keys.

Another issue that you need to take care of is the identification and removal of any duplicate records on your own database. Either because Hubspot does not have a mechanism to identify new and updated records or because of errors on your preferred data pipelines, duplicate records might be introduced to your own database.

In general, ensuring the quality of data that is inserted in a database is a big and difficult issue.

The best way to load data from Hubspot to Snowflake

So far, we just scraped the surface of what you can do with Snowflake and how to load data into it. Things can get even more complicated if you want to integrate data coming from different sources.

Are you striving to achieve results right now?

Instead of writing, hosting, and maintaining a flexible data infrastructure use RudderStack that can handle everything automatically for you.

RudderStack, with one click, integrates with sources or services, creates analytics-ready data, and syncs your Hubspot to Snowflake right away.

Sign Up For Free And Start Sending Data
Test out our event stream, ELT, and reverse-ETL pipelines. Use our HTTP source to send data in less than 5 minutes, or install one of our 12 SDKs in your website or app.
Don't want to go through the pain of direct integration? RudderStack's HubSpot integration makes it easy to send data from HubSpot to Snowflake.