Determining Sentiment Analysis With RudderStack User Transformations

Most companies are collecting feedback from their users and customers, but judging sentiment from this feedback isn’t easy. Whether you’re using a Net Promoter Score (NPS) or just building your own surveys, it’s difficult to gauge how your customers feel when only a small percentage of those solicited respond. Even then, outliers can skew the results in one direction or another.

So, at RudderStack, we built a system to address these challenges and give us a clear picture of customer sentiment. We use NPS and apply Google’s Sentiment Analysis API - which can be called using a RudderStack User Transformation - to score the feedback we collect via Halp, Slack Support, and contact forms. This sentiment score can then be passed to the warehouse for aggregation and reporting. More importantly, it can be used to triage and escalate support tickets and form submissions.

In this tutorial project, you’ll learn how you can replicate our sentiment analysis system within your own stack.

Project components

RudderStack Application

Webhook Source
Google Sheets Destination
User Transformation

Google Project

Natural Language API enabled (Requires billing account to be linked, but new users get a $300 credit)
Google Sheets API enabled
Creating & Using API Credentials

Postman Project

Sample payload included for you in this doc for testing purposes

How to build your own sentiment analysis system with RudderStack

Step 1: Sign Up For RudderStack

If you don’t already have a RudderStack account, sign up here for free. For this project we’ll create a simple Webhook Source and send some example payloads via Postman. These will simulate receiving messages from webforms, zendesk tickets, etc.

If you are brand new to RudderStack, check out our Guide to Sending Data In 5 Minutes to help you get oriented on the app and/or check out our docs for creating a Webhook Source in RudderStack.

Step 2: Connect your Webhook Source

Connect your Webhook Source to your data warehouse or, if you are just testing out the product, you can send your results to a Google Sheet. Don’t forget to enable the Google Sheets API after setting up your service account.

We will create a User Transformation for our Google Sheets destination which you can create during the setup or skip it and add it after you set up the google project.

Step 3: Create a Google Cloud project

Create a Google Cloud project and enable the Natural Language API following these instructions. The key steps in setting up your project are 1) enabling the Natural Language API within your project and creating the necessary credential within the APIs & Services menu within the cloud console. (Note: API keys are not in the IAM menu, but under the API’s & Services option)

After you create your API key, paste it into the User Transformation below.

Step 4: Create a User Transformation

Create a User Transformation to call the Google API by copying the sample code below.

Note: The Feedback property that gets mapped to the Content field in the submission. If you change your payload in Postman or are using a different event source, you will need to update this mapping. You can test the API by copying your source event payload in the Webhook Live Events viewer and pasting it in the Identify Window and clicking Run Test.

JAVASCRIPT
export async function transformEvent(event) {
      if (event.properties && event.properties.feedback){
        var payload = {};
        payload["document"] = {}
        payload["document"]["type"] = 'PLAIN_TEXT'
        payload["document"]["content"] = event.properties.feedback
        payload["encodingType"] = "UTF8"

        const res = await fetch("https://language.googleapis.com/v1/documents:analyzeSentiment?key=YOUR_GOOGLE_CLOUD_API_KEY", {
            method: "POST",
            body: JSON.stringify(payload)
        });
        
        if (res.documentSentiment && res.documentSentiment.magnitude){
            event.properties.sentiment_value = res.documentSentiment.magnitude
            if (res.documentSentiment.score){
                event.properties.sentiment_score = res.documentSentiment.score
            }
        }
        event.response = JSON.stringify(res);
    }   
    return event;
}

Test Payload From Webhook Source Live Event Viewer:

JAVASCRIPT
{
    "type": "track",
    "event": "Sample Feedback Event",
    "properties": {
        "name": "Benji Walvoord",
        "email": "benji@rudderstack.com",
        "feedback": "Rudderstack is a great product.  I highly recommend it.  It is my favorite SAAS product of all time."
    },
    "integrations": {
        "All": true
    }
}

Step 5: Send sample payload from Postman:

Postman is a great tool for testing RudderStack, especially when testing Webhook or SDK sources. Within postman create a new HTTP request with the following:

Method: Post
URL: <YOUR DATA PLANE URL:>/v1/track
Authorization: Basic username: webhook_source_write_key password: (leave blank)
Body: Raw / JSON

Sample Body:

TEXT
{
 "userId": "benji@rudderstack.com",
 "event": "Sample Feedback Event",
   "properties": {
           "email": "benji@rudderstack.com",
           "feedback": "Rudderstack is a great product.  Their customer success team is great.  I would recommend this product to anyone.",
           "name": "Benji Walvoord"
   },
   "integrations": {
       "All": true
   }
}

Once we send the event, we can view it in the Live Event Viewer for both our Source and Destination. Notice how in the Google Sheets destination view, the columns are mapped based on the field mapping from the Google Sheet destination setup.

Webhook Source Live Event Viewer

Destination Live Event Viewer

Step 6: Send a negative comment

Send a negative comment using the same payload but with different feedback: “Rudderstack is just ok. I can see what they are trying to do but their documentation leaves a little lacking and I would rather talk to someone on the phone than have to rely on slack for support. If you could improve on these two fronts I might use the product more in the future.”

Step 7: Review results in Google Sheet:

Our initial positive feedback returned a Sentiment Score of 0.9 with a Sentiment Value of 2.8 while our second test returned a negative score of -0.1 with a Value of 0.8.

Step 8: How to interpret the results

Interpreting the results generally boils down to whether the sentiment score was positive or negative and the weight of the sentiment value. For a detailed review of interpreting the results, check out the Interpreting Results section of Google’s own documentation. So, let’s take our same positive feedback example and test it with Google’s Natural Language API Demo. This gives us the following visual to explain how we attained our Score and Value:

The document’s sentiment score indicates the direction or overall emotion of the document while the magnitude indicates how much emotional content was provided. In general, the magnitude is generally proportional to the length of the document or message.

As you will note, we are only passing the entire document results in this example but if we go back to the User Transformation and run the test on our sample payload, we will see the same sentence by sentence analysis as returned above:

JAVASCRIPT
[
  {
    "type": "track",
    "event": "Sample Feedback Event",
    "userId": "benji@rudderstack.com",
    "rudderId": "620954bc-fe71-4cb8-b8a4-4149d19011b2",
    "messageId": "cf92099f-9824-4230-9d09-dfec1d68c6e4",
    "properties": {
      "name": "Benji Walvoord",
      "email": "benji@rudderstack.com",
      "feedback": "Rudderstack is a great product.  Their customer success team is great.  I would recommend this product to anyone.",
      "sentiment_value": 2.8,
      "sentiment_score": 0.9
    },
    "integrations": {
      "All": true
    },
    "response": "{\"documentSentiment\":{\"magnitude\":2.8,\"score\":0.9},\"language\":\"en\",\"sentences\":[{\"text\":{\"content\":\"Rudderstack is a great product.\",\"beginOffset\":0},\"sentiment\":{\"magnitude\":0.9,\"score\":0.9}},{\"text\":{\"content\":\"Their customer success team is great.\",\"beginOffset\":33},\"sentiment\":{\"magnitude\":0.9,\"score\":0.9}},{\"text\":{\"content\":\"I would recommend this product to anyone.\",\"beginOffset\":72},\"sentiment\":{\"magnitude\":0.9,\"score\":0.9}}]}"
  }
]

If we break down the response from Google we can see how it aligns with the image above:

TEXT
"{\"documentSentiment\":    
{\"magnitude\":2.8,\"score\":0.9},
\"language\":\"en\",
\"sentences\":[
{\"text\":{\"content\":\"Rudderstack is a great product.\",
\"beginOffset\":0},
\"sentiment\":{\"magnitude\":0.9,\"score\":0.9}},
{\"text\":{\"content\":\"Their customer success team is great.\",
\"beginOffset\":33},
\"sentiment\":{\"magnitude\":0.9,\"score\":0.9}},
{\"text\":{\"content\":\"I would recommend this product to anyone.\",
\"beginOffset\":72},
\"sentiment\":{\"magnitude\":0.9,\"score\":0.9}}
]
}"

Conclusion

We hope in this tutorial you’ve learned just how easy it is to use RudderStack and how you can leverage RudderStack User Transformations to enhance your data. Once you have your project set up and running, take a look at the other API options available, including Entity Sentiment, Classifying Syntax, and Analyzing Syntax. If you’d like to learn how RudderStack can be used to train and test more advanced, industry-specific modeling using Google’s AutoML service, contact our team today.

October 26, 2021