Streaming Events From Salesforce for Lead Enrichment With RudderStack’s Webhook Source

In this post, we’ll walk you through RudderStack’s Webhook source, and how we’re using it for automation internally. We’re streaming new lead created events from Salesforce through Rudderstack for lead enrichment then back to Salesforce and other downstream tools like Customer.io and Snowflake.

For those not familiar with Webhooks, a Webhook source in RudderStack is like hanging a digital piece of flypaper that captures anything you throw at it as long as it’s encoded as JSON. Webhooks do not request information, they only receive it via HTTP requests from your website, application, apache service, etc. So, they’re best suited to process notifications and forward them to your downstream destinations and warehouses.

How we enrich new leads created in salesforce with Clearbit data

Like a lot of companies, we use Salesforce as a CRM. In fact, our Salesforce integration is one of our most popular destinations, but most people don’t typically use it as a source for streaming event data. We also use Clearbit for lead enrichment and have published a user transformation in our git library that we routinely call prior to sending new leads into Salesforce.

But we ran into a problem when we didn’t have enough data attached to a new lead to trigger a meaningful response from Clearbit. We needed a way to trigger Salesforce to re-fire the Clearbit Enrichment Transformation once sufficient information was collected on the lead record – a perfect use case for our Webhook source. Yes, we could have opted to purchase the more expensive version of Clearbit that already integrates with Salesforce, but in addition to the cost savings of using the API version, with RudderStack firing the transformation, we can send that enriched data immediately to all other downstream tools like Customer.io, custom audiences, and Mixpanel.

For this particular tutorial, we will call the Clearbit Lead Enrichment API for all leads created in Salesforce. We will create a new process builder to fire in Salesforce once a new lead is created and saved. The process builder will call a generic Salesforce Apex Class (that we will write) to pass whatever fields we want from our lead record to our RudderStack Webhook source as a JSON payload. We will create a user transformation similar to the one in our git library, but we will tweak it to support the needs of our Salesforce Destination.

If you would like to build this solution or learn more about how to set up something similar, check out our Webhook Source Documentation in our user guide and sign up for a free trial account if you haven’t already.

Step 1: Create a new Webhook source in your instance of RudderStack

Create a new Source and select the Webhook option under Event Streams in the RudderStack User Interface.

After you give it a name, you can find the specific URL for your webhook on the settings tab:

The Dataplane URL can be found on the top of your main Connections page.

Step 2: Register the URL with Salesforce

Once you have the URL for your webhook, the next step is to add it to your list of Remote Sites in Salesforce. Without registering the URL as safe, Salesforce will block the outbound call.

From the Salesforce Setup menu, click Security Controls then Remote Site Settings. Enter the URL for your dataplane.

Note: You do not need to include the full URL of your webhook. Mark it as Active.

Step 3: Create an Apex Class

In our example, we will trigger the webhook from a process builder (you can also create a flow or trigger to do the same), and it will call a generic Apex Class to convert the data to a JSON payload and call the URL. We could do all of this in a single trigger, but this example creates a generic processing tool that can be called by any object via Process Builder where even the Webhook URL is passed in the Process Builder UI. This creates a “code it once” solution that, once deployed, can be used for multiple objects and RudderStack Webhook sources.

In the Developer Console, create a new Apex Class called SendToRudderStack.apxc

JAVA
/**
 *  Sends a Record To Rudderstack. Invocable from Process Builder.
 */

public with sharing class SendToRudderStack {
    public class Payload {
        @InvocableVariable(label='RudderStack Webhook URL')
        public String webhookURL;
        @InvocableVariable(label='Object API Name')
        public String objectName;
        @InvocableVariable(label='Field 1 API Name')
        public String field1Name;
        @InvocableVariable(label='Field 1 Value')
        public String field1Value;
        @InvocableVariable(label='Field 2 API Name')
        public String field2Name;
        @InvocableVariable(label='Field 2 Value')
        public String field2Value;
        @InvocableVariable(label='Field 3 API Name')
        public String field3Name;
        @InvocableVariable(label='Field 3 Value')
        public String field3Value;
        @InvocableVariable(label='Field 4 API Name')
        public String field4Name;
        @InvocableVariable(label='Field 4 Value')
        public String field4Value;
        @InvocableVariable(label='Field 5 API Name')
        public String field5Name;
        @InvocableVariable(label='Field 5 Value')
        public String field5Value;
        @InvocableVariable(label='Linked Record ID')
        public String recordId;
    }

    @InvocableMethod(label='Send To RudderStack Webhook')
    public static void sendToRudderStack(List<Payload> payloads) {
        Payload p = payloads[0];
        Map<String,String> record = new Map<String,String>();
        if(!String.isBlank(p.field1Name) && !String.isBlank(p.field1Value)) {
            record.put(p.field1Name, p.field1Value);
        }
        if(!String.isBlank(p.field2Name) && !String.isBlank(p.field2Value)) {
            record.put(p.field2Name, p.field2Value);
        }
        if(!String.isBlank(p.field3Name) && !String.isBlank(p.field3Value)) {
            record.put(p.field3Name, p.field3Value);
        }
        if(!String.isBlank(p.field4Name) && !String.isBlank(p.field4Value)) {
            record.put(p.field4Name, p.field4Value);
        }
        if(!String.isBlank(p.field5Name) && !String.isBlank(p.field5Value)) {
            record.put(p.field5Name, p.field5Value);
        }
        String body = JSON.serialize(record);
        System.enqueueJob(new QueueableRestApiCall(p.webhookURL, 'POST', body));

    }
     
    public class QueueableRestApiCall implements System.Queueable, Database.AllowsCallouts {
        private final String endpoint;
        private final String method;
        private final String body;
         
        public QueueableRestApiCall(String endpoint, String method, String body) {
            this.endpoint = endpoint;
            this.method = method;
            this.body = body;
        }
         
        public void execute(System.QueueableContext ctx) {
            HttpRequest req = new HttpRequest();
            req.setEndpoint(endpoint);
            req.setMethod(method);
            req.setHeader('Content-Type', 'application/json');
            req.setBody(body);
            Http http = new Http();
            if(!Test.isRunningTest()) {
                HttpResponse res = http.send(req);
            }
        }
    } 
}

You will also need a test class called SendToRudderStackText.apxc

JAVA
/**
 * Test class for SendToRudderStack
 */
@isTest
public class SendToRudderStackTest {

    @isTest
    static void sendToRudderStack () {
        
        SendToRudderStack.Payload payload = new SendToRudderStack.Payload();
        payload.webhookURL = 'https://www.example.test/12345';
        payload.objectName = 'Account';
        payload.field1Name = 'Field 1';
        payload.field1Value = 'Value 1';
        payload.field2Name = 'Field 2';
        payload.field2Value = 'Value 2';
        payload.field3Name = 'Field 3';
        payload.field3Value = 'Value 3';
        payload.field4Name = 'Field 4';
        payload.field4Value = 'Value 4';
        payload.field5Name = 'Field 5';
        payload.field5Value = 'Value 5';
        
        Test.startTest();

        SendToRudderStack.sendToRudderStack(new List<SendToRudderStack.Payload>{payload});
        
        Test.stopTest();
    }

}

The Apex class creates a container that has five fields of data, the record ID and the URL of the webhook destination. These are generic and allow us to determine the object and the fields to be passed (and their actual labels) in the Process Builder itself. (See Next Step)

NOTE: Apex Classes can only be created in your sandbox environment. If you are not familiar with deploying packages from your sandbox to production, please check out this Salesforce support thread on the topic.

Step 4: Create the Process Builder

From the Salesforce Setup menu, type Process Builder in the Quick Find menu and click New to create a new one. In our example we want to send details of all newly created lead records back to RudderStack so we selected the Lead Object and chose to start the process when a record changes.

In the next step we define the criteria for which this action should fire. In our example we want all leads to fire once they’ve been created so we will not apply any filters and check the “No criteria - just execute the actions!” option.

If we were firing the Clearbit API at some other point in the process, we might look to evaluate whether any of the clearbit fields were actually populated (or create a checkbox that would be set to true if the API was already called) and then apply logic to not fire the webhook in that event.

In the Immediate Actions menu, we want to select the “Call APEX” option and then find the Send Lead To RudderStack Webhook option. This name mirrors what we created in our Apex Class. If you don’t see this option, verify that your deployment of the inbound package was successful.

We then map the fields and field names as well as the Webhook URL for our RudderStack Source. As we noted above, the Apex class was designed to take any object or data passed through it. For this reason, we pass not only the Record ID, but a static string value that this object is a Lead. We also pass the URL for the webhook source, so, in the event we need to set up different sources in Rudderstack, we can re-use the same Apex class.

Save and Activate the Process Builder.

Step 5: Create the User Transformation to call Clearbit

In this example, we will be creating a new user transformation that will be applied to a new Salesforce Destination.

Note: In a more advanced setup, we could apply this transformation to an existing Salesforce destination and filter the transformation to only fire when the metadata tells us this event came from our particular webhook source. If there was already a user transformation applied to that destination, this could also be built as a library and only called if the metadata indicated the same event source.

User Transformation:

JAVA
export async function transformEvent(event) {

        event.traits = {
            email:event.properties['Email'],
            lead_id:event.properties['Lead_ID'],
            company:event.properties['Company'],
            firstName:event.properties['FirstName'],
            lastName:event.properties['LastName']
            
        }
        event.context = {
            externalId : [{
            id:event.properties['Lead_ID'],
            type:'Salesforce-Lead'
            }]
        }

        // Only send identify() calls and leads with email addresses
        if (event.traits.email) {
            const res =	await fetch('https://person-stream.clearbit.com/v2/combined/find?email='+event.traits.email, {headers: {'Authorization': 'Bearer sk_your_token_id'}});

            const nameProperties = res.person ? { 'Clearbit_Person': res.person.name.fullName } : {}

            const companyProperties = res.company
                ? {
                    Clearbit_company_name: res.company.name,
                    Clearbit_company_category_industry: res.company.category.industry,
                    Clearbit_company_category_industryGrou: res.company.category.industryGroup,
                    Clearbit_company_category_sector: res.company.category.sector,
                    Clearbit_company_category_subIndustry: res.company.category.subIndustry,
                    Clearbit_company_domain: res.company.domain,
                    Clearbit_company_foundedYear: res.company.foundedYear,
                    Clearbit_company_geo_city: res.company.geo.city,
                    Clearbit_company_geo_country: res.company.geo.country,
                    Clearbit_company_location: res.company.location,
                    Clearbit_company_metrics_employees: res.company.metrics.employees
                    }
                : {}
            
            const rawTraits = event.traits;


            const whiteListTraits = [
              'utm_campaign',
              'utm_content',
              'utm_content',
              'utm_medium',
              'utm_source',
              'utm_term',
              'raid',
              'email',
              'company',
              'conversion_page',
              'title',
              'conversion_page',
              'multiple_form_fills',
              'number_form_fills',
              'createddate'
            ];

            const filteredTraits = whiteListTraits.reduce((obj, field) => ({...obj, [field]: rawTraits[field]}), {})


            const newTraits = Object.assign(filteredTraits, nameProperties, companyProperties)

            event.traits = newTraits
            event.integrations =  {Salesforce: true };
            event.type = 'identify';
            delete event.properties;     
            delete event.event;     
            
            
        }

    return event;
}

Step 6: Create a new Salesforce Destination

Create a new Salesforce destination and connect it to the webhook. Add your newly created Clearbit User Transformation and now we’re ready to go.

Step 7: Create a lead - fire the Webhook!

As soon as a lead is created in Salesforce, the process builder is fired and sends a message back to RudderStack. We can see the inbound event in the event viewer for the Webhook Source. For our test lead in Salesforce, we only had First Name, Last Name and Email address populated so those were the only properties sent.

The payload is then routed through the user transformation which calls the clearbit api. We can see the resulting JSON payload in the Salesforce Destination viewer as well as the resulting before and after screens from the lead in Salesforce.

JSON
{
    "body": {
        "FORM": {},
        "JSON": {
            "Clearbit_company_category_industryGrou__c": "Software & Services",
            "Clearbit_company_category_industry__c": "Internet Software & Services",
            "Clearbit_company_category_sector__c": "Information Technology",
            "Clearbit_company_category_subIndustry__c": "Internet Software & Services",
            "Clearbit_company_domain__c": "rudderstack.com",
            "Clearbit_company_foundedYear__c": 2019,
            "Clearbit_company_geo_city__c": "San Francisco",
            "Clearbit_company_geo_country__c": "United States",
            "Clearbit_company_location__c": "96 S Park St, San Francisco, CA 94107, USA",
            "Clearbit_company_metrics_employees__c": 45,
            "Clearbit_company_name__c": "RudderStack",
            "Company": "n/a",
            "Email": "benji@rudderstack.com",
            "LastName": "n/a"
        },
        "XML": {}
    },
    "endpoint": "https://na173.salesforce.com/services/data/v50.0/sobjects/Lead/00Q5x00001tf0EgEAI?_HttpMethod=PATCH",
    "files": {},
    "headers": {
        "Authorization": "Bearer XXXXXXXX",
        "Content-Type": "application/json"
    },
    "method": "POST",
    "params": {},
    "type": "REST",
    "version": "1"
}

Empty Clearbit fields:

After Cleabit Update:

Now that you have a generic event handler, there are a million use cases for why you might want to stream events in real-time instead of a more traditional cloud-streaming option. At the top of the list are opt out notifications that need to be immediately broadcast to all of your other tools. You may also want to send closed opportunities to Slack or Microsoft Teams channels announcing a new deal. Please join us on Slack and let us know what Webhook Source uses you come up with!

September 23, 2021