As Salesforce moves from the standard Rollup-by-Lookup (RBL) to the enhanced framework used by the Data Processing Engine (DPE), it’s imperative that developers and admins understand how to hit the ground running with this powerful ETL tool from Salesforce.
I’ve learned a lot during the multiple Salesforce Financial Services Cloud (FSC) implementations I’ve worked on. Here’s a few of the tips, tricks, and lessons I have learned along the way…
What Is the Data Processing Engine (DPE)?
Per Salesforce, the Data Processing Engine is defined as:
“Enhanced Rollup-by-Lookup (RBL) framework that uses [the] superior processing power of Tableau CRM for faster calculation of RBL rules.”
In reality, it’s an optimized way to transform and update records in Salesforce that is also being utilized as a framework for the RBLs. The DPE is a super enhanced ETL tool that moonlights as a Rollup-by-Lookup framework.
RBL vs. DPE
Rollup-by-Lookup (RBL) is a feature of Salesforce Financial Services Cloud (FSC) that allows you to aggregate client financial data and rollup to the individual or group level using configuration rules.
Examples of where RBLs are utilized in your Salesforce Org:
- “RBL… Give me the TOTAL balance of all Loan accounts for each Client”
- “RBL… Give me a SUM of all open Deposit accounts of a Household/Group”
- “RBL… Give me a COUNT of all closed accounts of just the Primary Owner”
Rollup-by-Lookup
RBL is a rule based calculations only available for the following FSC Objects:
- Assets & Liabilities
- Claims
- Financial Accounts
- Financial Account Roles
- Insurance Policies
- Revenues
Data Processing Engine
DPE is a high powered ETL framework:
- Multiple data sources
- No Object limits
- Related Object Lookups
- Datasets from Analytics
- Robust ETL Toolkit
- Grouping & Aggregation
- Advanced logic with formulas
- Joins, Appends, and Filtering
At its core, the DPE is a high-powered, no code ETL tool with calculation capabilities (that you didn’t know you needed), available in the Salesforce environment.
But just how fast is it? I pulled data from multiple objects, transformed our raw data, and loaded the calculated data into 800K+ records in less than 15 minutes. That’s 4-5 hours for an RBL batch job. That’s lightning fast.
Tips, Tricks, and Lessons Learned
Yes, it can do our rollups, but the DPE does so much more, all without leaving your Salesforce instance. You can create a DPE that mitigates data exposure risks by updating all records while keeping your data enclosed in your Salesforce instance.
For those already building with the DPE, or those just about to jump in, I’ve compiled a short list of the key lessons learned, tips, tricks, and insights I’ve accumulated along the way. These will help you take full advantage of this powerhouse ETL tool you have under your Salesforce hood.
Troubleshoot Smarter
The DPE actually creates a temporary data recipe that runs behind the scenes in a batch job. You can leverage this for assistance as needed.
While the DPE is running, you can:
- See the real-time status of your recipe batch job nodes from within Monitor Workflow Services.
- Access auto-generated flow-like visual presentation of your recipe from the Data Manager Recipes tab (the DPE user interface lacks this functionality).
When the DPE is complete, you can:
- Use reference IDs of the DPE batch job from the Task tab of the DPE monitoring page.
- Utilize a REST utility such as Workbench to query for finer granularity of failed results.
Building it Simple
DPE builds follow the same pattern as any other recipe. To help explain this, we’re going to use the recipe for making peanut butter and jelly sandwiches as an example:
- Think about and visualize what you want to make (a delicious PB&J).
- Select the right ingredients at the start (peanut butter, jelly, bread).
- Combine the ingredients in the correct order and with the right tools (butter knife, plate, grandma’s famous PB&J recipe).
- Deliver the final results to the end location (eat to satisfy the craving).
Likewise, the process of creating a DPE requires you to:
- Brainstorm and Visually map out your end goal (Grouped Sums of Client Data).
- Get the right ingredients at the start (Object and Field Data).
- Combine the ingredients in the correct order and with the right tools (Transform, Join, or Aggregate Data nodes).
- Deliver the final results to the correct records (Writeback Node to Salesforce Records).
Top tip: DPE should only contain the data you need. You don’t need mayo for a PB&J, and likewise, only pull the fields you need for your DPE.
User Stories are Orders on the DPE Menu
A user story is a bite-sized nugget of information describing a feature from the user’s perspective that informs and articulates the newly requested capability.
They are generally formatted: As a < type of user > I want < some action > so that < benefit / value add >. For example: As a chef, I want real-time visibility of my PB&J supplies and client data, so that I can track progress and improve delivery where needed.
You can utilize the ‘what’ of user stories to help clarify the ‘how or solution’ of your DPE. Once gathered, you can pivot and start to view those stories through the lens of the organization’s data model.
In this example, you can see that you will need the following data sources:
- Financial Account
- Account
- Financial Account Role
From our Source Data, the following fields will be needed to build our DPE:
- Balance, Type, ID, and Record Type from Financial Account
- Record Type, Household ID from Person Account
- Role Field from Financial Account Role
Clarifying questions further inform our solution build, which will determine additional nodes in our build.
Which data sources contain our raw source data?
- Financial Account Records – Balance Field
What calculations need to occur?
- Sum of Balance Field from Financial Account
- Count of Record ID from Account
What is the final destination for my calculations?
How is our calculated data being filtered? How is our target object grouped?
- Financial Account Role – Role Field
- Financial Account – Record Types
- Financial Account – Type Field
- Account – Record ID
How are my data sources related?
- Financial Account Records connecting to Financial Account Role
- Financial Account Role connecting to Account
- Financial Account connecting to Account
Visualize the DPE Before You Build It
Grab a whiteboard, pen and paper, Lucidchart, or any medium that tickles your fancy to visualize the path. Be specific and deliberate about visually developing a plan for getting from point A to point B, before ‘clicks’ are even considered.
Aggregate Node Tips
For each “Group By” field you list out in our Aggregate node, additional record details are added, expanding the granularity of aggregated data.
Let’s reuse our PB&J example, this time for a PB&J delivery service.
You need a count of Customers by Peanut Butter Type and by Jelly Type, then to log this information into each customer’s individual record for tracking and delivery purposes.
If you only selected “Jelly Type” (Strawberry, Grape, Raspberry) for the “Group By” field, you would get an output table of sandwich count by “Jelly Type”, with no other granularity.
| Jelly Type | Count of Jelly Type |
|---|---|
| Grape | 3 |
| Strawberry | 3 |
| Raspberry | 1 |
By adding additional “Group By” fields, you augment the aggregate data with finer details, returning useful raw data.
| uniqueId | Jelly Type | Butter Type | Count of uniqueId | Count of Jelly Type | Count of Butter Type |
|---|---|---|---|---|---|
| pbj_0001 | Grape | Almond | 1 | 1 | 1 |
| pbj_0002 | Grape | Peanut | 1 | 1 | 1 |
| pbj_0003 | Strawberry | Almond | 1 | 1 | 1 |
| pbj_0004 | Raspberry | Sunflower | 1 | 1 | 1 |
| pbj_0005 | Strawberry | Almond | 1 | 1 | 1 |
| pbj_0006 | Strawberry | Hazelnut | 1 | 1 | 1 |
| pbj_0007 | Grape | Peanut | 1 | 1 | 1 |
Top tip: Always keep the Record ID of your Writeback node object as a “Group By” field in aggregation nodes to maintain record level detail granularity. After all calculations are complete, add a final aggregate node before your writeback node where the only “Group By” field is your Writeback Object record ID field.
Final Thoughts
While this is just the tip of the DPE iceberg, I hope that you have greater insight into one of the most powerful data tools now at your disposal, built into your Salesforce org. You can transform and load your Salesforce data, roll-up aggregate data, and expand your capabilities to provide Customer 360 solutions to your team at lightning speeds, all while mitigating data risks by keeping your data internal to your Salesforce instance.
That’s what I call a recipe for success.
