(Part 3 of our series)
As we discussed in previous posts in this series, creating a centralized data enrichment service gathers data from all of your enrichment providers whenever any of your applications have an enrichment request as opposed to calling enrichment services in a distributed way across your different applications. To recap, the benefits of a service like this are:

• Reduce service volume
• Potentially lower service costs
• The ability to perform data quality and gap analysis
• Add or remove services without touching your applications

But who says that requests for enrichment can only contain an email? What could we do if we start sending data? Let’s explore a use case that enables our Centralized Data Enrichment service to accept users’ communication and privacy preference information.

The Preference Management Problem

GDPR, CASL, CCPA, and other privacy regulations require organizations to ask for and acquire explicit permission from users before they are marketable. In response, organizations ask users permission-centric questions throughout the various points of the customer journey – e.g. May we email you? May we text you? But that alone is not enough. Organizations must also have explicit permission from customers who enter the database in other ways, such as through a customer service experience or an interaction with sales  This creates a new set of challenges to ensure an organization is, both, capturing appropriate privacy permissions as well as synchronizing them across the MOPs and RevOps technology stack.

Disparate databases in larger organizations can cause challenges when creating efficient data capture for compliance and have resorted to creating new fields in each of their applications with application-specific preferences. Then they create point-to-point integrations to share that data across applications creating a problem similar to that of decentralized enrichment where preference information spans across the enterprise with no single source of truth at any given time.

Further, when contacts provide conflicting preferences through different capture points, there is no way to reconcile them to determine survivorship – that is, which set of preferences live on and which are overwritten.

Finally, decentralized preference management presents legal risks since different parts of the business may unknowingly violate a contact’s privacy preferences. Legal challenges brought by users when privacy preferences are not honored are also more difficult to litigate without a central change log that captures when modifications were made and by whom.

Organizations need ways to:

• Create a master data set of contact privacy and communication preferences
• Make the most up-to-date contact preferences available to every application in as close to real-time as possible but within 24 hours
• Ensure traceability of privacy and preference changes (as required by law)

And while off-the-shelf preference management solutions are available starting at just $10k per year, organizations need to remember to account for the soft costs of internal time and resources it takes to integrate all required applications and assess the difficulty in switching providers if needed in the future.

Centralized Enrichment for Preference Management

The Centralized Data Enrichment solution we’ve previously discussed creates a single place where all enrichment services are called from and to which data is stored for later use. This same model can be used to send new preferences to a central service. The processing steps would be as follows:

1. Preference changes within each application would trigger calls to a central API. This applies to BOTH department-specific preferences (e.g. support or sales) as well as global preferences (e.g. unsolicited communications).
2. Each call would contain the users’ email address, the source application, the triggering event short description, and 2 permission objects:
   1. Department-specific preferences – These apply to specific business services such as support, sales, customer care, etc.
   2. Global preferences – These apply to unsolicited communications including permission to call, text, email, and direct mail for marketing purposes whether the communication is coming from the marketing team or not.
3. The contents of the incoming request are written to a logging table for traceability.
4. The department-specific object overwrites or merges with the existing department-specific preferences and a “updatedAt” timestamp is added.
5. The global preference object merges with the existing global preference object and receives a timestamp as well.
6. The new preferences are then “Published” out to an Event Bus. Applications that must always receive the latest preferences whenever they are available are “Subscribed” to the event bus. In this way, they will always receive any changes in a timely fashion.

iPaaS platforms such as Workato can be used to build this flow as an API with managed authentication. Workato also has an event bus with pub/sub features that allow you to publish permission change events to topics. Other applications throughout the enterprise can then subscribe to this topic to receive the changes at pre-determined synch intervals. That said, there are a few things you should consider when building the flow above.

First, if you DO NOT need near-real-time preference synch, rather than using an event bus you can create a nightly job that enables each application to import any updated preferences from your central database. This will reduce network chatter as applications will always pull the latest preferences one time each day regardless of how many times a contact’s preferences may have changed.

If you DO need real-time synch, you can use the event bus as discussed in step six or you can use the event bus to capture all changes right from the start. Applications could subscribe directly to these changes and in parallel you can subscribe your database as a recipient. In this way, you’re setting up more of a parallel flow where the database and your applications are receiving the latest updates at the same time. The key here is to log each change and to store the latest record somewhere. The drawback of architecting the solution in this way is that any merging logic you have will need to be owned by each application rather than being managed centrally.

Next, give particular thought to infinite loops. You’ll want to make sure that when applications receive any updated preferences from your central service, this update doesn’t trigger them to send an update to the API again. Updates that come from your central preference management service should be watermarked to enable you to filter them out of triggering an update to the central database.

Finally, there are clear legal implications to any solution you implement for regulatory compliance. One of the benefits of out-of-the-box preference management systems is that legal risk is assumed by the platform provider, at least in part. That said, the benefits of a home-grown solution are lower cost, increased control, and stronger data security since you’re not sending data to a 3rd party. You should weigh the benefits of both approaches with your legal team to determine the path that works best for your organization.

A parting thought on this is that privacy laws and internal processes for complying with them are constantly changing. Your best bet at having a system that can flex with these evolving needs is to have full control over the processing logic… and that can only truly be accomplished via a homegrown system.

The Benefits of Custom Centralized Preference Management

Creating your own preference management system has a lot of advantages.

• No need to send data to a 3rd party
• No on-going service costs
• Ability to create custom processing logic according to your legal requirements
• Ability to integrate easily into any of your applications
• Ability to use any identifier you want (phone, email, an email hash, etc.)

Combining this service with centralized data enrichment you can also store attributes that go beyond privacy to include topic preferences that can help improve targeting accuracy for marketing purposes. And because iPaaS platforms like Workato have a high throughput, we can create services that validate lists of hundreds of thousands of contacts before sending emails if needed.

Combining the two services means that whenever enrichment is requested, the latest preferences can also be returned and whenever preferences are requested, the latest enriched fields are returned. With every request, your systems will be ensuring that the latest data across these two areas is synched to your applications.

What’s more, when you merge who people are (enrichment), along-side what they want to learn (preferences), or whether they want to hear about it at all (privacy) you’ll be cultivating an audience much more likely to engage with your communications.

Stay tuned for our next post where we’ll discuss considerations for building centralized preference management using Workato’s iPaaS platform, Segment’s CDP platform, and Etumos’ FlowBoost.

Have questions, need help, or just want to give a hearty “hello?” We can help, set up a time to speak with one of our consultants.