Etumos’ clients run the gamut of small to large businesses and across the marketing operations maturity curve, and within that client set, as tech stacks have become more complex and adoption of more and more tech becomes the norm for companies of all sizes, we have quickly reached the limits of the out-of-the-box connectors used in integrating these systems, especially when it comes to the reliability of the data.
This can present a huge problem, particularly to our clients who are not only working with complex tech stacks but at the scale of 1M + records in their CRMs. If you’re trying to stitch together all your data to create truly personalized customer journeys based on current marketing best practices and scale your marketing technology, you may find your current setup is limiting the full potential of your data.
And you know what? It shouldn’t be! The “out-of-the-box” application integrations across the marketing technology stack shouldn’t limit what you can do with your data. When you’re at the point of scaling to a more sophisticated, integrated tech stack with a high volume of records you need to be taking the “Service Bus” approach.
Service Bus – Hello, Operator?
Think of a Service Bus in terms of the evolution of the modern telephone. In the beginning, there was the telegraph – a direct 1:1 connection from one person to another. Effective if you only needed to connect person A to person B, but the moment you needed to add in-person C, that message had to be relayed, opening it up to challenges in translation. Not to mention any message had to be short and simple because tapping out long messages was labor-intensive and error-prone.
As demand grew the telephone and manual switchboard were invented – now you could talk and share more detail (more data and more complexity). But it was so successful the manual switchboard just couldn’t keep up with the number of calls between people (a growing number of data sources sending higher volumes of data to an increasing number of destination systems). This ever-growing complexity and an exponential increase in connections between people all across the world created the demand for the automated switchboard and other technologies and standards which connected all callers to their destinations without human error and enabled wireless connectivity and even data transfer (the Internet).
The evolution of telecommunications is a great analogy for understanding the evolution of application integration in your own martech stack: the telegraph = limited, fixed 1:1 application integration, telephone = Service Bus and iPaaS solutions negotiating and securing custom connections between applications, phone number standards = REST and SOAP APIs, switchboard automation and other telecom advancements = a whole host of other technologies like API Management and the Event Bus that can help to scale and govern integration infrastructure.
The Service Bus is that portfolio of technologies used for integrating applications and automating your business in a performance-optimized, scalable, reliable, and secure fashion. A Service Bus creates a system of queueing, redundancy checking, and logging for all points of integration, ensuring not only that you know what is happening within your systems, but that you have error reporting and triage tools readily available to alert you of problems, help diagnose and correct issues and to recover gracefully when a failure does occur.
Maturity Curve:
Category A: 0 – 2mm records in your CRM AND just a few pieces of martech
In this group, we typically see 2 major applications and a few secondary applications (e.g. Marketo, Salesforce, On24 and, say, Drift) – think telegraph in our analogy. At this scale, with this amount of data, and so few API connections, the native API connectors are likely going to do just fine. Most companies are here and don’t have to worry about the next steps until they’re talking about extreme integrations of other data sources (such as real-time Product usage activity which is important for creating trial-based marketing automation).
However, since this is the category many of our clients find themselves in, we want to make sure that we caution this group to start thinking about how data and API calls will be organized in the future, so as to avoid building technical debt that will hinder your ability to actually do marketing the way you want to. Our caution here is that even in this category, you should think critically about your overall integration infrastructure design even as you explore the capabilities offered through native connectors. If you’re at a mid-sized company you should anticipate that your business will quickly expand cross-application integrations and that your needs will quickly outpace the basic configuration options that come out of the box. Keep a close eye on the number of “short-term” mini-hacks and workarounds you’re putting in place and the use of tools like Zapier since these are indicators that you are pushing the limits of system design and beginning to build technical debt.
Category B: 2mm – 10mm CRM records OR 3-6 major pieces of martech
In this group, we start to see native integrations tapping out. You’re simply trying to get more out of the native system than it was designed for – enter the Telephone, automated switchboard and international dialing standards. Indicators of the need to get more serious about your integration infrastructure are:
- You’ve implemented numerous under-documented “mini-hacks” and workarounds
- You’ve started to use basic tools like Zapier to quench your thirst for integration
- You’ve hit API limits in your marketing automation platform or CRM
- You’ve seen lots of errors and missing records starting to build up in your CRM and MAP
- Your systems are lagging and you don’t know why
- Native CRM syncs seem to take hours though no errors are reported
- Data isn’t always making its way to where it should
At this point, you’ll want to utilize an iPaaS (“Integration Platform as a Service”) platform to manage your integrations. Don’t be afraid of the word “platform” but do consider your choices and select something that can carry you through to enterprise-scale without the initial enterprise price tag. Workato is an excellent option as it works well for both IT (sophisticated flows) and Marketing (easy-to-use connectivity) and can grow with you as your transaction and data volume increases. Tray.io is also good for sophisticated automation and small-to-mid data volumes. While we find it is easy to use it does feel more like a traditional developers platform when compared to Workato so evaluate the experience to see what’s right for your team. On the premium side, Mulesoft is probably overkill at this stage but it may be something in use at your organization already and if that’s the case, using it will likely require IT help for integration projects.
Onboarding this technology can be a heavier lift than you might feel comfortable within the early stages of Category B but it will save you a lot of headaches later.
You will likely be storing data somewhere in your org by now – and if you aren’t doing so for marketing, it’s beyond time to start thinking about a data storage and warehousing solution. Take the opportunity to start storing data that is extracted from your applications for easy reuse. Make sure that data records are organized, formatted and retrievable in a way that makes sense for data consumers in your organization. Your iPaaS solution can move data into databases on a centralized data store – a data warehouse – where it can then be transformed, used for analysis and shared between systems – it can even act as a backup of all key data in your critical applications.
Category C: 10mm + CRM records OR 6 + major martech applications
At this point, a well-thought-out architecture is a necessity because there is just too much data to rely on native integrations and middleware alone can’t solve the scalability problem. Looking back at our phone analogy, here we’re talking fully automated switchboard technology with wireless and satellite connectivity for reliable phone service across the world!
API Management will help to manage numerous connections across your applications while ensuring that connecting applications work within the limits of each application’s API. For example, at this stage, we’ve become keenly aware of the Marketo API’s rate limits, quotas, etc. API Management acts as a traffic cop for systems like Marketo – making sure that connecting applications play nice. It also does a lot of other things that IT just loves like improving security and governance.
Rather than having all applications pinging each other for new data, the applications that have data write it out to an event bus, and applications that will receive the data subscribe to it. The event bus is built to handle extreme volumes of data reliably so that nothing is lost in transit. Applications that connect in this way are said to communicate via a publisher-subscriber model. They receive a notification when new data is available and retrieve it using a built-in subscriber mechanism or using tools like Boomi, Mulesoft, or Workato. At this scale, your IT team likely already has a platform in this category in place.
Reporting and Visualization
Okay! Whew. You have data! It’s clean! It’s organized! That was 95% of the work, honestly.
At this point, you should have started moving beyond the native, platform-specific reporting tools that you’ve been running your business on. Data for reporting and analysis are coming from your data warehouse. With a more mature data storage solution housing clean data across applications, your Marketing Intelligence team can come up with some very intelligent dashboards, reports, and visualizations for the entire company (or subsets of the company, as appropriate). Make those dashboards useful and pretty.
What to do with all this information?
Once you have the right data synced at the right time, you suddenly have that elusive big data set to work with. You’ll be running marketing automation that is triggering off the right signals without data lags messing things up. You can run regressions, create multivariate testing and really understand your marketing data. You can have a marketing intelligence scientist or analyst build machine learning models customized for your business, and not rely on black box 3rd party predictive tools. Or you can integrate 3rd party predictive tools and use your own 1st party data to verify how those models are working. Your ability to use marketing data once it’s correct, clean, and warehoused is pretty much infinite and a well-designed “Service Bus” will keep your applications playing nicely with one another and moving data reliably.
Still have questions, or need Etumos to implement? Jump on the phone (see what we did there?) with one of our consultants and we can help.
