Let’s talk more about the emerging discipline of marketing devops or marketing systems developer with one of the masters, Sanford Whiteman. Marketo Nation users know Sanford as the ultimate resource for scripting and deep systems answers, helping people put together custom experiences that are not in the box.

The context is that there are some people who aren’t quite engineers or marketers, but who work to integrate martech systems or create non “out of the box” experiences for people. Sanford goes deep into the reasons for coding integrations and on-page magic as well as the kinds of code that may be needed.

If you are a web developer, self-taught coder, or practical coder a role in marketing-devops may be for you. If you are like me, a marketing operations pro, Sanford’s insight is valuable for a deeper understanding of the customer experience, stack integrations, and how a developer can help you make better decisions.

Josh: Is there an emerging discipline of “marketing devops” and what does that look like to you?

Sanford:
I’d like to think we’re carving out a new discipline!

It’s becoming clear to marketing groups large and small that SaaS doesn’t mean “It just works!” From the start, it’s just meant that a product’s feature set will work (relatively) reliably in the cloud and your IT staff can be (relatively) detached from day-to-day maintenance.  But there’s never been a guarantee that the new features and integrations you want will be built quickly, or ever!  SaaS vendors can’t survive without APIs these days, and those APIs can smooth over user frustration…but you need to have someone on hand (if not on staff) to work with those APIs.

I think there are three distinct tiers where this discipline is expanding:

• the client/browser tier.
• the middle tier, where you integrate services from within the martech stack itself.
• the app/database tier.

I’m lucky to be able to work on all sides, but that may not work for everyone.

At the client, you’re doing stuff like

• customizing how forms package data for the CRM/MA stack.
• tweaking web tracking to integrate anonymous analytics with named leads.
• integrating third-party landing page builders with your back end.
• tracking lost web activities and all manner of browser-based troubleshooting.

In the middle, you’re

• sending leads to outside services, typically using webhooks, and working with returned data.
• writing your own webhook endpoints for advanced workflows.
• looping back to call the vendor API from within the vendor’s infrastructure (very meta!).

On the app/database level, you may be

• populating a martech data warehouse with lead data and behaviors
• building internal-use web apps to request campaigns or query leads
• importing internal product data into the vendor database as custom records

Josh: What is the skillset of a marketer-developer?

Sanford:
Working at the client tier — where, in my experience, it’s easiest to get a foothold in this new marketer-developer world — you have to have “be one with the browser.” So above all, true knowledge of JavaScript — not jQuery or other simplifications, but raw, down-and-dirty code. Whether or not client-side martech embeds a JS framework (it often does not, in order to keep the code footprint lean) underneath it all is standard JS, so if you want to troubleshoot, tweak, and extend, come armed!

You don’t have to be an advanced JS developer, don’t get me wrong, but you must have done more in the past than copy-and-paste snippets from blogs.

In working on a company’s public-facing website, after all, you’re taking the company’s reputation and revenue into your hands. You should already think of yourself as a coder and been paid as such in the past, even for small projects. I think one of the biggest liabilities in martech right now is non-devs nervously impersonating developers because management doesn’t yet get the need for a specialist. It’s a nerve-wracking position to be in, and these folks are happy just to have someone who trusts her/himself to do the work. Even if it takes a few iterations because the person is relatively junior, at least the marketer’s job isn’t on the line anymore.

[Josh – I believe copy-and-paste tweaking of code is a great way to learn the basics, but Sanford’s right that you should not do this with business dependent sites. I always tell people I can do just about everything until we need to code and I’ll go find someone like Sanford to code the experience]

Tied for second, after JS, would be understanding of the HTML DOM and at least some of how HTTP works under the hood.  You’ll be doing things like adding 3rd (or is it 4th?) party enrichment libraries to a form, where the form itself is embedded from your martech system into your CMS, and the CMS has its own JS module loader. So you have to know where script order and dynamic DOM elements intersect, when cross-domain posts (CORS, et al.) work or don’t work, all that fun stuff.

Working in the middle tier may be the easiest of the three.  If you’re only calling simple APIs, you need to get JSON and XML to understand HTTP responses, and you need to be familiar with URL encoding, query strings, and GET vs. POST. 

In the grand scheme, really basic. But there’s a huge difference between someone who can instantly catch that a question mark needs to be encoded, or that a JSON object has an extra level of nesting, and someone to whom that stuff remains a frightening foreign language.

Again, the difference is in fundamental comfort level more than top-tier expertise: you’ll walk into places with broken configs all over the place and you have to take in mistakes at a glance. You might waver about whether your changes will fix the problem, but you can’t waver in making those changes.  One of the things you’re offering is the confidence about wider technology that both marketers and marketing ops folks can (understandably) lack.

If you’re building endpoints for the middle tier, you can get by with pretty-good PHP, together again with that core understanding of HTTP requests and responses. Better still would be knowledge of “endpoints-as-a-service” like Lambda, where you can use JS, Java, or Python in environments that are perfect for the stateless character of webhooks, like a hook that changes a date format or throws a single row into a remote database. My coolest webhooks are under 10 lines of JS hosted on AWS.  Once you’re adding persistence, of course, things can rapidly get quite advanced. I can’t tell you whether to use Redis, MongoDB, or CouchDB but you should acquire an opinion about such things!

Working at the data/app integration tier, you need to have conceptual command of how long and wide streams of JSON or XML, often embarrassingly de-normalized, can be transformed into a scalable SQL schema. (I don’t see NoSQL favored in this space.)  You need to think about bulkifying and checkpointing and intelligent retry logic, since you’ll be fetching data from rate-limited and (let’s put it nicely) not-always-up services.  You may not be able to choose your weapon from among MSSQL/MySQL/Oracle/Postgres because your client/employer already has a vendor commitment, so you should be ready for any of these (if you’ve been using MySQL-only awesomeness, that’s cool, but be aware of what’s standard and what’s not). For your application platform, I love me some PHP, but this is more the place for .NET, Java, or NodeJS where long-running processes are more comfortable. Using Apache Camel and like integration engines will lead to more resilient implementations that you won’t need to check in on constantly, which is preferable if you’re a busy consultant.

Together with all the above, you need an ease with the vendor platform as used day-to-day by marketers, admins, and marketing ops managers. Note I said ease — not necessarily expert knowledge of every non-API-exposed nook-and-cranny of the product. Similarly, you don’t need to attempt thought leadership in the marketing space. There’s only so much you can know, and believe me, you will have your hands full trying to figure out how familiar UI features are surfaced, translated, and sometimes mangled beyond recognition when accessed via API. You’ll spend hours filling in missing documentation or correcting doc errors, which doesn’t leave time for book learning. And it’s okay to have some lingering blind spots about the product. I couldn’t tell you much about Marketo’s Calendar, for example (though now that I’ve said that in print, I’ll have to go learn!).

Josh: How much of a programmer or Computer Science engineer do you need to be successful in this role? Is it overkill to be in CS?

Sanford:
Good question!  Because of the inevitable emphasis on JavaScript, I have to say a CS background would help. JS was once derided as too simple for “real” programmers, but it turns out to be a laboratory to explore (and get bitten by) core CS concepts, like race conditions, concurrency, async programming, event-driven architectures, and OOP.

You’ll see me responding to posts in the Marketo Community with remarks like, “That code has a race condition and will lead to unexpected consequences in such-and-such situation,” although I’m pretty sure the person who pasted the code has no idea what I’m talking about. Sometimes there’s no way to rephrase such concepts in layman’s terms, so “talking the talk” is an advantage.

That being said, I don’t have a formal CS education myself: I was a theater major! But I’ve had many years since then to hone my impersonation of a no-nonsense engineer. In all seriousness, though I was always a math person, even after entering IT professionally I spent a long time on the systems side (managing firewalls, networks, and mail servers) before getting into code proper. So there’s clearly no need to have been a coder since birth. But you have to be willing to peek at CS books, and when you’re not in school anymore, that stuff can be really dry.

Even better than an academic CS education, frankly, is working directly with mail, database, web, and/or DNS servers, since your martech universe is made up of those layers. Unfortunately, with services moving to the cloud, those positions are relatively scarce now.

In the (g)olden days, there’d be an in-house person poring over SMTP logs for delivery errors, managing DNS records, and looking into HTTP 500 errors: that person, adding a bit of dev experience, would be perfect to move into the marketer-developer role. If you do have one of those jobs now and are looking for a change, it’s a great time.

Josh: If you were hiring someone to take on your job, what could they expect to do every day?

Sanford:
Lots of debugging. I experience the web in a very different way from most people: it’s a lot of inspecting production sites in Chrome or Firefox DevTools and/or running them through a proxy like Fiddler. If I’m lucky, I get to build solutions on a dev site, but more often the damage is done by the time I’m brought in — a form’s not posting or not rendering, Munchkin isn’t logging clicks, a video embed was rolled out without analytics hooks — and my contact in marketing has no access to the underlying CMS. So I spend a lot of time watching the wire and the DOM and testing JS fixes in the console. (I use debugging tools like CodePen and ngrok to isolate issues when possible, but working directly in the browser is a reality.)  And the clock is ticking because the site is live to the world and there are lot of unhappy people.

Then, once I’ve found the fix, I send the code back to my contact so they can go through channels to update the site. It can be a heavily bureaucratic process, so you have to know how to test your own code, including cross-browser, so it doesn’t fail in-house validation and you lose trust.

Of course, the above are the bad days! When there’s not an acute crisis, I work on packaged client libraries which are managed more properly with WebStorm + git and have a real QA and release process.  An example would be multi-touch enhancements for Marketo forms. Getting my clients to go from single-use scripts to packaging all their Forms 2.0 behaviors in one place is a prime objective. Hard to enforce, though, when indeed you can just throw a whenReady listener on any page.

When working on the app/database side, there’s a similar debugging-heavy experience. It’s not just calling a REST API endpoint and parsing a predictable response into a db. It’s analyzing the ways in which the response departs from documented expectations, how the data structure changes in response to changes made in the UI, whether foreign keys or point-in-time de-normalized values are returned, how values can go missing from an API perspective (a non-no for a martech data mirror, as you can imagine).  There’s an healthy mistrust that you develop, which as I remarked above results in you writing your own documentation for commercial software.

Helping users pro bono. Online and offline, at least every couple of days.  As you might recall, Josh, there was a point in mid-2014 when I suddenly sprang up on the Marketo Community as a know-it-all about Munchkin and forms! It wasn’t as effortless then as it is now, but I knew the only way to continue to learn was to look at somebody’s else’s real-world problem and solve it. With the attitude that “I am going to figure out how this can be done.”

 It can be taxing, but it’s not wholly selfless: beyond your clients, user communities for enterprise products are where you’re going to learn about the real world. Some community users will make choices you just know are off, and you can guide them toward a different goal and its solution, learning how to justify that decision. Other times, you bite the bullet, realizing that a marketer’s desire to have a form work a certain way may come from higher-ups more than their own stubbornness: either way, you’ll meet clients with similar unbending requirements, so get used to it!

Scanning the Community back then, it was clear that the marketer-developer was a common fantasy, but probably didn’t exist. There were so many threads where the initial answer was “hire a developer” and the response was “we have a developer, but s/he has no idea how this works.”

It also seemed like (and unfortunately still does) even if a developer were in the picture somewhere, they didn’t have a Marketo account, or they had one but didn’t feel like logging in. As a result, all the questions are routed through a non-technical marketer (usually without any sample code, just “my developer says it isn’t working”). I still see this happening and it’s so frustrating. I don’t mind saying that any developer who can’t ask their own questions is probably in the wrong position. A marketer-developer definitely needs to engage with the user community, even if it means seeing (eek!) uninteresting questions about channels and programs.

Fielding off-the-wall ideas. No two ways about it, once you offer yourself as a resource who can do anything with a martech stack, people will come up with things that, well, could be done yet shouldn’t be done.  Whether it’s because of reliability, scalability, expense, legality, ethics, or all of these.  Having a bit of a marketing mind is useful here because you can honestly say, I understand the desire… but you can also serve as a logical voice because you know about the practical difficulties, and can stop bad ideas from becoming budget items.  On the ethical front, as a techie you’re probably better tuned in to privacy, security, scams, and scandals, even if you’re not yourself particularly zealous.

Evaluating vendor promises. Some things can’t be done, but a vendor is upselling them anyway. I’ll keep the details vague, but a vendor recently promised to perform a certain type of data mining across a 5-million lead Marketo database, real-time, non-batched, updating constantly throughout the day in response to user behaviors.

The vendor was sure they could work it out… as soon as my client signed the contract, of course. It was great to be there to think it through, poke holes on the client’s behalf, and ultimately save them six figures of vaporware. That’s one of the things you’re there for: to be called first, before anyone makes any hasty technical commitments. You might not charge for every hour you spend doing this kind of work, but you prove yourself indispensable.

Dipping into deliverability. Because of my reputation as “the” marketing-affiliated techie, I’m occasionally brought in on deliverability problems because in-house IT has washed their hands of such matters. They won’t even do a DNS dig, because “Isn’t this all sent by your high-and-mighty SaaS platform?”

 But waiting for vendor support to tell you why email bounced to certain domains is a slow process (that’s going to be a low-priority ticket, since after all, the bounce was probably not in error). So it’s good, though not necessary, to be available for extra work outside of coding. Remember, you’re often helping folks that literally don’t know the first thing about SMTP, and they’re in a panic. You don’t have to be an expert, but as long as you are confident in your use of tools you can add value to the situation.

The real work. When I can get a break from the above, I code. 🙂 To be honest, the marketer-developer role feels quite reactive overall. Though I have longer-term commitments as well, I’m frequently doing one- or two-day site-specific engagements, like a form with some specific JS-driven UX and data management expectation… that’s almost like the one I built for another client the other week…but turns out to be 80% new code (plus I always like to try new tricks). Even though such projects are scoped out in advance, they have a built-in urgency because, well, you allocated 4 hours, so you’ve got to keep moving!  Even a longer-term project with very stable expectations still involves the debugging stops-and-starts above, since nothing is exactly what it seems in martech.

Josh: Which kinds of firms benefit from having this as a separate role?

Sanford:
Without someone in the marketer-developer role, the switch to SaaS can leave marketing in a permanent state of “settling.” I feel all firms with a martech investment should at least have a trusted consultant serving as marketer-developer. [Josh: I agree most of the time]

It’s not practical for someone to be a pure marketer for N days in a row, then switch gears suddenly into technical mode. They won’t keep, let alone build, technical skills that way, and the only way to do the job well is to spend time dissecting martech every day.

Unfortunately, save for some tech firms, companies typically don’t want to pay their employees to help other people online, or to set up artificial “what-if” problems and solve them. So this role may be best fulfilled by a consultant unless the firm can ensure that there’s a consistent flow of technical projects.

Firms that have a significant in-house CRM team and a robust martech investment, yet don’t even have an on-call mar-dev, are losing out. Of this I’m sure, even if I only have case studies! Here’s a good one, though: my favorite client from way back, for whom I do all kinds of other non-martech stuff like proxy servers and database clusters, had been using Marketo for a full year without having lead activities tracked on their main site. Even though they knew it was broken, there was no one to own the problem.

Awesome Apex devs and SFDC admin? Check. Awesome .NET web team? Check. But someone to do that “be the browser” type of troubleshooting I mentioned above, knowing they were working on a SaaS product that supposedly Just Worked? Nope. So the problem remained unsolved until I got approval to find and fix it (took several hours and was a Munchkin bug, for the record). Needless to say, that particular company has no trouble understanding why a marketer-developer on speed dial is critical. But for every company who reaches out, there must be who-knows-how-many who’ve given up on solving problems.

Josh: How can a marketer-developer help marketers tell their story better?

Sanford:
With apologies to Bishop Berkeley: if a marketer tells a story and doesn’t get multi-touch attribution, did they make a sound?

In seriousness, if we look at storytelling as a series of episodes told across different media (transmedia) then stitching together audience reactions to the collective story is critical. Whether you adapt the story on the fly like a choose-your-own-adventure style, a.k.a. real-time personalization — or maintain a single narrative, your technology needs to be a step ahead of you, not behind.

Perhaps the marketer-developer can be the story’s chorus, maintaining the collective throughline — from a Twitter card, to an Unbounce LP, to print, to Facebook, to email. Without the marketer-developer, you have only episodes, and no way to know if they’ve been heard.

Wow, thanks Sanford. This is unique insight into an important part of creating a customer experience. It’s not good enough to use one or three platforms anymore. Your entire stack needs a seamless, human oriented experience for the audience. Often, the only way to do that is with code.