The Empathy Engine: Big Data and the Necessity of Understanding

Originally written February 2015

If true creativity has always rested on the ability to first master the nature of the problems that you wish to solve, then the advertising industry’s refusal to truly master the data that unravels the behaviour of their customers is, at its most basic, terrifyingly complacent. However, the advent of the Big Data era must surely signal the end of this reticence: this stream of data is the product of a media landscape that is more personalised than ever, and understanding that data at an individual level can help us understand the solutions that our consumers want. It is this data-driven empathy that should form the backbone of great advertising in the near-future.

A creative industry ignoring the science of creativity

At an industry level, advertising has long ignored the central importance of getting the right input into the creative process. The brainstorming model, established by Alex Osborn of BBDO in the 1940s and 50s and perhaps the most influential method for creativity in the industry, identified four rules of great idea generation: focus on quantity of ideas first, withhold criticism, welcome the unusual, and combine and develop the output[i]. The need for preparation - understanding the issues at hand; what people respond to, what triggers purchase, where buying happens - is left out of the traditional approach. Instead, planners and creatives since time immemorial have been expected, in the words of David Ogilvy, to be “governed by intuitive hunches and inspired by the unconscious”[ii] Data, classically, is to be feared rather than embraced; Bill Bernbach warned against buying into advertising as “science”[iii]. Ad men worked primarily inside their own minds; “create things you want”, warned George Lois, not what people “think they want”[iv]. The pursuit of knowledge was to be viewed with suspicion.

Sadly, like many of the beliefs that drive everyday behaviour in our industry, our views on creativity are at odds with what all of the available science tells us on the subject. First, the “no idea is a bad idea” rule of Osborn’s brainstorming process is entirely without foundation - debate and criticism are in fact a vital part of the creative process and help generate more good ideas than the classic positively minded model[v]. Secondly, it’s important to gain stimulus from outside sources, original voices that are able to show you something new about a problem[vi]. Lastly, and most critically, rather than existing as a skill born of unconscious intuition, creativity depends on knowledge and understanding: people are better at generating ideas when they are intimately acquainted with the nature of the problem they are trying to solve. New ideas are not produced by liberating your unconscious but by fully preparing yourself; understanding the challenge; acquiring skills. As the inventor Jacob Rabinow pointed out, becoming creative rests firstly on having a “tremendous amount of information”[vii]. Most certainly, creativity is not something hidden within our unconscious, waiting to be liberated, but a power developed and sharpened by learning and preparation.

The centrality of knowledge and learning to the creative process is easily overlooked but in plain sight across the path of history. As Mihaly Csikszentmihalyi reminds us, the birth of Renaissance architecture in Florence was only possible when scholars of the time rediscovered the ways of their Roman ancestors[viii]; inventions like Morse code were developed iteratively over decades, where incremental solutions were produced by learning from previous failures[ix]; even apocryphal “aha!”-type discoveries, as Watson & Crick’s imagining of the double-helix structure of DNA or indeed Alexander Fleming’s discovery of penicillin are often told, is almost always the product of years of study (in the case of Watson & Crick, they had years of research and of failure behind them; Fleming had been studying bacterial infection since World War One). Learning & understanding your problem is an inescapable precursor to creativity. It is not optional.

How better data can improve the creative process

There is no reason to suspect that advertising requires any less information and preparation in order to be creative, and the problem we generally have to understand is a behavioural one: what people do, and how to get them to change, even infinitesimally, in our favour. In this context, Big Data offers us unparalleled insight into what people do, but because it is a technological phenomenon as much as a behavioural one - the product of a new, personal age of media, one where our data trail is a product of our desire for the customised and the on-demand - it offers us both the ability to better understand our problems and to act on that understanding.

On a very basic level, never before have so many planners and creative people within the industry - irrespective of the wealth of their agency masters or influence of their client - had the ability to generate credible, robust insight into the preferences and habits of their potential audiences. The backbone of this has been the macro-level view of population behaviour: the data into interests and triggers that can be found across the Google suite of tools, helping us better understand when people are in market, where they come from and what helps convert them; the insight into the topics of conversation that are driving the zeitgeist, from something like Twitter; consumer preferences and product usage insight can be gleaned from the millions of public reviews left on Amazon and other crowd-fuelled review platforms; even location data from public transportation bodies are now being released, allowing us to better understand population flows across days, months and years. If preparation and understanding is the name of the game, the curious planner now has the ability to arm themselves with huge amounts of insight into when people are likely to be in market for their products, what triggers purchase, where best to talk to them, what the cultural drivers of interest and conversation might be, what features might be most compelling to the buyers, even the emotional state of ones audience at a given time of day or year.

For any industry, this birth of new data would be a huge benefit, but in advertising - an industry traditionally sceptical of the benefits of knowledge and logic - it offers something more revolutionary, allowing us the ability to finally plan on what people do rather than what they say. The classic planning aids - the so-called “qualitative” insight generated by the focus group or vox pop; the consumer surveys like TGI built on claimed behaviour - now can be twinned and corroborated by data built by actual behaviour. We don’t need to wonder if consumers tell us the truth when they tell us they talk positively of our product, or whether they want to pay more for it, or that they thoroughly research their purchase before making it - we can immediately check if that matches what they actually do.

The personal world demands a more personal approach

So whilst creativity demands better preparation & understanding, and that a large amount of open-source behavioural data offers the advertising world an opportunity to prepare and to learn more effectively, unlocking the true creative power of Big Data demands that we understand why Big Data exists. And Big Data - the stream of personal and collective data we leave behind as we interact with any form of digital media or platform - is at once both a structural phenomenon and a behavioural one; it is produced on one hand simply by the nature of digital technology and the way that technology is set up, but at the same time it reflects what we want - nay, what we expect - from that digital technology. It is precisely because we want a personalised experience of our media - buying suggestions, curated links to the articles and videos we might want, tailored search results, individual discounts, exclusive access - that we allow the world to collect our data without a second thought. Not only does this personalisation save us time, sorting through the myriad options of the digital world to get closer to the limited set of options that humans tend to prefer, it improves what we see, providing more relevant and welcome content. That’s why Big Data exists: it is an output of the way we seek to shape our experience of the digital landscape around our preferences and desires.

It is in this personalised and bespoke world where the power of Big Data to properly transform the creative process becomes apparent. In a world where millions of people default to their Facebook newsfeed as their primary broadcast platform, where we devolve control over what we see and what we choose to the algorithms powering our Netflix account, our Amazon homepage, our Google listings, we expect the machines to learn about what we want. We like to know what people like us bought, or what videos are being watched right now, or what the most salient news story might be at a given time; the digital world, without this layer of empathy, would be too confusing otherwise. And Big Data, for advertisers, is nothing more than an empathy engine: it allows us to understand the desires of the people we’re advertising to, in the same way the Guardian might understand the stories you’d like to see. Data, in today’s world, is understanding; harnessing the power of data means understanding the wants, needs and desires of an audience on an almost individual basis.

The necessity of empathy: three ways Big Data can transform the creative process

The idea of data as a creator of empathy means we can quickly get to a personalised view of advertising that matches people’s personalised expectations of digital media. Not only can we better understand the desires of our audience - as expressed not by their words but by their deeds - we can begin to shape what we expose them to on the basis of that empathy, as the world of Big Data allows not just better understanding but greater facilitation and quicker evaluation: we can make more informed guesses about what people want, get it to them more directly and more accountably, and understand the benefits of that approach in real-time.

The first benefit is the most clear: greater empathy through Big Data means we can craft advertising messages and campaigns that better trace back to what people want from our products to how they use them. Textual analysis of product reviews can allow you to understand the way consumers will respond and react to product features, not through asking them to guess pre-purchase but by understanding how they use post-purchase; understanding the conversion rates of specific pieces of copy in your search listings allows you to test lines and copy in the real world rather than a lab; time and location data around purchasing allows you to understand what sort of contextual triggers really stimulate purchase. You can finally, properly, make real consumer-driven empathy a part of the creative process.

The second benefit is then about how the world of Big Data today allows us to connect the superior ideas, driven by genuine consumer empathy, back to the people for whom they are most interesting. This is not about re-embracing segmentation - as anyone that has studied the work of Andrew Ehrenburg or Byron Sharp will confidently tell you, anything that compromises or reduces overall reach is to be loathed by marketers everywhere[x] - but about taking a personalised broadcast approach; speaking to millions on a more one to one basis, where we pull out the features and brand assets from a broader pool and communicate them with the people most likely to respond positively. We can see how today the breadth of platforms like Facebook allow us to take a more personalised broadcast approach, but as digital TV broadcasters slowly open up their platforms and allow more of a programmatic approach in those spaces, a more consistently personal broadcast approach will be possible.

Thirdly, the principle of empathy is central not just to predicting what someone likes at the beginning of the process, but being able to better understand how they react once they’ve been exposed - and here is where Big Data again helps us be more creative, opening a clear flow of understanding between advertiser and shopper, allowing us to more rapidly, in the word of David Ogilvy, “back our winners and abandon our losers”[xi]. No longer should I have to keep seeing the ad I do not respond to, or the copy that doesn’t inspire a click in a competitive search context; we can root out our under-performers and repurpose on the fly.


We see then the way that Big Data, through its ability to create more genuine, ongoing empathy amongst advertisers, can produce more creative, more effective and more powerful ideas: not only does it allow us to better learn about the problems we face - an essential part of the creative process - it also improves the quality of information we have access to, and the technological benefits of a Big Data world allow us to implement that understanding on an almost individual level, quickly tailoring, serving and optimising our ideas to the ways in which we can observe their effects. By dismissing Big Data as a distraction, or a handbrake on creativity, we continue to make the mistakes we have always done: operate as if we, and we alone, can intuit the desires and needs of our audience. Instead, we should embrace Big Data’s power to truly deliver a more empathetic advertising model, improving our output and making it fit for the personalised media landscape we all exist in today.

[i] Keith Sawyer, Group Genius, Basic Books, 2007, p.59-60

[ii] David Ogilvy, Confessions of an Advertising Man, Atheneum, 1963

[iii] Taken from Ad Age, 1999:

[iv] George Lois, Damn Good Advice, Phaidon Press, 2012

[v] For more on this, see the work of Charlan Nemeth at UC-Berkeley, who has consistently demonstrated the power of debate & dissent to generate more ideas than the positive approach pioneered by Osborn:

[vi] Here, the work of Brian Uzzi is instructive: groups with a balance of established connections, creating a sense of comfort, and new voices, introducing stimulus, outperform those where participants know one another well, or have never met

[vii] Mihaly Csikszentmihalyi, Creativity, Harper Perennial, 1997, p.48

[viii], Ibid, p.32

[ix] Sawyer, Group Genius, Basic Books, 2007, p.99

[x] For an up to date view on the implications of Ehrenburg’s work, see Byron Sharp, How Brands Grow, OUP, 2010

[xi] David Ogilvy, Ogilvy on Advertising, Prion, 1983

Charlie Ebdy