How Do Ads Know What I'm Thinking

Earlier this year, my friend Max gave me a knife from Japan as a gift. That evening, as I was lying in bed looking at Instagram, I scrolled passed an ad of what looked like exactly the same knife. I did a double take, got out of bed, retrieved the knife from the kitchen and compared it to the one my screen—it was a perfect match, a Masomoto KS. I hadn't Googled the knife, taken a picture of it, or event sent a text about it. The only interaction I had about the knife was face to face with Max when he gave it to me. This felt like more than a coincidence — it felt like I was being listened to.

Tega Brain and Sam Lavigne, two Brooklyn-based artists whose work explores the intersections of technology and society, have been hearing a lot of stories like mine. In June, they launched a website called New Organs, which collects first-hand accounts of these seemingly paranoiac moments. The website is comprised of a submission form that asks you to choose from a selection of experiences, like "my phone is eavesdropping on me" to "I see ads for things I dream about." You're then invited to write a few sentences outlining your experience and why you think it happened to you.

They have already collected close to 700 testimonies. It turns out that the feeling of being listened is among the most common experiences, along with seeing the same ads on different websites, and being tracked via geo-location. "We've also had a bunch of stories of people having a conversation with a friend and then their phone will suddenly chime in with some unwanted suggestion," Tega said. "That's pretty invasive." Sam adds that they've also heard stories from people who have just started dating someone new and then start getting ads suggesting products targeted at their new lover. But Tega's favourite, so far, was from someone who said that they were thinking about donuts because they hadn't eaten lunch that day, and then their Google maps rerouted them for some unexplained reason right past a Dunkin' Donuts, as if they were being monitored by a type of algorithmic ESP.

While the rumour that internet companies are listening in through our phones has persisted over the years despite repeated denial—most recently by Mark Zuckerberg at the congressional hearing following the Cambridge Analytica fall out—evidence suggests that they are telling the truth. To wiretap users would not only be illegal but would require storing unrealistically large amounts of data as well as software sophisticated enough to parse the ambiguities of human language and pick out valuable information.

The New Organs website captures uncanny targeted advertising experiences.

But according to Brain and Lavigne, while Facebook and other platforms might not be using microphones to listen in, they are developing increasingly opaque and complex technical systems for tracking, collecting and processing information about users. "What we want to show is that you are being spied on, but not in the way that you might imagine," Brain said. "We are stuck in this 20th century idea of spying, of wiretapping and hidden microphones. But really there is this whole new sensory apparatus, a complicated entanglement of online trackers and algorithms that are watching over us."

It is this new sensory apparatus that Brain and Lavigne metaphorically refer to as "new organs", as if the online surveillance framework used by social media platforms like Facebook has somehow transfigured into a semi-living organism. Their project aims to reveal more about how the new organs function and what impact they are having on our experience of the world. "These new organs don't actually need to listen to your voice to know that you like Japanese knives," Lavigne told me. "They actually have ways of coming to know things about you that we don't fully understand yet." In other words, these new methods of data collection have become so uncannily accurate in their knowledge of you as to occasionally feel indistinguishable from actual ears listening in on and understanding intimate conversations.

There are a few things that we do already know about these new "organs" of data processing, as defined by Brain and Lavigne. We know, for instance, that they have an insatiable appetite for personal data. They gather this by first tracking online activity, which is enough to tell them what people like, what they search for, what they listen to, what they read, where they're walking for dinner, and also, worryingly, who their friends are and what they like, read, purchase—data that is gathered without their awareness. But, then, the organs also gather information purchased from commercial data brokers about people's offline lives, like how many credit cards they own, what their income is, and what they purchase when they go grocery shopping. And all of this information is triangulated with friends' data, because if they know what those dear to you are buying—a Japanese knife, for instance—there is a good chance that that person will be interested in that very same thing.

The new organs process this enormous amount of information to break you down into categories, which are sometimes innocuous like, "Listens to Spotify" or "Trendy Moms", but can also be more sensitive, identifying ethnicity and religious affiliation, or invasively personal, like "Lives away from family". More than this, the new organs are being integrated with increasingly sophisticated algorithms, so they can generate predictive portraits of you, which they then sell to advertisers who can target products that you don't even know you want yet.

While public interest in online privacy has intensified this year following the Cambridge Analytica fiasco, most social media platforms continue to hide the workings of their surveillance methods from users. Facebook, for instance, allows you to see a handful of the categories it assigns in your ad preferences, but this is a performative transparency that conceals the scope of their surveillance. My Facebook, for instance, told me 41 of my categories, leaving out the other 52,000 or so ways in which it could know me.

Both Brain and Lavigne find this opacity concerning, and their original idea was to forensically untangle how these new spying apparatus work in order to educate the public about the creepy realities of what they call surveillance capitalism. "So, with your knife story," Lavigne explains, "we might reach out to the marketing company responsible for the ad, talk to them about what groups they were targeting and why, and then try to reverse engineer the whole system."

What they've discovered, though, is that the opacity of these systems runs so deep that a full explanation of how they work might be impossible, even for experts. "Not even the marketing people and data scientists completely understand how the algorithmic distribution of their content works," Brain said. "They've created this big black box that perceives and understands the world in a way that is so complex that it is actually mysterious."

Instead of trying to reverse engineer the mechanisms that power the creepy omniscience of these new organs, Brain and Lavigne have decided to use the testimonies they collect on their website to produce a series of videos about the subjective experience of being spied on by machines and how conspiratorial thinking is evermore present in our online experiences.

"It's getting harder and harder not to feel paranoid, I find," Brain said. "Even though I know that my microphone isn't listening to me, sometimes when I'm reading through the responses that people submit to the website I catch myself thinking that they're all true." Lavigne adds that this paranoia is only going to intensify in this age of mass surveillance because it reflects an undeniable truth of our digital age: we are being watched by technological organs that we don't fully understand. "I think ultimately the reason we feel this way," Lavigne said, "is because it is all a conspiracy. Just a different type of conspiracy to what we're used to."

Oscar Schwartz is a freelance writer based in New York.