Paperjam.lu

Facebook’s messenger app. Photo credit: Kārlis Dambrāns via Flickr (CC BY 2.0) 

The patent application describes a system where an audio fingerprint embedded in TV shows or ads, inaudible to human ears, would trigger the phone, tablet or long-rumoured smart speaker to turn on the microphone and start recording “ambient audio of the content item”. The recording could then be matched to a database of content to allow Facebook to identify what the individual was watching – like Shazam for TV, but without the individual choosing to activate the system.

Diagrams accompanying the patent highlight how the technology would know which adult or child within a household was watching a particular broadcast.

A diagram in the patent application. Photograph: US Patent OfficeA diagram in the patent application. Photograph: US Patent Office

The patent, first spotted by the New York Times, positions the technology as a way for broadcasters to know exactly who is watching their TV shows or ads and for how long. The same system could then be used to build viewing profiles of individual members of a household for better content recommendation and more targeted advertising.

Privacy experts are concerned about the intrusion into people’s homes, particularly as the ambient audio recording would likely catch snippets of people’s private conversations without their knowledge.

“It’s extremely disconcerting for privacy to have an inaudible beacon as it means they want to make it not obvious to the user that the device is listening,” said William Budington, a senior staff technologist at the Electronic Frontier Foundation.

Such a system could also give Facebook a better understanding of people’s social connections as it would show the social network which people were meeting up in real life.

Facebook was quick to downplay the patent filing.

“It is common practice to file patents to prevent aggression from other companies. Because of this, patents tend to focus on future-looking technology that is often speculative in nature and could be commercialised by other companies,” said Facebook’s head of intellectual property, Allen Lo, in a statement.

“The technology in this patent has not been included in any of our products, and never will be.”

Facebook isn’t the first company to design a system that uses secretly broadcast audio signals to track people’s viewing habits. In 2015, a company called SilverPush developed ultrasonic audio “beacons” within TV commercials that could be identified by any device running apps that incorporated SilverPush’s software.

The system allows for far more accurate tracking of people’s viewing habits at an individual level.

In March 2016, the Federal Trade Commission (FTC) issued warning letters to app developers who had installed the software encouraging them to notify people that their app could allow third parties to monitor their viewing habits.

“These apps were capable of listening in the background and collecting information about consumers without notifying them,” said Jessica Rich, the director of the FTC’s Bureau of Consumer Protection. “Companies should tell people what information is collected, how it is collected, and who it’s shared with.”

Less than a week later, SilverPush announced it was killing the tracking software to move into “new areas of growth and profitability”.

Five key moments from Mark Zuckerberg’s testimony. Video: Guardian News

The idea that Facebook already listens to people’s conversations and uses that information to deliver targeted advertising has been a popular conspiracy theory for years. It appeared to originate in May 2014 when the company launched a feature called “Identify TV and Music”, which listened to ambient noise when a user was writing a status update – with their permission. At the time the company had to assuage people’s fears that the feature was always on, eavesdropping in the background.

Despite denials, the rumour persisted and in October 2017 the company’s head of advertising Rob Godman had to issue another denial in response to an investigation carried out by tech podcast Reply All.

“I run ads product at Facebook. We don’t – and have never – used your microphone for ads. Just not true,” Goldman tweeted.

Olivia Solon in San Francisco