Meet Pixoneye, the firm using raw photo data to better understand consumers

Elliott Haworth
Follow Elliott
Rather than looking for a face, coffee cup, or tree in an image, for example, Pixoneye’s SKU is trained on stock images of such things to understand the data behind the objects.

The approaching data economy will be dominated by those who can extract the most value from raw ones and zeroes.

With the proliferation of handheld technology, almost every action we take creates data in one form or another – whether walking the dog, buying lunch, or posting on Facebook.

But if there’s one plainly identifiable habit of the modern consumer, it has to be taking photos, of everything, all the time. The photos we take, predicted to be some 1.3 trillion annually, arguably provide more insight about our daily lives and habits than any other data imaginable.

The advent of biometrics has made it easy for a business like Facebook to scan and identify people and objects in your photos, which, let’s face it, can feel a bit creepy. If only there was a way of extracting information from raw photo data – essentially, lines of code – without interacting with the images, or identifying specific persons, places, or objects.

Of course, there is. Pixoneye, an Israeli software company that has found a second home in London, produces a software developer kit (SDK) – a kind of programme that operates in the background of a business’s own app – that does just that.

So when an app asks if it can have permission to use your photos, there’s a good chance it’s Pixoneye’s SDK gleaning insights about you to personalise the service – or target advertising.

How does it work?

Clearly, this is no simple task, so I caught up with co-founder and chief executive Ofri Ben-Porat, to find out how it works.

He says that it’s only as processors have grown faster, and less latent, that Pixoneye’s SDK has become possible. “You can train devices to learn processes that you previously would have had to take out from the device and run on a separate server,” he says. “Let’s take a simple thing – image processing, or image understanding. When it started, in order to run a picture – a photo – through a processor or classifier that would be able to detect things in the picture, you would need 10GB of power, just to understand one photo. Today, we can run these huge classifiers on 1MB. Which means automatically, we can take all of these crazy capabilities and put them onto a device.”

Founders Nadav Tal-Israel (left) and Ofri Ben-Porat (Source: Pixoneye)

Rather than looking for a face, coffee cup, or tree in an image, for example, Pixoneye’s SKU is trained on stock images of such things to understand the data behind the objects.

“So unlike computer vision, in the sense of object detection, which is used a lot for meta-tagging, or to tag objects, it’s raw data. It’s what we call signalling, that’s the layman’s term – the scientific term is cross-vectoring, it’s the opposite of object detection.”

I ask how it differs from the sort of recognition software that Facebook and others use.

“So object detection says: ‘bike, dog, person, tree’. Cross-vectoring says ‘green, green, green, red, red, this texture, this angle’. But it takes out signals. So let’s say that there’s four objects in a photo, it will have five signals. So it’s the scenery of the photo, and four objects, each determined as a signal, rather than what it physically is.

“And then all the signals from all your galleries get condensed into one single vector, and we call that the feature vector. This feature vector looks like a graph, a bit like a heartbeat monitor, all condensed; just raw feature data. So I can’t look at the data and say ‘ooh here’s a dog’ – it doesn’t translate to anything; but it can be extracted.”

Pixoneye has developed its own specific language to read the feature vectors, Ben-Porat says, that “breaks down, not into objects, because that would be meta tagging, but into profiling. Our big thing is not taking photos and tagging them for what’s in the photo, but taking them and trying to tell a story about a person.”

Pixoneye has derived 150 characteristics that correspond to the vectors – for example gym-goer, watches sports, wine drinker, or holiday with kids.

The idea is that it scans your photos for insights, without actually scanning your photos, putting users into groups that are useful for personalisation or targeting, while “respecting privacy”.

But Ben-Porat says that it also negates the need for businesses to constantly buy third-party data, by giving them their own data management platform. “The data belongs solely to the app that has installed the SDK – we don’t own the data,” he says. “In fact, if you have two apps on the device that both carry our SDK, it analyses it twice, and we have no way of cross referencing. We double dip on the revenue because we get paid from each app!”

Related articles