The smartphone is the ESP

I love this short film below which, yes, is just video captured by a iPhone with music in the background. Only...

The soundtrack is created in real-time based off the visual information presented. That is, music is being created, instantly, by the device, based on what the camera sees.

It's from the app, Sonified.

Sonified takes real time visual information from your iPhone/iPad video camera and uses it to mix 16 stereo tracks of CD quality music in real time, translating what your video camera sees into sound. A real time instrument, Sonified uses your iPhone/iPad's video camera to translate qualities of light and color into music, a process called "sonification". Using Sonified allows the iPhone and iPad user to connect vision and sound in new ways, creating video and soundtrack at the same time. Record Sonified QuickTime movies with CD quality audio and share with friends and other Sonified users.

Why stop there?

Essentially, Sonified creates a real-time soundtrack to what our smartphone is pointed at. What happens when everything, every home, street, mailbox, person, lamppost, business, tree, rock, pet and everything else is connected to the web?

What data will they offer that, seen only by our smartphone, not our own eyes, will alter how we see, hear and perceive? 

What messages might people leave at every spot that stand in? Can we soon upload to the web, via a Twitter 2.0, say, that tells the world exactly how we feel?

Groggy and nervous, perhaps. 

What if a person does this while waiting at a bus stop? And the next person and the next? And before we even take our place in line, from at least 100 feet away, we know --because of our smartphone -- that the mood hanging over the bus stop is dark, angry, or maybe unexpectedly giddy?

How might that alter our interactions with those people and that place? How might it alter our interactions once our smartphone takes that real-time 'mood' data and combines it with all the historical mood data, traffic patterns, information on all the businesses in that area, criminal statistics, demographics of the neighborhood?

Our smartphones are fundamentally altering our interactions. Soon, they will fundamentally alter our perceptions.

More on the creator of this app:

Now a prolific multimedia artist and writer named Perry Hall [Flash required], who was born with his own version of Nabokov’s quirky gift, has developed an iPhone/iPad app called Sonified that enables even those low on the synaesthetic spectrum to experience light, colors, and movement morphing into sounds.

I first became interested in Hall’s work seeing a series of haunting HD videos made in 2006 that he called Material Study, featuring light dancing on the surfaces of ferrofluids that surge and swell like some kind of protean lava. While convalescing from a bout of Lyme disease, Hall decided that he needed to set his synaesthesia loose in the wild, as he puts it. He and his digital collaborators developed software that siphons the luminance and color values from the video cameras in iPhones and iPads (only later-generation devices like the iPhone 4, 4S and iPad 2 will work correctly) and uses them to trigger stereo samples from a library of CD-quality audio composed for the purpose.

The effect of the audio-visual-kinesthetic link-up was unexpectedly profound. Instead of feeling like Sonified was imposing its digital soundtrack on the world, I felt I was accessing a layer of reality that is normally hidden from us. It was like a little dose of Morpheus’ red pill in The Matrix.