Meet Pranav Mistry, digital SixthSense inventor!

Twenty-nine-year-old Pranav Mistry may be a household name in research and scientific circles across the world, but memories of building sandcastles with his grandparents on Chowpatty Beach can still get him excited.

Today we're sitting a world away from the cluttered delights of that popular hang-out on India's west coast, but just a few blocks away at a cafe from Mistry's brainstorming hideout -- the Media Lab at the Massachusetts Institute of Technology (MIT), Cambridge.

We're sipping flavoured iced tea, not my favourite tipple, I must admit, but it's worth the 'suffering' just to catch up with the creator of 'SixthSense' digital prototype.

Mistry, though, carries his fame with the ease of a grey-bearded veteran. I had conducted a telephonic interview with him about a year ago from India and was looking forward to meeting him in person. I found that I didn't have to do much to break the ice.

(Please browse through to the end of the slide show for a video of the SixthSense magic!)

The reference to sandcastles is provoked by my disclosure that I am a Mumbaikar on sabbatical. Although he was brought up in Gujarat, he says, he also studied at IIT Bombay. "Every summer afternoon, I would unfailingly spend time with my grandparents and we built sandcastles on Chowpatty beach," he recalls.

. . . 

Those make-believe castles could well have been the real thing had Mistry followed his father Kirti's footsteps and become an architect. In fact, he began his career at the Centre for Environmental Planning and Technology (CEPT) University in Gujarat after standing first in the All India Entrance Examination.

As it turned out, he's something of an outlier in the Mistry family. "All my relatives are architects. I'm the only one who has pursued technology as a career. At all social gatherings, I'm the only geek surrounded by my numerous architect relatives," he says, sipping an orange-flavoured iced tea through a straw. I choose the same; it does nothing to enhance my opinion of iced tea but luckily chatting with Mistry makes up for it.

So why did he abandon architecture? "I realised that I had started thinking design like my father. But I wanted to make my own designs, make mistakes and learn from them. I wanted to chart my own course." So, he gave up a career in architecture and branched out into technology -- his next passion.

Was his father perturbed? "Not at all. In fact, he was happy that I was coming into my own. After all, it's my father who taught me that it was okay to make mistakes. And my father loves technology too."

Kirti Mistry, a JJ School of Architecture graduate, cut no corners to encourage his son's penchant for gadgetry -- even if it meant buying an expensive Commodore 64 (together with IBM-compatibles, the C64 8-bit computers were among the earliest home PCs and were introduced in 1982) or even building a prototype that served as a ping-pong video game. "These were my initial creative learnings -- to rip apart every electronic gadget and understand their innards," Mistry recalls with a grin.

. . . 

Having chosen technology, Mistry did his Master's in Design in Visual Communication from IIT Bombay. "It was here under the tutelage of Ravi Poovaiah that I learnt how functional design should not distract. In other words, you may see the lack of design but functional design should not be noticeable," he explains.

It was during his IIT days that he was spotted by Srini Koppolu, then managing director of Microsoft India Development Centre (he resigned this month). After IIT, Mistry worked in the Hyderabad-based outfit for around four years -- first as an intern, then as an employee.

"I worked on several projects here including Akshar, which was basically an attempt to create a mechanism for inputting Indic scripts in digital devices like mobile phones, kiosks, interactive TVs or personal computers. I use this to talk to my mother," he says, explaining that data-entry methods for Indian languages like Hindi, Gujarati and in general are not keyboard-friendly and using the standard QWERTY keyboards for this is complicated.

But Mistry wanted greater creative challenges and sought admission as a PhD student at MIT. "I was clear that I wanted the 'Fluid Interfaces' department and to work with one professor -- Pattie Maes," he says. Maes, an associate professor in MIT's programme in media arts and sciences, founded and directs the Media Lab's Fluid Interfaces group.

It was under Maes's guidance that Mistry created the 'SixthSense' device, which MIT later patented, to help him get closer to his childhood dream of melding the flexibility of the digital world with the physical one.

. . . 

The device comprises a pocket projector, mirror and web camera bundled in a wearable pendant-like gadget. The projector can turn anything into a touch screen. The webcam (and colour-coded finger-gloves worn on the index finger and thumb) can recognise the movements of a user's hands, which enables gesture-commands. A "square frame" gesture, for instance, will prompt the device to take a photograph.

The device can also recognise a book that the user selects from a bookstore -- either by image recognition or radio frequency identification (RFID) tags -- and project information, like an Amazon rating, onto it. The system can also project a keyboard to type on, detect items on grocery shelves and compare online prices.

A newspaper can prompt the device to search for news video clips (the device's smartphone uses an internet connection to retrieve information). "The possibilities are immense but it's a work in progress," Mistry says.

Cold tea is not conducive to a long conversation and my glass is almost empty but I pretend to draw on the straw, the better to prolong our chat.

For Mistry, the 'SixthSense' device is one of several such projects.

. . . 

For instance, one of his earlier projects, 'Sandesh' attempts to bridge the digital divide. It contains a message-receiving unit in villages and kiosks in cities with visual aids. It uses print- or sound-based media to convey messages.

And his 'mouseless' technology does away with the need for an external computer mouse (the instruction video starts with his favourite 'Tom & Jerry' cartoon characters showing the mouse spilling ink on his body to become invisible).

The device consists of an infrared (IR) laser beam and an IR camera -- both embedded in the computer. Users cup their hand, just as they would do if a physical mouse were present underneath, and the laser beam lights up the hand that is in contact with the surface.

The IR camera detects the bright IR blobs using computer vision. The change in the position and arrangements of these blobs are interpreted as a mouse cursor movement and . . . 

 "I thought of that option. It's logical. But I believe I still have five or six years of creativity left in me," he says, adding: "But I might branch out into something else eventually. Don't act surprised then."

Mistry has been thinking about using his experience in film-making. He has had 'friendly discussions' with Avatar director James Cameron, and there are other directors who are 'interested' in his technology.

Meanwhile, he has added many feathers to his cap. The Obama administration, for instance, invited him to pick his brains on technology-related issues and the Chinese and Canadian governments have solicited his help in technology-related fields.

"The Chinese government wants me to set up a research lab in their country and is offering a very neat sum for research," he says, but asks with more than a hint of disappointment: "Why can't India follow suit? Why are there always more words than action?"

That is, of course, a rhetorical question. We've both run through our teas. There's so much more to talk about but we've also run out of time, so we shake hands with the promise to catch up soon.

The author, on a sabbatical from Business Standard, is an MIT Knight Science Journalism Research Fellow 2010-11.