Building on the algorithms I’ve implemented as part of my Diplom Thesis, I’ve now implemented a demo of a 3D view on the Nokia N900. The idea is to adjust the camera through which the 3D scene is viewed to the user’s head position. The screen then appears to be a window into the virtual world.
The 3D scene is easily constructed on the N900 using OpenGL ES 2.0. A small box and a few foreground objects are sufficient. In order to determine the head position I use the face detection and tracking code I’ve recently implemented (and already used for a little tech demo). The two things are easily put together and the result is this:
The method is not new:
- It was first publicized by Johnny Lee in 2007 using a Wiimote.
- It was recently used in a Nintendo DSi game where hidden objects have to be found.
The most difficult thing is to get the perspective right as there is a lot of conflicting information out there. Johnny Lee used the Direct3D function PerspectiveOffCenterLH, which is obviously not available in OpenGL. The best hint I found is here. There projection matrix M should look like this:
Where n is the distance to the near clipping plane, f the distance to the far clipping plane, t and b define the horizontal clipping planes, l and r define the vertical clipping planes.
I think the effect is pretty cool and I’m currently contemplating making a game with it.
Tags: 3d, detection, face, front camera, head, n900, nokia, opengl es 2.0, thesis, tracking
April 28th, 2010 at 22:35:25
Social comments and analytics for this post…
This post was mentioned on Twitter by j66k: 3D view on the Nokia N900 through head tracking http://bit.ly/9O7G3A #n900 #maemo…
April 28th, 2010 at 22:37:01
that is just wicked! does this mean I can get it on my N900 without the need for any peripherals? will u put an installable demo somewhere? my friends will go nuts if they see this
- written on my N900
April 28th, 2010 at 22:49:46
This is awesome. Well done
Not sure if this is possible, but you *could* try using the accellerometer data to try and smoothen out the motion which is slightly chopy. It would no doubt be slightly complicated but it might be possible and help?
April 28th, 2010 at 23:19:52
Thanks!
Yngve: Yes, is works solely by using the N900’s front camera. I don’t have a demo ready (yet), but that might change sometime in the future.
Mike: The choppiness is a result of the face detection that can be inaccurate at times due to lighting conditions and the poor quality of the N900’s front camera. I don’t see how the accelerometer data would help with that problem. While recording the video the device wasn’t moved.
April 28th, 2010 at 23:41:47
Ah yes, of course
I meant when you stay stationary and instead tilt the device left/right/up/down. Then you’d have some spatial data on how the device is situated in relation to the head (which you could still be tracking).
April 29th, 2010 at 01:51:24
[...] Dieser Eintrag wurde auf Twitter von Skyler, Vincent Lefèvre, Skyler Visconti, MeeGoPortal, Yngve I. Levinsen und anderen erwähnt. Yngve I. Levinsen sagte: RT @jfk: 3D view on the Nokia #N900 through head tracking http://bit.ly/9O7G3A #maemo [...]
April 29th, 2010 at 02:56:23
Man, very very very cool, would you let us download this demo !? I’m from Brazil, I have a N900 and I found your project amazing ! congrats ! keep the excelent work !
April 29th, 2010 at 10:00:48
Yeah, a tech demo/binary or something would be very cool, the source code even more so
April 29th, 2010 at 10:16:56
Wow that is great!!!
This should be implemented in every game
What kind of game do you have in mind for this?
Can’t wait to try out a little tech demo!
April 29th, 2010 at 11:51:33
wow!!!!!!!!!!!! this has so much potential!!
April 29th, 2010 at 11:55:15
This is truly awesome
As soon as you have an installable demo, I will try it out straight away, just to get impressed and of course impress others
Keep up the good work.
April 29th, 2010 at 12:48:29
[...] Fonte [...]
April 29th, 2010 at 14:16:32
Maybe you should try the option to use the back camera and put the display on a nice big 50″ screen!
That way the 3d will be more immersive and the camera tracking more accurate…
April 29th, 2010 at 14:17:07
Fantastic, congratulations!
April 29th, 2010 at 23:34:39
Thanks everyone!
Mike: Now I get the idea. That might work, but it would restrict the usage to holding the device in your hands. Also, the algorithm would probably have to be calibrated as different positions might be possible.
Lennart: Well, the easiest game would be something where you have to pick up certain objects and dodge others. The technology is probably best suited for small/casual games.
toucan: Yes, that would also work. But then you can also directly use a normal computer with a webcam.
April 30th, 2010 at 04:35:24
Wow, THANKS for developing this! I was hoping to do something like that on my graduation project (diplom thesis) which will be next year. Mind sharing some info or some ideas to improve this? Maybe I could enhance your code, instead of rebuilding the wheel.
April 30th, 2010 at 07:18:19
i thought i was dreaming about this. But this is real…. Thank you so much. Ps my N900 is happy
April 30th, 2010 at 13:10:03
[...] Fuente: http://johanneskuhlmann.de/blog/2010/04/28/3d-on-the-nokia-n900-through-head-tracking/ Únete a nuestra página en Facebook. [...]
April 30th, 2010 at 17:31:06
WoW!! WoW!! Waiting eagerly for your game!!
May 1st, 2010 at 02:47:49
A tunnel shooting game would be Epic!!!
May 1st, 2010 at 10:09:49
Hey
For web browsing on small screen this feature would be great! You could easily
scroll the screen.
-T
May 1st, 2010 at 18:46:11
Amazing!!
Download please?
May 1st, 2010 at 20:40:18
[...] Props to http://johanneskuhlmann.de/blog/2010/04/28/3d-on-the-nokia-n900-through-head-tracking/ [...]
May 1st, 2010 at 22:08:56
Honestly I think just a game is thinking a little small. You could in theory create a development platform where people could code around your engine. Of course if you were in it for the money you could always charge for access and use of the platform engine. I may be a huge fan and advocate of open source. but I’m also realisic in my beliefs that if there isn’t money involved work will be slow if at all. I’m really looking forward to progress on this. Gj
May 3rd, 2010 at 18:16:11
Very nice demo. For smoothing you might want to look into double exponential smoothing – Joa LaViola has a nice paper on the application here. It’ll be faster than a Kalmen filter and will probably get you anything you need. Let me know if you try that out.
In terms of the accelerometer, you might actual consider that if you ever plan to pick up the device. Right now, you’ve got it in a very stable location sitting on the table. Picking it up will introduce new errors that you’ll want to smooth out somehow.
May 5th, 2010 at 01:57:24
Very cool! Are you tracking the head in full 3D, including distance from the N900, or just left/right/up/down? will this tracking code ever be available for others to use?
May 8th, 2010 at 12:04:53
great great great,
i want this on my phone, how?
May 10th, 2010 at 15:35:48
Thanks again everyone!
Spidey: I’m not sure yet if I want to share my code. Sharing my thesis (on which this is based) itself might also be difficult because it’s in German only.
I’d be happy to share any info you need though. Just write me an e-mail (see the contact area of this website).
Wake: Interesting thought. Someone should test how well this approach works for different kinds of games.
sean: There is no Kalman filter involved if that’s what you want to say. Right now I’m just taking the average of the last three detections. When I have some time I might look in that exponential smoothing.
blair: I’m also able to roughly determine the face’s size (which obviously corresponds to the distance), but my tests showed that the size detection wasn’t stable enough. So I didn’t include it in the demo.
May 14th, 2010 at 15:06:27
Plase share this demo on maemo.org! or just add a download link
love it!!! it’s great!
May 15th, 2010 at 04:47:12
You are a genius
.. its fantastic .. i hope u publish the demo online soon so we can try it on our N900 
its greaaaaaaaaaaaaaaaaat 
and thank you
May 17th, 2010 at 19:41:52
[...] Johannes’s Blog This entry was posted on Tuesday, May 18th, 2010 at 1:41 am and is filed under HI-Tech. You can [...]
May 22nd, 2010 at 10:43:52
How does this work with two eyes? Do you take the average position to yield a view slightly off for each eye? Does it look more realistic when closing one eye, or is the difference negligible?
May 30th, 2010 at 00:29:42
WOW.. I saw this on TED once, I would love to play with this and perhaps do more.
..
Demo please.. please. please..
June 26th, 2010 at 12:06:08
Dude that is likeee the most awsome thing i have ever seen….
Please dude share the program….
It has been three months and i am dieing to see this on my n900….
Nokia made the n900 compatable so that every one could enjoy it….
and explore it their way…
Thanks…….
August 2nd, 2010 at 07:29:53
Make a clone of Curveball where you gotta hit the ball with your face!
August 21st, 2010 (2 weeks ago) at 21:23:11
There is a downloadable version of this project? It’s very urgent!! Please…