• Subscribe

    Subscribe to This Week In Panospace by eMail.
    Subscribe in a reader
  • License

    Creative Commons License
    This work is © 2008-2012
    by Yuval Levy
    and licensed under a
    Creative Commons License.
  • Entries

    October 2017
    M T W T F S S
    « Dec    
     1
    2345678
    9101112131415
    16171819202122
    23242526272829
    3031  
  • Archives

The Ergonomics Of Panoramic Interactions Continued

TouchShield SlideBruno’s comment about touch-screens got me thinking. While most users still interface with the computer via mouse, keyboard and a one-way display, things are going to change fast in the coming years. The old KVM (Keyboard/Video/Mouse) user interface is being replaced by more powerful and natural tools. The point&click / drag&drop metaphors popularized by Apple’s Macintosh since 1984 after the invention of the ball mouse at the Xerox Parc are due for an update. Ever smaller and powerful mobile devices, accelerometers, touch screens, 3D screens. How will they interface between the user and the VR Panorama?

The solutions I have observed so far simply hard wire the behavior of these new devices to the mouse. This is no different than my 1998 Wacom Tablet (which still works!). The simulation of the mouse limits the interaction designer to define the device in relationship to mouse behavior and either mimic it or its inverse.  After half a century it is time to break those limits; to look at the interactions anew and to design device/context specific metaphores; to mold the intearction around the human. I see a combination of relevant factors, including the device’s physical characteristics and the context in which it is used. A touch-screen on a desktop requires a different metaphore than one on a smartphone. And what to do when two competing input devices with conflicting metaphores are attached, such as an accelerometer (3D mouse) and a touch-screen?

For the desktop touch-screen and for the laptop touch-screen I tend to agree with Bruno that the Google StreetView metaphore is the way to go, at least until the computer can discern if the index finger is at a nearly perpendicular angle and fully straightened as in a pointing; or it has a smaller angle and a slightly more curved posture as in a natural dragging movement.

Things become more touchy (pun intended) with mobile devices which typically have an accelerometer and a touch-screen. Which one should drive the VR panorama interaction and how? To me the most natural would be to use the accelerometer and point the iPhone in the direction I want to see, but in some situation such explicit movements are embarassing, unconvenient, inappropriate, or all of the above; and the more discreet dragging by finger on the touch-screen is the right way to go.

Whichever it is, I think modern panoramic viewer should make provisions to accommodate both behaviors – dragging and pointing. Ideally the system would tell the VR player in what context it is playing and the VR player would adapt, using an appropriate metaphores. Currently the browser only let the VR player assume the presence of a pointing device, and the devices all interface by mimicking the mouse. In the current situation making the mouse behavior a parameter, as implemented in the KRpano viewer, is the best thing to do. When a reliable detection mechanism can tell the player what device is attached, the choice may be automated.

I look forward to see the results of León’s Google Summer of Code project adding QTVR playback and Wiimote interaction capabilities to the VLC media player. In the meantime I got help from the Liquidware guys in my still unsuccessful attempts to make their Antipasto Arduino IDE work on my Ubuntu notebook. The TouchShield Slide touch-screen rocks and I am keen to toy on new interfaces with it.

Accepted Students for Google Summer of Code 2009

MascotteThe dice have rolled. For the third year in a row the Hugin community will mentor five Google Summer of Code students as they go about writing free software:

  • James Legg of Fast Preview fame will tackle the Layout Panorama Model mentored by Bruno Postle, improving the handling of stacks.
  • Tim Nugent of Celeste fame will tackle Straight Line Detection for Lens Calibration mentored by Tom Sharpless, improving the handling of fisheye lenses.
  • Lukáš Jirkovský, too young in 2008, has become a significant contributor to our community over the past year and will tackle Ghost Removal for Enfuse, mentored by Andrew Mihal.
  • Dev Ghosh will add Mosaic Mode to Hugin/Panotools, mentored by Daniel German.
  • Last but not least Yulia Kotseruba will add functionality like multiple blending to Lighttwist, mentored by Sébastien Roy.

I am particularly proud of the collaboration with Sébastien Roy on Lighttwist. First, because I believe there is a cultural fit between his Vision3D laboratory and the Hugin community. Second, because it is good to see academics releasing the result of their research (which, let’s not forget, is often funded with public taxpayer’s money) to the general public under an Open Source license.  And last but not least, because I think that Lighttwist is the natural extension to Hugin. I would dare to advance that Lighttwist is to panoramic photography in 2009 what QuickTimeVR was to it in 1999. In a private email exchange Ken Turkowski, inventor of QuickTimeVR and one of our team, acknowledge the analogy.

Then there is a sixth project that I care about. We have teamed up once again (and we hope to be luckier this time) with VideoLAN to bring QTVR playback to this popular media player. This time with an extra bonus: Wiimote control. León Moctezuma, who in 2007 added SPi-V playback to FreePV, will build on his expertise of FreePV and add the Wiimote on top of it, mentored by Antoine Cellerier of VideoLAN.

What’s next for the students? this article I wrote exactly one year ago applies.

Good Luck, students, I hope to congratulate you all in a few months.

Apple’s QuickTime Update Breaks Detection

Recently Apple updated QuickTime. I received the update on my Windows box via Apple Software Update and I agree with John Lilly, Mozilla CEO, when he blasts Apple. It undermines the trust relationship great companies have with their customers, and that’s bad – not just for Apple, but for the security of the whole Web”. Indeed, it tried to sneak once again iTunes on my machine, despite me repeatedly checking the “ignore this upgrade” box.

The latest QuickTime update also undermines some of the plug-in detection code used to find out if a visitor has QuickTime in the browser, including in Apple’s own Safari. I found out about it the hard way last weekend at the first Toronto Panoheads Meeting. Lucky me I could divert my talk and talk about the World Wide Panorama instead.

Plug-in detection is critical to the proper display of plug-in dependent content such as VR. Trying to display a VR using a plug-in that is not available on the client results in awful degradation.

Detecting the availability of the plug-in is the first step in graceful degradation: if the plug-in is detected, chances are that the content will display properly. If it is not detected, the website can degrade gracefully and show the visitor a static image or a link to download the relevant plug-in.

There are two ways to detect the plug-ins: ActiveX probing (for Internet Explorer) and parsing the navigator.plugins array (for all other browsers). In Firefox, it is possible to display the content of the navigator.plugins array in a user readable format by typing about:plugins in the address bar:

Until recently, QuickTime’s version was listed in the form of Major.Minor.Revision. Now they seem to have moved to Major.Minor (Revision) – and some parsers will choke on that. Mine did.

I not only detect QuickTime, I also try to discern the real QuickTime from the many third party plugins that set their name so to fool the detectors and return positive to detection, like Totem or VLC. Hence, I check with a strict syntax, and when Apple breaks the syntax, my detection returns a false negative.

False negatives are IMHO better than false positives. The reason why third-party plug-ins want to fool the detector is because they think they can play QuickTime content. It may be true for audio/video and other linear content. Unfortunately it is not (yet) true for VR, objects or panoramas. Discerning whether the original QuickTime is installed or a copycat, makes the difference between a gracefully degraded website or one showing a broken movie.

Fixing my detection code was quick and all websites using it have been updated automatically. Of course it would be nicer if Apple would stick to its own conventions in the first place.

When the dust settles

It’s a few week before the official coding starts. The students are bonding nicely with the community. Ideas are flowing, and some patches too. I can’t wait to see what they will do when they are officially on Google’s payroll and assigned to work full time on hugin.

While Pablo and Alexandre P. are at LGM, time to introduce properly this year’s team that will participate in the Google Summer of Code. There is a lot to be written, and more may come over the next few months as the summer unfolds. For now, here is an overview of the six projects we’re working on, of the people, and of exciting things to come.

The Projects

There are six projects this year in our portfolio, even though only five are listed on the Hugin/panotools entry at Google. The sixth project is a joint effort with VideoLAN on their leading cross-platform media player. It is listed on their page at Google.

OpenGL hugin preview

The preview windows is central to the panorama making process. It is here where the author looks at his composition before rendering it. It is here that framing decisions are being made, interactively. Right now the interaction is slow.  James Alastair Legg, mentored by Pablo d’Angelo, will improve the panorama preview windows by giving it the speed of OpenGL. We expect near real-time interaction for the author when composing his panorama.

Automatic Feature Matching for Panoramic Images

Critically important to stitching panoramas is to identify overlapping images in two features and align them in space. In the past, this was tedious, manual work. Then control point detectors came along. Those available to hugin users are still tainted by patents. It’s a two step process: detecting features and matching them. In Google Summer of Code 2007 Zoran Mesec wrote the detector, MatchPoint. This year Onur Küçüktunç will write the matcher, mentored by Alexandre Jenny, author of the original autopano and of Autopano Pro. A beautiful example of cooperation and co-existence between the proprietary and open source world. We expect to have a fully patent free control point detection process.

Masking in GUI

Overlap regions between two images are the nature of stitched panoramas. Since the world is in movement, such overlaps often present challenges that won’t match. The current solution is to render each image on a separate layer, and then mask out manually one of the two images so to display only one frozen instance of the moving object. This can often be a painful work at the single pixel level. Fahim Mannan, mentored by Daniel M. German, will introduce a simpler workflow: just mark the moving object with a couple of approximate brush strokes and his code will work out the exact object boundaries automatically. We expect an improvement for those using hugin to stitch action panoramas.

Batch Processing

Photographers often come back from the field with tons of photographs to stitch. A lot of this could be automated. Even more so with the up and coming pano-videography. Marko Kuder, mentored by Zoran Mesec, will improve hugin’s batching abilities. We expect to be able to process repetitive tasks without human intervention.

Machine-based Sky Identification

Some areas of photographs are better suited for control points than others. The sky, with its moving clouds, definitely not, as good control point don’t move between images. Timothy Nugent, mentored by Yuval Levy, will train a support vector machine (SVM) to identify clouds in the sky as a bad area for control points, so that it can be masked out before triggering the control point detection. Once working the method can be extended to other features as well, such as foliage and water.

Panorama Viewing in VideoLAN

19 years after Tim Berners-Lee invented the web there is still no universal format to view panoramas on the web. Apple’s QuickTimeVR, the original technology to display full spherical panoramas, is not available on platforms other than Windows and the Macintosh, and it is no longer developed by Apple. A lot of good things have happened with Flash panoramas in the last year. Nevertheless, a lot of legacy content out there is captive of the QuickTime format, like the World Wide Panorama. In Google Summer of Code 2007 Leon Moctezuma added functionality to FreePV. This year Michael Ploujnikov mentored by Yuval Levy will integrate panorama viewing in VLC, a leading cross-platform media player. We expect Linux users and users of other alternative platforms to have access to the majority of QTVR content soon.

The team

I’m happy to pass the admin role to Alexandre Prokoudine this year. We had more available mentors, student applications and project ideas that we would have loved to follow through, but resources are limited also for large corporations and Google is already very generous with us. We would have loved to see students working on leading edge image processing under the supervision of Andrew Mihal of enblend/enfuse fame, or John Cupitt of VIPS. Maybe next year. Other mentors that registered with our organization on the Google Code page and are left without student are Bruno Postle, Jim Watters and Ken Turkowski. We are a team, and like last year I expect a lot of help and community support to the six lucky students.

Partners

Cooperation is a topic I particularly care about for this edition of the Google Summer of Code. We are leveraging the Summer of Code to reach beyond our small world. I am proud that we found an ally in the VideoLAN team, a larger mentoring organization. Granted, we are natural allies: hugin/panotools is used to create media; and VLC is used to play it. Nevertheless, this cross-collaboration, whose idea is the result of a meeting with Jean-Baptiste Kempf at the 2007 Google Summer of Code Mentor Summit, is IMO a demonstration that the whole is worth more than the sum of the parts and that an initiative like the Google Summer of Code adds much more value to the world of FOSS than what can be stated in (highly appreciated) numbers. 175 mentoring organization, 1125 students, 90 countries.

And in our small world we’re working a partnership deal to further motivate and propel the hugin/panotools team, similar to what we did last year. Stay tuned for an announcement.

World Wide Panorama

The latest World Wide Panorama event is online. I did not have much time to process my entry, an objectVR movie.

Unfortunately the above site requires QuickTime, and that QuickTime is not available for Linux. There are plenty of media players that play linear QuickTime content, but support for QTVR – cylindrical and cubic panoramas as well as object VR movies – often falls short.

Last year the FreePV viewer was a Google Summer of Code project. This year I hope that we can take it further with the help of the team at VideoLAN. The plan is to integrate it in VLC, one of the most popular free media players out there, and my favorite on both Windows and Linux.