Wolfgang Stuerzlinger's Research Projects

Human Computer Interaction (HCI), Virtual Reality (VR), and 3D User Interfaces (3DUI)

CoViD - Collaborative Virtual 3D Design (ISRG)

• Towards Better 3D Input Devices (ISRG) [TAS09a] [TAS09b] [TPSM09] [TPS08] [TS08a] [TS08b]
Common 2D input devices, such as the mouse, outperform most 3D input devices on frequently used tasks in 3D environments. This seems counterintuitive. One aspect of the problem is that simultaneous control of three degrees of freedom is more difficult for humans compared to just two degrees of freedom. Hence, most successful systems use 3D input only rarely, see e.g. the work on Guidelines for 3D User Interfaces, mentioned below. The other aspect pertains to differences in 2D and 3D input technologies. Examples for differences include variations in latency, jitter, the effect of co-location of input and output, as well as the existence of a supporting surface. To investigate this, we have performed a series of experiements that document the effects of each factor.
Part of this research is performed as part of the NSERC CREATE Program in Computational Approaches to Sensorimotor Transformations for the Control of Action.

• SESAME - Easy-to-Use Conceptual Design System (ISRG) [SDO06] [OSD06b] [OSD06a] [OSD05] [OS05]
This project investigates a new conceptual design system. The main goal of is to enable even naive users to quickly create and modify 3D content to communicate design ideas. SESAME (Sketch, Extrude, Sculpt, and Manipulate Easily) systems is based on the solid modeling paradigm and requires only a 2D pointing device. User studies have shown that naive users can quickly learn to use this system to generate interesting content. A comparison of SESAME with sketching on paper showed no significant differences in terms of creativity, but a comparison with a user group familiar with standard CAD tools clearly shows that SESAME encourages more creativity than current tools. Some Videos can be found on the video page. A demo version of the SESAME system is also available.

• User Interfaces for 3D Manipulation (ISRG) [TS08c] [TS08d] [TS07a] [TS07b] [TS07c] [OS05]
One of the problems of VR systems is that 3D input devices are still in their infancy. Even more importantly, the software technologies that map the raw user movements to 3D object manipulation are also immature. This is best seen by comparing the performance of intelligent algorithms for 3D manipulation in a desktop environment with a mouse with "traditional" VR manipulation techniques using 3D trackers. For many common tasks the mouse excels here. This project investigates new manipulation algorithms that improve performance for 3D manipulation tasks. We also investigate which properties of various input devices makes them well suited for 3D manipulation. The results of the compartive studies indicate that intelligent 3D sliding algorithms radically simplify and speed up 3D object manipulation. With such algorithms, we believe that manipulation with 3D trackers in general VR systems can be made to be almost as efficient as manipulation with a mouse in a desktop 3D modeling system. However, the performance of a mouse in a desktop 3D modeling system can also be improved with better algorithms, which is another avenue of current work.
Part of this research is performed as part of the NSERC CREATE Program in Computational Approaches to Sensorimotor Transformations for the Control of Action.
• Virtual LEGO - A 3D Construction System (ISRG) [BS06b] [OS04a] [OS04b] [OS03]
Lego blocks are a simple way to create 3D shapes. The Virtual Lego system introduces simple techniques to quickly create and manipulate Lego models. User studies showed that users without any 3D experience could quickly create 3D content with this system. Some videos can be found on the video page. You can also download a demo version of the Lego system. A user study of a haptics version of Lego has yielded interesting results, see the respective publication.

• MULTI - Collaboration on Large Display Surfaces (ISRG) [PS08b] [PS08a] [St05] [PS04b] [OS02]
Lasers can be used as an interaction device for large display surfaces. This project focusses on a new kind of laser-based input device, and is one of the few technologies which support multiple simultaneously active users. Evaluations show that laser pointers can serve as effective input devices for large screens. Another part of the research focussed on a state-of-the art evaluation of the performance of various remote pointing devices. A video demonstrating the multi-user aspect with a simple puzzle can be found on the video page. Our multi-user laser pointer input technology provides a basis for collaborative shared display groupware (SDG) and computer supported cooperative work (CSCW) applications. Beyond that we are currently completing a new kind of collaborative hardware setup designed to faciltiate collaborative work by teams of 2-10 people with interdisciplinary backgrounds. Seamless collaborative work in turn encourages creativity, which leads to better end products. The system is called MULTI (Multi-user laser table interface) and provides several fully interactive table and wall surfaces. Also, a collaboration with a Kinesology researcher about pointing movements has started.

Other Human-Computer Interaction projects

• The Effects of Technology on Pointing Performance (ISRG) [PS09b] [PS09a]
Many novel input devices have been presented for computer systems. Beyond standard technology such as mice and pens, there are also game console controllers (Nintendo Wii Remote, etc.) and many other approaches. While each technology has different ergonomic aspects, they also are based on different technical implementations. This project investigates the effect of technical factors such as delays (commonly called latency), variations in delay (i.e. time jitter), spatial jitter, and several other factors on pointing performance. The trade-offs between these factors that are documented through our work allow input device designers to make better choices for high-performance input devices.
A side project investigates how artifically introduced movement delays over interesting features, such as word boundaries, affect pointing performance in text selection.




• User Interface Façades - Towards Fully Adaptable User Interfaces (ISRG, together with O. Chapuis & N. Roussel from inSitu, Paris-Sud, France) [SOPR06]
This project presents a new technology that lets end-users adapt the user interface of arbitrary applications to their needs without resorting to coding. The user can select one or more widgets and drag them into other windows to create new GUIs (or drop them onto the desktop to create a new façade). Alternatively, users can replace widgets or change the mapping of mouse events to adapt any GUI to their own requirements and patterns of usage. The current version of User Interface Façades is based on the accessibility interface provided by most GUI toolkits and a novel GUI server. More information, videos, and source code, can be found on the User Interface Façades WWW page.




• Text Entry Methods, User Performance Modeling, and Cognitive Modeling (ISRG) [AS09] [DS08f] [DS08d] [DS07a] [PS05a] [PS04a] [PS03]
Current research in this project concentrates on models to predict text entry rates for novice users. All other models focus on experts only, which provides only information about peak speeds, which are often very unrealistic. Furthermore, this project investigates a new method to enter text on cell phones, called Less-Tap. User experiments have shown that Less-Tap is significantly faster than the standard method (Multi-Tap), and that it is competitive with dictionary-based methods (e.g. T9).

Recent work focuses on predictive models to simulate the transition of novices to experts. Another aspect focuses on error behaviours in text entry.




• Perception-Based Grouping (ISRG) [DS09b] [DS09a] [DS08a] [DS08b] [DS08e] [DS07b] [DS06a] [DS06b]
The direct manipulation of objects and efficient selection of objects is an integral part of modern user interfaces. Most systems support only rectangle selection and shift-clicking for group selection. In this project we investigate a new selection technique, which is based on the way human perception naturally groups objects.




• Novel Layout Mechanisms for Graphical User Interfaces (ISRG) [SS09]
The way various user interfaces elements (i.e. widgets) are placed on inside a window is described via layout mechanisms. This becomes particularly relevant, when the size of the window is changed, as the layout mechanism also incorporates the resizing behaviour. Commonly used layout methods are fairly simplistic and have their limitations. While there are very powerful methods to define layouts, the associated methods and programming interfaces are hard to understand and graphical user interface builders for such layouts are difficult to use.
This work investigates a new, easy-to-understand layout mechanisms and evaluates its implementation. Part of the work focuses on a new user interface builder systems that includes a novel form of preview window to illustrate the design choices immediately to the user, while still enabling easy access to all necessary parameters.

• Context-Sensitive Cut, Copy and Paste (ISRG) [KS08]
Creating and editing source code are tedious and error-prone processes. One important source of errors in editing programs is the failure to correctly adapt a block of copied code to a new context. This occurs because all semantic dependencies to the surrounding code need to be adapted in the new context and it is easy to forget some. Conversely, this also makes such errors hard to find. Our research investigates a new method for identifying some common types of errors in cut, copy and paste operations. The method analyzes the context of the original block of code and tries to match it with the context in the new location to find such errors.

• Enhanced Methods for Document Version Management (ISRG) [DS08c]
Comparing and selecting text from multiple versions of a document is a common task in collaborative scenarios. Even single users benefit from versioning facilities when working on a document. Text and diagram versioning methods are not well documented in the scientific literature, even though implementations of text versioning are abundant in commercial and non-commercial software. Our work presents several new methods for text versioning. We validated the results with user studies. Moreover, we present new techniques for diagram version management.

• Behavioural Training with Mobile Computers (ISRG, together with P. Ritvo)
This project investigates how a mobile computing platform can be used to help people to adhere e.g. to a diet. A new version of this system is currently in the works. It will be based on Web 2.0 services and feature support for offline access and data entry.
• Pen-based Computing (ISRG) [KS06] [KMS05]
Tablet PC's and personal digital assistants (PDA's) are becoming more and more popular. However, interaction techniques for manipulating objects in drawings/designs/diagrams are often based on ideas from mouse-based interfaces. We are performing research into steering motions and explore new techniques for the interactive selection and manipulation of arbitrary groups of objects. This will make work on large scale diagrams/designs/drawings easier.
The issue of zooming in user interfaces is also investigated in this project, in an collaborative effort with researchers at the University of Saskatchewan.

Other Virtual Reality projects

• IVY - Immersive Virtual Reality (VGR, CVR, together with M. Jenkin, R. Allison) [RLZ+02] [RLZ+01]
This project created a new kind of immersive Virtual Reality system: a six-sided CAVE, which is a room where every side (including the floor and the ceiling) displays computer generated imagery. The immersive device, called IVY, was completed in 2002. Novel aspects include a ventilation system for the enclosed space and a novel tracking system.




• 6 DOF Tracking (CVR together with R. Allison) [VSHA06] [VHS05]
The pose of an object in space can be described by 6 numbers, corresponding to 6 degrees of freedom (6 DOF). The Hedgehog is a new kind of 6 DOF tracking device, which features a large number of computer controlled laser diodes pointing outwards to project unique spots onto the walls as well as cameras outside IVY (or a CAVE) to track these spots. From these spots the position and orientation of the tracking device is computed in real-time. This in turn can be used to project the correct images for the current position of the users's head, which the tracking system is attached to. Translations can be tracked at least as accurately as current commercially available solutions. Rotations can be tracked 10 times more accurately than other systems. Hence, this technology greatly improves immersion in Virtual Reality and Augmented Reality systems. We are working to improve this technology further to make it more generally useable, depending on the availability of funding.

• Network Lag Compensation (ISRG together with R. Allison) [TAS07]
In collaborative Virtual Reality systems and networked games, it is necessary to transfer information about the state of the world between multiple systems. Such transfer is associated with transmission time-lag, and humans are reasonably good at dealing with a constant lag. However, freely accessible public networks exhibit significant variation in transmission lag due to the presence of unpredictable traffic flows. Such variations affect human performance very strongly. This makes public networks often unsuitable for real-time collaborative activities.
We recently presented a new predictive lag compensation scheme, which evens out these variations in lag in an optimal manner. Results show that a prototype implementation performs close to the theoretical optimum.

Computer Graphics projects

• HDR Systems - High Dynamic Range Video, Displays, and Projectors. (CVR, ISRG, together with others) [PS05b] [PVS05] [SHS+04] [SSV+03] [YS01d]
Real scenes and real photographic images exhibit a much larger dynamic range than current technology provides for. This project investigated how images with high dynamic range (HDR) can be acquired and how images with high dynamic range can be displayed on current hardware. One result is a system that can acquire HDR images at video rates. A collaborative project led by UBC and with G. Ward and other researchers at McGill and York University as well as several companies resulted in a new HDR display system (colorcoded HDR images by G. Ward, left: input data, middle: image of standard monitor, right: image of HDR display). The technologies were being commercialized by the startup Brightside Technologies. This company was recently acquired by Dolby.
My current research includes:
• New high dynamic range technologies, a first high-dynamic range projector was presented recently. [PS05b]. NEW: Additional images showing details of the HDR projection are available here.
• User interface issues of high dynamic range displays. [PVS05]

Other research interests

• Low-overhead database system (RZL)
• Fast access to compressed databases (RZL)
Two successful projects out of my commercial background. We are evaluating the performance of these database systems and comparing them with other implementations.

Inactive research projects

Human-Computer Interaction and Virtual Reality

• MIVE - Multi-modal User Interface for Virtual Environments (ISRG, VGR) [SS02] [SSS01b] [SSS01a] [SS01b] [SS01a] [SLS00a] [GS99]
The creation of object models for computer graphics applications, such as interior design or the generation of animations is a labor intensive process. Today's computer aided design (CAD) programs address the problem of creating single geometric object models quite well. But almost all users find common tasks, such as quickly furnishing a room, hard to accomplish.
This project investigates 3D interaction techniques that are easy to use, yet allow users to quickly construct 3D environments. The user interface are being evaluated with user tests. First results indicate that users take less than half the time with our new system.
Individual publications focus on the following issues:
• [SS02] describes the object group interaction techniques.
• [SSS01b] describes a detailled evaluation of the interaction techniques.
• [SSS01a] describes an evaluation with complex tasks (scene creation & modification).
• [SS01b] describes details of how constraints work in the MIVE system and discusses also the automatic creation of constraints
• [SS01a] describes an evaluation of the semantic constraints.
• [SLS00a] describes an attempt to integrate an intelligent assistant.
• [GS99] describes the first implementation of semantic constraints.
Videos can be found on the video page. There is also a demo version of the MIVE system.

Computer Graphics and Image-Based Modeling

• New Planning Methods for Image-Based Modeling (VGR) [PS04c]
Several commercial solutions exist for scanning of 3D objects. The result is a geometric model of the object. Although impressive results have been demonstrated, user intervention is still required to generate complete object models.
This project addresses the problem with techniques that check during acquisition for missing or badly sampled parts and direct the acquisition device to capture new views of such parts. The goal is to create an automatic acquisition system that can also be used in applications such as 3D faxing.
Furthermore, the research is targeted towards real-time acquisition, merging and planning. First results show that with such techniques it is possible to process even very spacious environments (e.g. a level of the popular game Quake) in reasonable timeframes.

• How many images are needed for Image-based Modeling? (IBR) [St99] [St98a]
Today many systems exist to generate geometric models of existing scenes and objects. One way to capture surface texture data is to record a series of images that, collectively, captures all visible surfaces of the object. Finding good viewpoints for this task is not easy. This project presents a new heuristic method to find a good set of viewpoints for a given geometric model. Taking images from the computed viewpoints will show every visible part of every surface at least once.

• Natural Phenomena
• Editing of Fractal Terrains (VGR) [SS05]
This project explored new ways to modify fractal terrains. A video can be found on the video page.
• Rendering Clouds (VGR) [ES01]
This project investigated new ways to render natural objects.
• Real-time Rendering
• Image-Based Rendering
• High-Quality, Real-Time Image-based Rendering (VGR) [PS06], Sergey Parilov's Master's thesis 2002, [PS02]
Image-Based Rendering (IBR) uses images to create new images. This new paradigm has demonstrated a lot of advantages over conventional computer graphicsi methods. Based on a novel visibility method, we created an IBR system that generates images of scenes with billions of samples at real-time speeds (>20Hz). Ultimately we investigated the trade-off between image quality and rendering speed by taking the capabilities of the human visual system into account. Some videos can be found on the video page.
• Real-Time Rendering of Penumbras (VGR) [BS06a]
Shadows significantly enhance the realism of images. This project presented a new method to geometrically compute area shadows (penumbras) in real-time.
• Massive Model Rendering (WALK) [ACW+99]
Visualization of very complex models in real-time (a 13 million triangle power plant @ 5-20 Hz). I designed and implemented the distant geometry replacement technique to ensure scalability (together with K. Hoff). The technique employs TDM's (Textured Depth Meshes), an image-based rendering primitive.
• Planar Reflections with Image-based Rendering techniques (IBR) [BS98]
• Hierarchical Image Cache (GUP) [SS96]
Scenes with very large polygon counts cannot be rendered in real-time on current graphics hardware. This paper presents an image-based rendering technique, where rendering effort is independent of polygon count. An image cache stores previously rendered images of parts of the static scene. Error bounds control the re-use of these images. Hierarchical combination of images provides scalability. An almost identical technique was developed independently by Shade et al.
• Rendering for Multiple Projectors and Multisurface Displays (IBR) [RCWS98b][RCWS98a]
The tech report includes a performance analysis.
• Interactive Rendering of Global Illumination Solution for Glossy Surfaces (WALK)[SB97]
This contribution introduces the first interactive display of a full global illumination solution of an environment with glossy surfaces i.e. surfaces that are neither diffuse nor perfectly specular. The method is best suited for low glossy surfaces found in many office environments.
• Computing LOD's - Geometric Approximations (GUP) [SS95a]
For faster rendering a new method to compute LOD's (Levels of Detail) is introduces which reduces the complexity of geometric models while preserving their appearance.
• Advanced Global illumination
• Computation of a Global Illumination Solution with Glossy Surfaces [St98c] (GUP)
• Optimized Local Pass [St96a] (GUP)
The visual quality of a displayed radiosity solution often suffers from deficiencies in the underlying mesh. The local pass technique re-computes the illumination at each visible surface point. This avoids visual artifacts but involves considerable computational effort. The contribution speeds the local pass by stochastically sampling only the most important contributors to the illumination of a surface point. One advantage of this technique is that it generalizes to non diffuse radiosity solutions, too.
• Exact Local Pass [SS95b] (GUP)
• Photo-realistic Rendering
Advanced Photon Map Techniques (GUP)
• Ray tracing
• Free-Form Surfaces (APM) [St98b]
Based on previous work together with W. Barth [BS93] this paper presents the first ray tracing method for triangular free form surfaces, which are becoming more common in CAD applications. Another important contribution introduces a compact and efficient description of complex trimming curves such as those created by combining objects described by free form surfaces. Furthermore, the paper discusses the basis for an efficient triangulation method that converts trimmed triangular free form surfaces to planar triangles.
• Optimizations (APM,GUP) [ST94]
An optimization for the traversal of bounding volume hierarchies which brings at least 50%. A speed-up method based on subdividing direction space.
• Radiosity
• Parallel Radiosity, Parallel Visibility, Dynamic Load Balancing (GUP) [SSV95b][SW94c][SW94a]
These publication presents a massively parallel method to compute radiosity solutions on distributed memory machines with hundreds of processors. The key method is a distributed visibility computation technique that consumes less bandwidth compared to other approaches. An efficient dynamic load balancing technique is another important aspect of the presented approach.
• Adaptive Discontinuity Meshing (GUP) [St94b][St92b]
• Radiosity with Voronoi Diagrams (APM) [St92a]
• Point Clouds for Bounding Volumes (GUP) [St96b]
• Hemispherical Projections (GUP) [St95a]
• Object oriented rendering systems (APM, GUP): Flirt [STS93], FXFire [St93], Generic Interfaces [St96c]

Computer Vision

• Automated training of vision systems (GUP) [BBS95]
Faster training of a robot by simulating the images it sees during navigation.

Companies and Start-Ups

• Brightside Technologies, recently acquired by Dolby.
• Interactive Health
• RZL Software GesmbH

News Coverage

Funding

• NSERC
• CRESTech
• CFI
• ORDCF
• IRIS
• CANVAC National Centre of Excellence
• FWF
• BMWV
• Alias|Wavefront
• MD Robotics

List of Research Labs

• ISRG Interactive Systems Research Group at the Dept. of Computer Science and Engineering, York University in Toronto, Canada (formerly HCIlab).
• VGR (Vision, Graphics and Robotics) group at the Dept. of Computer Science and Engineering, York University in Toronto, Canada.
• CVR (Centre for Vision Research) at York University in Toronto, Canada.
• WALK (Walkthrough) group and IBR (Image-Based Rendering) group at the Dept. of Computer Science, UNC (University of North Carolina) in Chapel Hill, USA.
• GUP (Computer Graphics and Parallel Processing) group at the Institute of Telematics and Technical Computer Science, Johannes Kepler University in Linz, Austria.
• APM (Algorithms and Programming Methodology) group at the Institute of Computer Graphics, Technical University of Vienna, Austria.
• RZL Software GesmbH - software for tax consultants in Ried, Austria.