Biography

Vincent Lévesque is a co-founder and member of Tactile Labs (Montreal, Canada). He was a Senior Research Scientist, Research Scientist II, and Postdoctoral Research Scientist with Immersion Canada (Montreal, Canada) from 2011 to 2016. He was a Postdoctoral Fellow in the Computer Science Department at the University of British Columbia (Vancouver, Canada) from 2009 to 2011. He received a B.Eng. in Computer Engineering (2000), and a M.Eng. (2003) and PhD (2009) in Electrical Engineering from McGill University (Montreal, Canada). His research interests include interaction design with novel haptic interfaces, tactile displays and rendering, and applications of haptics for persons with visual impairments. He is the recipient of several awards including Best Paper Awards at the 2012 Haptics Symposium and the 2011 ACM CHI Conference for his work on touch interaction with programmable friction, the Best Reviewer Award at the 2010 Haptics Symposium, the Best Demonstration Award at the 2008 Haptics Symposium for his work on dynamic tactile graphics, and a Best Paper Award at the 2007 IEEE World Haptics Conference for his work on refreshable Braille.

Research

Enhancing Touch Interactions with Programmable Friction

This project investigates the benefits and outcomes of augmenting touchscreen interactions with programmable surface friction. Using a technology developed by Ed Colgate and his team at Northwestern University, we vary the coefficient of friction at the surface of a touchscreen to create haptic effects that enhance the sense of realism, engagement, and enjoyment experienced by users.

Selected Publications:
      CHI'11 (Best Paper Award), HAPTICS'12 (Best Paper Award)

Haptics for Persons with Visual Impairment

My work on laterotactile stimulation brought me to consider the broader needs of visually impaired persons and the ways in which technology - particularly haptic technologies - could improve their quality of life. In an effort to share my findings, I am writing a survey that I hope will serve as a primer for haptics researchers interested in applying their research to visually impaired persons. An early version of the survey was published as a technical report.

Selected Publications:
      PhD Thesis, Technical Report

Tactile Graphics

The main topic of my Ph.D. thesis was the display of refreshable tactile graphics by laterotactile stimulation. I have developed rendering methods that produce tactile patterns with vibration, an undulating texture and small dots.

Selected Publications:
     PhD Thesis, EuroHaptics'10, ASSETS'08, HAPTICS'08, CHI'06

Learn more on laterotactile.com:
     Tactile Graphics, Haptic Memory Game

Braille

This project investigated the feasibility of using laterotactile stimulation for the refreshable display of Braille. As a first step, we implemented a 1D laterotactile display and showed that it could be used to produce a line of Braille dots. We more recently extended this work to the display of complete Braille cells with the STReSS2, our lab's latest 2D laterotactile display.

Selected Publications:
      PhD Thesis, WHC'07 (Best Paper Award), TAP 2005

Learn more on laterotactile.com:
     Virtual Braille, Virtual Braille Dots

Mobile Interaction

I played a supporting role in a project that looked at ways in which laterotactile stimulation can be used in a mobile context. My colleague Jerome Pasquero, in collaboration with Joseph Luk and Karon Maclean of the SPIN lab at the University of British Colombia, integrated a miniature laterotactile display into a PDA-like device and is now looking at how the novel feedback it allows can be used to improve interaction with such devices.

Selected Publications:
      Trans. Multimedia 2007, CHI'06 (Best Paper Award)

Learn more on laterotactile.com:
     Haptics for Mobile Interactions

Measurement of Skin Deformation

The research topic for my Master's thesis was the measurement of skin strain by tracking of anatomical landmarks of the fingertip. Techniques borrowed from the field of biometry were used to extract fingerprint features from movies of a finger sliding over a transparent surface (with or without simple geometrical features.) Variations in skin strain were measured by observing changes in inter-feature distance over time. 

Selected Publications:
      J. R. Soc. Interface 2013, J. R. Soc. Interface 2011, EuroHaptics'03

click here for more details

Publications

Journals

Finger pad friction and its role in grip and touch
Michael J. Adams, Simon A. Johnson, Philippe Lefèvre, Vincent Lévesque, Vincent Hayward, Thibaut André, and Jean-Louis Thonnard
Journal of the Royal Society Interface, 10(80), 2013.

Abstract

Many aspects of both grip function and tactile perception depend on complex frictional interactions occurring in the contact zone of the finger pad, which is the subject of the current review. While it is well established that friction plays a crucial role in grip function, its exact contribution for discriminatory touch involving the sliding of a finger pad is more elusive. For texture discrimination, it is clear that vibrotaction plays an important role in the discriminatory mechanisms. Among other factors, friction impacts the nature of the vibrations generated by the relative movement of the fingertip skin against a probed object. Friction also has a major influence on the perceived tactile pleasantness of a surface. The contact mechanics of a finger pad is governed by the fingerprint ridges and the sweat that is exuded from pores located on these ridges. Counterintuitively, the coefficient of friction can increase by an order of magnitude in a period of tens of seconds when in contact with an impermeably smooth surface, such as glass. In contrast, the value will decrease for a porous surface, such as paper. The increase in friction is attributed to an occlusion mechanism and can be described by first-order kinetics. Surprisingly, the sensitivity of the coefficient of friction to the normal load and sliding velocity is comparatively of second order, yet these dependencies provide the main basis of theoretical models which, to-date, largely ignore the time evolution of the frictional dynamics. One well-known effect on taction is the possibility of inducing stick–slip if the friction decreases with increasing sliding velocity. Moreover, the initial slip of a finger pad occurs by the propagation of an annulus of failure from the perimeter of the contact zone and this phenomenon could be important in tactile perception and grip function.

Effect of skin hydration on the dynamics of fingertip gripping contact
Thibaut André, Vincent Lévesque, Vincent Hayward, Philippe Lefèvre, and Jean-Louis Thonnard
Journal of the Royal Society Interface, published online 13 April 2011.

Abstract

The dynamics of fingertip contact manifest themselves in the complex skin movements observed during the transition from a stuck state to a fully developed slip. While investigating this transition, we found that it depended on skin hydration. To quantify this dependency, we asked subjects to slide their index fingertip on a glass surface while keeping the normal component of the interaction force constant with the help of visual feedback. Skin deformation inside the contact region was imaged with an optical apparatus that allowed us to quantify the relative sizes of the slipping and sticking regions. The ratio of the stuck skin area to the total contact area decreased linearly from 1 to 0 when the tangential force component increased from 0 to a maximum. The slope of this relationship was inversely correlated to the normal force component. The skin hydration level dramatically affected the dynamics of the contact encapsulated in the course of evolution from sticking to slipping. The specific effect was to reduce the tendency of a contact to slip, regardless of the variations of the coefficient of friction. Since grips were more unstable under dry skin conditions, our results suggest that the nervous system responds to dry skin by exaggerated grip forces that cannot be simply explained by a change in the coefficient of friction.

Exploration multimodale d'images pour des utilisateurs ayant une déficience visuelle
Gregory Petit, Aude Dufresne, Vincent Lévesque, and Vincent Hayward
Sciences et Technologies pour le Handicap, 2(2), pp. 175-186, 2008.

Résumé

Cet article preésente une recherche visant à rendre accessibles des images aux usagers ayant une déficience visuelle. Le logiciel MaskGen a été développé afin de transformer interactivement ces images « visuelles » en images multimodales (tactiles et audio). Une méthodologie a été développé pour transposer ces images et les afficher sur le Tactograph, un appareil multimodal dynamique.

Abstract

This article presents research on making images accessible for people with visual impairment. The MaskGen system was developed to interactively transpose these "visual" images into multimodal images (tactile and audio). A methodology was designed to transpose the images and prepare them to be displayed on the Tactograph, a refreshable multimodal device.

Haptically Enabled Handheld Information Display with Distributed Tactile Transducer
Jerome Pasquero, Joseph Luk, Vincent Lévesque, Qi Wang, Vincent Hayward, and Karon E. MacLean
IEEE Transactions on Multimedia, 9(4), pp 746-753, 2007.

Abstract

This paper describes the design, construction, and initial evaluation of a handheld information device that supports combined tactile and graphical interaction. The design comprises a liquid crystal graphic display co-located with a miniature, low-power, distributed tactile transducer. This transducer can create electronically-controlled lateral skin deformation patterns which give the sensation of sliding over small shapes. It is integrated within a slider mechanism to control scrolling. It also functions as a detent when pushing on it. Tactile feedback and the combination of visual and tactile feedback in a mobile context enable the development of new functions, such as multimodal navigation within large graphic spaces.

Display of Virtual Braille Dots by Lateral Skin Deformation: Feasibility Study
Vincent Lévesque, Jerome Pasquero, Vincent Hayward, and Maryse Legault
ACM Transactions on Applied Perception, 2(2), pp 132-149, 2005.

Abstract

When a progressive wave of localized deformations occurs tangentially on the fingerpad skin, one typically experiences the illusion of a small object sliding on it. This effect was investigated because of its potential application to the display of Braille. A device was constructed that could produce such deformation patterns along a line. Blind subjects' ability to read truncated Braille characters ('lowered lowered', 'lowered raised', 'raised lowered', and 'raised raised') using the device was experimentally tested and compared to their performance with a conventional Braille medium. While subjects could identify two-character strings with a high rate of success, several factors need to be addressed before a display based on this principle can become practical.

Conference Proceedings

ReFlex: A Flexible Smartphone with Active Haptic Feedback for Bend Input
Paul Strohmeier, Jesse Burstyn, Juan Pablo Carrascal, Vincent Lévesque, and Roel Vertegaal
Proc. ACM TEI'16 Conference on Tangible, Embedded and Embodied Interaction, Eindhoven, The Netherlands, February 2016.

Abstract

ReFlex is a flexible smartphone with bend input and active haptic feedback. ReFlex's features allow the introduction of sensations such as friction or resistance. We report results from an experiment using ReFlex in a targeting task, as well as initial users' reactions to the prototype. We explore both absolute and relative tactile haptic feedback, paired with two types of bend input mappings: position-control and rate-control. We observed that position-controlled cursors paired well with relative bend feedback, while rate-controlled cursors paired well with absolute bend feedback to indicate targets. We also explored an eyes-free condition. Results suggest that while eyes-free, haptic feedback conditions were more error-prone than visual-only conditions, the size of the error was relatively small, and users were able to complete the task in all cases. We present two application scenarios that take advantage of the unique input and output modalities of ReFlex and discuss its potential for within document navigation.

Video

News

Exploring the Design Space of Programmable Friction for Scrolling Interactions
Vincent Lévesque, Louise Oram and Karon MacLean
Proc. Haptics Symposium 2012, Vancouver, Canada, March 2012.
Best Paper

Abstract

Scrolling interactions are an important aspect of the design of usable touchscreen interfaces, particularly for handheld devices that can only display a limited amount of information at once. Using a touchscreen capable of dynamically altering its surface friction, we explore the design space of haptically-augmented scrolling interactions and investigate programmable friction's ability to provide appropriate feedback in envisioned usage scenarios. We performed five user experiments to evaluate respectively the identifiability of a set of iconic detents, the countability of detents, the perception of detent density, the synchronization of tactile feedback to on-screen events, and the optimal friction pattern for a spring-like resistance. The results of these experiments provide valuable information that will inform the design of scrolling interactions that leverage programmable friction for an improved user experience.

Adaptive Level of Detail in Dynamic, Refreshable Tactile Graphics
Vincent Lévesque, Gregory Petit, Aude Dufresne, and Vincent Hayward
Proc. Haptics Symposium 2012, Vancouver, Canada, March 2012.

Abstract

We investigate gains in user appreciation and performance when the level of detail of tactile graphics is dynamically altered either at the press of a button or automatically, as a function of exploration speed. This concept was evaluated by asking 9 visually impaired participants to perform hierarchical spatial search tasks in a concert hall illustration. The tasks could be simplified by first searching for a section in a sparse illustration, and then a seat in a detailed illustration. The results show no improvement in task performance but indicate a user preference for explicitly controlling the level of details with the manual toggle.

Enhancing Physicality in Touch Interaction with Programmable Friction
Vincent Lévesque, Louise Oram, Karon MacLean, Andy Cockburn, Nicolas D. Marchuk, Dan Johnson, J. Edward Colgate, and Michael A. Peshkin
Proc. ACM Conference on Human Factors in Computing Systems (CHI '11), Vancouver, Canada, May 2011, pp. 2481-2490.
Best of CHI

Abstract

Touch interactions have refreshed some of the 'glowing enthusiasm' of thirty years ago for direct manipulation interfaces. However, today's touch technologies, whose interactions are supported by graphics, sounds or crude clicks, have a tactile sameness and gaps in usability. We use a Large Area Tactile Pattern Display (LATPaD) to examine design possibilities and outcomes when touch interactions are enhanced with variable surface friction. In a series of four studies, we first confirm that variable friction gives significant performance advantages in low-level targeting activities. We then explore the design space of variable friction interface controls and assess user reactions. Most importantly, we demonstrate that variable friction can have a positive impact on the enjoyment, engagement and sense of realism experienced by users of touch interfaces.

Video

News

Laterotactile Rendering of Vector Graphics with the Stroke Pattern
Vincent Lévesque and Vincent Hayward
Proc. EuropHaptics 2010, Part II, Kappers, A.M.L. et al. (Eds.), LNSC 6192, Springer-Verlag, pp. 25-30.

Abstract

Raised line patterns are used extensively in the design of tactile graphics for persons with visual impairments. A tactile stroke pattern was therefore developed to enable the rendering of vector graphics by lateral skin deformation. The stroke pattern defines a transversal profile and a longitudinal texture which provide tactile feedback while respectively crossing over the stroke and tracing its length. The stroke pattern is demonstrated with the rendering of lines, circles and polygons, and is extensible to other vector graphics primitives such as curves. The parametric nature of the stroke allows the representation of distinctive line types and the online adjustment of line thickness and other param- eters according to user preferences and capabilities. The stroke pattern was informally evaluated with four visually impaired volunteers.

The effects of moisture on fingertip skin deformation during loading and slipping
Thibaut André, Vincent Lévesque, Vincent Hayward, Philippe Lefèvre, and Jean-Louis Thonnard
39th Annual Meeting of the Society for Neuroscience, Chicago, USA, 2009.

Abstract

During tactile interaction, skin deformation stimulates the mechanoreceptors, enabling the nervous system to become aware of the properties of the touched surface. Even on perfectly smooth, glassy surfaces, the fingertip skin deforms in complicated ways when slip occurs. The aim of the present study is to assess the effects of moisture content and interaction forces on the fingertip skin deformation occurring in this condition.

Twelve subjects having a large range of skin moisture (from dry to wet) participated in this study. They were asked to exert a constant normal force against a glass surface with the index fingertip and to increase the tangential force until slip occurs. Each subject performed five slippages for six different levels of normal force (0.2, 0.5, 1, 2, 5, 10N). Forces and moisture were measured with a force-torque sensor (ATI F/T Mini40) and a moisture evaluator (Corneometer®), respectively. Skin deformation at the contact surface was acquired with a specifically designed optical apparatus that allowed the evaluation of the relative sizes of the slipping and sticking regions.

The area of the sticking region decreased when the tangential force increased. It varied linearly with the ratio of net tangential/normal force components. The slope of this relationship was strongly influenced by both the normal force exerted and the moisture content of the skin. Finally, the static coefficient of friction could be extracted when the sticking regions vanished and the slipping region spread to the whole surface of contact.

This study described the deformation of the skin occurring during interaction with smooth surfaces and evaluated the effect of interaction force components and moisture on this deformation. A skin deformation model that takes into account these two parameters is presently being developed. This research was supported by a grant from Prodex (contract numbers 90063, 90064, 90231, 90232), ESA (European Space Agency), PAI, FNRS, FRSM, ARC, and NANOBIOTACT-project (EU-FPG-NMP-033287).

Tactile Graphics Rendering Using Three Laterotactile Drawing Primitives
Vincent Lévesque and Vincent Hayward
Proc. 16th Symposium on Haptic Interfaces For Virtual Environment And Teleoperator Systems, March 13-14, 2008, Reno, Nevada, pp. 429-436.

Abstract

This paper presents preliminary work towards the development and evaluation of a practical refreshable tactile graphics system for the display of tactile maps, diagrams and graphs for people with visual impairments. Refreshable tactile graphics were dynamically produced by laterally deforming the skin of a finger using the STReSS2 tactile display. Tactile features were displayed over an 11 x 6 cm virtual surface by controlling the tactile sensations produced by the fingerpad-sized tactile display as it was moved on a planar carrier. Three tactile rendering methods were used to respectively produce virtual gratings, dots and vibrating patterns. These tactile features were used alone or in combination to display shapes and textures. The ability of the system to produce tactile graphics elements was evaluated in five experiments, each conducted with 10 sighted subjects. The first four evaluated the perception of simple shapes, grating orientations, and grating spatial frequencies. The fifth experiment combined these elements and showed that tactile icons composed of both vibrating contours and grated textures can be identified. The fifth experiment was repeated with 6 visually impaired subjects with results suggesting that similar performance should be expected from that user group.

Graphisme tactile appliqué aux illustrations de manuels scolaires à l'usage d'enfants ayant une déficience visuelle
Gregory Petit, Aude Dufresne, Vincent Lévesque, Vincent Hayward, and Nicole Trudeau
Proc. IHM 2008, Sept. 3-5, 2008, Metz, France.

Résumé

Cet article présente une recherche visant à rendre accessible, aux étudiants ayant une déficience visuelle les illustrations de manuels scolaires. Le logiciel MaskGen a été développé afin de transformer interactivement ces illustrations en graphiques tactiles. Une méthodologie a été développée pour transposer ces graphiques et les afficher sur le STReSS2, un appareil tactile dynamique. Nous avons expérimenté différentes associations de rendus tactiles et de retours sonores pour obtenir une version compréhensible pour les étudiants ayant une déficience visuelle. Nous avons testé trois graphiques tactiles (un plan, un histogramme et une carte) avec quarante participants : vingt voyants, dix adultes non-voyants et dix enfants non-voyants. Les résultats de l'expérimentation montrent que les participants ayant une déficience visuelle ont aimé les images tactiles, ont été capables de les explorer correctement et de répondre avec succès aux questions posées sur ces images.

Abstract

This article presents research on making schoolbook illustrations accessible for students with visual impairment. The MaskGen system was developed to interactively transpose illustrations of schoolbooks into tactile graphics. A methodology was designed to transpose the graphics and prepare them to be displayed on the STReSS2, a refreshable tactile device. We experimented different associations of tactile rendering and audio feedbacks to find a model that children with visual impairment could use. We experimented with three scientific graphics (diagram, bar chart and map) with forty participants: twenty sighted adults, ten adults with visual impairment, and ten children with visual impairment. Results show that the participants with visual impairment liked the tactile graphics and could use them to explore illustrations and to answer questions about their content.

Refreshable Tactile Graphics Applied to Schoolbook Illustrations for Students with Visual Impairment
Gregory Petit, Aude Dufresne, Vincent Lévesque, Vincent Hayward, and Nicole Trudeau
Proc. ASSETS 2008, Oct. 13-15, 2008, Halifax, Canada.

Abstract

This article presents research on making schoolbook illustrations accessible for students with visual impairment. The MaskGen system was developed to interactively transpose illustrations of schoolbooks into tactile graphics. A methodology was designed to transpose the graphics and prepare them to be displayed on the STReSS2, a refreshable tactile device. We experimented different associations of tactile rendering and audio feedbacks to find a model that children with visual impairment could use. We experimented with three scientific graphics (diagram, bar-chart and map) with forty participants: twenty sighted adults, ten adults with visual impairment, and ten children with visual impairment. Results show that the participants with visual impairment liked the tactile graphics and could use them to explore illustrations and answer questions about their content.

Braille Display by Lateral Skin Deformation with the STReSS Tactile Transducer
Vincent Lévesque, Jerome Pasquero and Vincent Hayward
Proc. World Haptics 2007, March 22-24, Tsukuba, Japan.
Best Paper Award for Haptic Application

Abstract

Earlier work with a 1-D tactile transducer demonstrated that lateral skin deformation is sufficient to produce sensations similar to those felt when brushing a finger against a line of Braille dots. Here, we extend this work to the display of complete 6-dot Braille characters using a general purpose 2-D tactile transducer called STReSS2. The legibility of the produced Braille was evaluated by asking seven expert Braille readers to identify meaningless 5-letter strings as well as familiar words. Results indicate that reading was difficult but possible for most individuals. The superposition of texture to the sensation of a dot improved performance. The results contain much information to guide the design of a specialized Braille display operating by lateral skin deformation. They also suggest that rendering for contrast rather than realism may facilitate Braille reading when using a weak tactile transducer.

A Role for Haptics in Mobile Interaction: Initial Design Using a Handheld Tactile Display Prototype
Joseph Luk, Jerome Pasquero, Shannon Little, Karon MacLean, Vincent Lévesque, and Vincent Hayward
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'06), Montreal, Canada, April 2006, pp. 171-180.
Best of CHI

Résumé

mappings between device capabilities and specific roles in mobile interaction, and the next step of hardware re-engineering.

Abstract

Mobile interaction can potentially be enhanced with well- designed haptic control and display. However, advances have been limited by a vicious cycle whereby inadequate haptic technology obstructs inception of vitalizing applications. We present the first stages of a systematic design effort to break that cycle, beginning with specific usage scenarios and a new handheld display platform based on lateral skin stretch. Results of a perceptual device characterization inform

Display of Virtual Braille Dots by Lateral Skin Deformation: A Pilot Study
Jerome Pasquero, Vincent Lévesque, Vincent Hayward, and Maryse Legault
Proc. Eurohaptics 2004, Munich, Germany, June 2004

Abstract

When a progressive wave of localized deformations occurs tangentially on the fingerpad skin, one typically experiences the illusion of a small object sliding on it. This effect was investigated because of its potential application to the display of Braille. A device was constructed that could produce such deformation patterns along a line. This enabled us to test blind subjects' ability to read the truncated Braille characters 'lowered lowered', 'lowered raised', 'raised lowered', and 'raised raised'. While subjects could identify two-character strings with a high rate of success, several factors need to be addressed before a display based on this principle can become practical.

Video

Experimental Evidence of Lateral Skin Strain During Tactile Exploration
Vincent Lévesque and Vincent Hayward
Proc. Eurohaptics 2003, Dublin, Ireland, July 2003

Abstract

This paper describes an experimental platform for the study of stretch and compression of the human fingerpad skin during tactile exploration. A digital camera records the sequence of patterns created by a fingertip as it slides over a transparent surface with simple geometrical features. Skin deformation is measured with high temporal and spatial resolution by tracking anatomical landmarks on the fingertip. Techniques adapted from the field of online fingerprinting are used to acquire high-contrast fingerprint images and extract salient features (pores, valley endings, and valley bifurcations). The results of experiments performed with surfaces with a bump or hole and flat surfaces are presented. This work is motivated by the need to provide meaningful 'tactile movies' for a tactile display that uses distributed lateral skin stretch.

Abstracts, Posters and Technical Reports

Refreshable tactile graphics using a lateral skin deformation device
Vincent Lévesque, Vincent Hayward, Gregory Petit, and Aude Dufresne
Abstract selected for oral presentation, 9th International Conference on Low Vision, July 7 - 11, 2008, Montreal, Quebec, Canada.

Abstract

Tactile graphics are useful to convey spatial information and concepts to visually impaired persons with maps, mathematical diagrams and other types of illustrations. Unfortunately, tactile graphics are currently produced on physical media, a slow and cumbersome process that results in static material. Tactile graphics also present unique challenges due to the relatively limited resolution of the tactile sense, often requiring a reduction in information density. Refreshable tactile graphics could alleviate these problems by allowing instant access to online illustrations as well as interactive control over features such as the visibility of layers of information.

Research on refreshable tactile graphics has focused on programmable arrays of moving pins that approximate 3D surfaces. For the past decade, we have been investigating an alternative approach based on lateral deformation of the skin. This approach relies on miniature bending motors that pull laterally on the skin, locally stretching or compressing it to stimulate tactile receptors. The latest prototype covers the surface of a typical fingerpad with a matrix of 8 by 8 actuators. Once mounted on a mouse-like planar carrier, the device can alter the tactile sensation felt in response to displacements by controlling the activation of the actuators, thereby creating a large virtual surface out of a small active area. The system can currently be used to draw refreshable tactile graphics with three types of features. The first produces a strong, localized vibration sensation. The second produces a sensation similar to that of brushing against Braille dots. The third produces the sensation of brushing against a grating texture with a programmable roughness and orientation. Recent experiments have shown that these features can be combined to display distinguishable textured shapes.

Recent joint work between McGill University and University of Montréal focuses on the adaptation of schoolbook illustrations for visually impaired children. Experiments were recently conducted to evaluate the effectiveness of the system at conveying the pedagogical content of graphics adapted from a high school history textbook. Participants answered questions based on their interpretation of a world map of the first civilizations, a bar chart of the population of ancient Athens, and an illustration of a cathedral. Results suggest that the system could become a valuable tool for the education of visually impaired children.

Blindness, Technology and Haptics
Vincent Lévesque
CIM Technical Report (CIM-TR-05.08), McGill University, Montreal, Canada, 28 pages, October 2005.

Abstract

The blind and the visually impaired are in a unique position to appreciate and make functional use of haptic devices. Designing devices for the blind is, however, more arduous than many researchers and inventors expect. It is thus important to fully understand the needs and requirements of that community before attempting to create devices for them. It is also important to learn from past research and development in the application of technology for the blind. This survey provides an overview of current knowledge on blindness and rehabilitation technology relevant for the design of aids for the blind, and more particularly for the use of haptics with the blind.

The survey begins with a demystification of blindness and a discussion of the differences between blind and sighted. Follows a broad overview of the many attempts at applying technological solutions to problems encountered by the blind. The survey ends with a discussion of lessons learned from previous failures and successes in rehabilitation technology as well as speculation on the future of haptics and other technologies for people living with blindness.

Display of Virtual Braille Dots by Lateral Skin Deformation
Vincent Lévesque, Jerome Pasquero, Vincent Hayward, and Maryse Legault
Poster, 14th Annual Canadian Conference on Intelligent Systems (IRIS), Ottawa, Canada, June 2004
3rd Prize, Best Poster
Lateral Skin Stretch Measurement by Fingerprint Minutia Tracking
Vincent Lévesque and Vincent Hayward
Poster, 12th Annual Canadian Conference on Intelligent Systems (IRIS), Calgary, Canada, May 2002

Abstract

This poster proposes a technique for the measurement of lateral skin stretch on the fingertip. A sequence of images is acquired as the fingertip moves across a transparent surface. Fingerprint minutia are detected using techniques borrowed from the fingerprint identification field (Stosz and Alyea, 1994) and tracked between pairs of successive images. Variations in inter-minutia distance are used to evaluate lateral skin stretch.

Demonstrations

Programmable Friction in Scrolling Interactions
Vincent Lévesque, Louise Oram and Karon MacLean
Demonstration, Haptics Symposium 2012, Vancouver, Canada, March 2012.
Nominated for Best Demonstration

Abstract

Scrolling is a pervasive interaction for accessing information that is currently off screen. Devices with small screens further leverage scrolling, and are often used in scenarios where the visual attention of the user is divided between the screen and the environment. Using a touchscreen that can dynamically alter its surface friction, scrolling interactions can be augmented, providing a non-visual source of feedback about the state of the system. Detents, and other distinct patterns created from changing friction, can be used to signal when a new item has entered the screen, and the density of this feedback can be seen as the rate of scrolling. Circular scrolling, variable rate scrolling (or scrubbing), as well as other exemplar applications will be used to demonstrate how these ideas, and others, can be used to effectively augment scrolling interactions.

Frictional Widgets: Enhancing Touch Interfaces with Programmable Friction
Vincent Lévesque, Louise Oram, Karon MacLean, Andy Cockburn, Nicolas D. Marchuk, Dan Johnson, J. Edward Colgate, and Michael A. Peshkin
CHI'11 Extended Abstracts on Human Factors in Computing Systems, Vancouver, Canada, May 2011, pp. 1153-1158.

Abstract

Touch interactions occur through flat surfaces that lack the tactile richness of physical interfaces. We explore the design possibilities offered by augmenting touchscreens with programmable surface friction. Four exemplar applications - an alarm clock, a file manager, a game, and a text editor - demonstrate tactile effects that improve touch interactions by enhancing physicality, performance, and subjective satisfaction.

Video

Refreshable Tactile Graphics with the STReSS2 Laterotactile Display
Vincent Lévesque, Andrew H. C. Gosline and Vincent Hayward
Technical Demonstration, 16th Symposium on Haptic Interfaces For Virtual Environment And Teleoperator Systems, March 13-14, 2008, Reno, Nevada.
Best Demonstration

Abstract

This demonstration will allow visitors to feel virtual tactile graphics rendered with the STReSS2, a tactile display that relies on lateral skin deformation to produce tactile sensations. The display consists of a fingerpad-sized array of 8 by 8 laterally moving actuators that are programmed to produce deformation patterns in response to movements of the device over a letter-sized workspace. Tactile graphics are drawn using three tactile rendering methods that respectively produce virtual dots, virtual gratings, and localized vibrations. Despite resulting from lateral deformation rather than indentation, virtual dots and gratings give a convincing illusion of brushing against raised dot patterns and embossed undulating textures. Localized vibrations produce less natural but stronger, sharper sensations. Visitors will be invited to explore a variety of tactile graphics produced by combining these three rendering methods, ranging from simple shapes (as investigated in the accompanying paper) to more complex illustrations.

Refreshable Tactile Graphics with the STReSS2 Laterotactile Display
Vincent Lévesque and Vincent Hayward
Technical Demonstration, EuroHaptics 2008, June 11-13, 2008, Madrid, Spain.

Abstract

This demonstration will allow visitors to feel virtual tactile graphics rendered with the STReSS2, a tactile display that relies on lateral skin deformation to produce tactile sensations. The display consists of a fingerpad-sized array of 8 by 8 laterally moving actuators that are programmed to produce deformation patterns in response to movements of the device over a letter-sized workspace (approx. 28 by 22 cm). Tactile graphics are drawn using three tactile rendering methods that respectively produce virtual dots, virtual gratings, and localized vibrations. Despite resulting from lateral deformation rather than indentation, virtual dots and gratings give a convincing illusion of brushing against raised dot patterns and embossed undulating textures. Localized vibrations produce less natural but stronger, sharper sensations. Visitors will be invited to explore a variety of tactile graphics produced by combining these three rendering methods, ranging from simple shapes to more complex illustrations. A similar demonstration of this technology was recently awarded the best demonstration prize at the 2008 Haptics Symposium in Reno, Nevada. More details on this work can be found on the project webpage at laterotactile.com.

A Haptic Memory Game using the STRESS Tactile Display
Qi Wang, Vincent Lévesque, Jerome Pasquero, and Vincent Hayward
CHI'06 Extended Abstracts on Human Factors in Computing Systems, CHI'06, Montreal, Canada, April 2006, pp. 271-274.

Abstract

A computer implementation of a classic memory card game was adapted to rely on touch rather than vision. Instead of memorizing pictures on cards, players explore tactile graphics on a computer-generated virtual surface. Tactile sensations are created by controlling dynamic, distributed lateral strain patterns on a fingerpad in contact with an electronic tactile display called STReSS2. The tactile graphics are explored by moving the device within the workspace of a 2D planar carrier. Three tactile rendering methods were developed and used to create distinct tactile memory cards. The haptic memory game showcases the capabilities of this novel tactile display technology.

Video

Display of Virtual Braille Dots by Lateral Skin Deformation
Vincent Lévesque, Jerome Pasquero, Vincent Hayward, and Maryse Legault
Technical Demonstration, 14th Annual Canadian Conference on Intelligent Systems (IRIS), Ottawa, Canada, June 2004
Minitature Tactile Display
Jerome Pasquero, Vincent Lévesque and Vincent Hayward
Technical Demonstration, 12th Annual Canadian Conference on Intelligent Systems (IRIS), Calgary, Canada, May 2002
Best Demonstration

Talks, Workshops and Tutorials

Designing the Haptic User Experience
Vincent Lévesque
Invited Talk, Technoculture, Art and Games (TAG), Concordia, June 11, 2014.
Programmable Friction and the Haptic User Experience
Vincent Lévesque
Invited Talk, Tactile Research Group, Psychomics 2013, Toronto, Canada, November 14, 2013.
Designing Interactions for Novel Haptic Technologies
Vincent Lévesque
Invited Talk, Haptic Interaction Design Workshop, Haptics Symposium 2012, Vancouver, Canada, March 2012.

Abstract

The haptics community is a great source of innovation, with novel means of stimulating the human sense of touch being constantly devised. Developing compelling applications for these innovative technologies, however, can be a significant challenge due to the initial lack of understanding of relevant human factors and the dedication required to design high-performance hardware. A common approach is to first optimize a device’s hardware design, and then methodically build up knowledge of its psychophysics prior to investigating applications. In this talk, I will propose alternative approaches that skip ahead to the exploration of a haptic technology’s interaction design space and to the evaluation of its benefits for the user experience in concrete applications. Drawing from my experience with laterotactile displays and programmable friction touchscreens, I will illustrate how rapid prototyping and user centered design can demonstrate the value of novel haptic feedback early on, justifying and guiding further investments in hardware refinements and human factor studies.

Do-It-Yourself Haptics: A Practical Introduction to Haptics for Consumer Electronics
Vincent Lévesque and Karon Maclean
Invited Tutorial, IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, ICCE, January 2011.

Abstract

What was once accomplished by directly interacting with physically embodied objects, tools and controls is now increasingly performed through abstract, intangible interfaces. The distinct edges, textures and clicks of physical buttons are, for example, gradually being replaced by the blank feel of a touchscreen, with only audio-visual feedback remaining to guide actions. While flexible, these interfaces impose greater attentional demands and do not fully exploit the richness of the sense of touch. Haptic feedback promises to restore natural, direct and responsive interaction to these digital interfaces and to thereby improve their physical esthetics and reduce their reliance on the overloaded senses of vision and audition.

This tutorial assumes no knowledge of haptics and aims not only to provide a broad overview of the topic to consumer electronics practitioners, but also to engage you in a dialogue about the challenges and benefits of integrating this technology in their products. The tutorial will introduce you to current uses of haptics in consumer electronics, the basics of haptic perception in humans, the design and effective use of simple haptic interfaces, and cutting edge haptic interfaces being explored in research labs around the world. It will provide a solid basis for consumer electronics professionals to further investigate this important emerging technology.

Enhancing Touch Interactions with Programmable Friction
Vincent Lévesque
Invited Talk, Workshop on Vibrotactile Haptics for Touch Screens, IEEE World Haptics Conference 2011, Istanbul, Turkey, June 2011.

Abstract

Touch interactions take place through flat surfaces that lack the tactile richness of physical interfaces and hence do not fully exploit our sense of touch. This collaborative work explores the reintroduction of rich tactile feedback in touch interfaces through the use of programmable surface friction. Using Northwestern University's Large Area Tactile Pattern display (LATPaD), we demonstrate that programmable friction can not only improve performance in targeting tasks but also have a positive impact on the enjoyment, engagement and sense of realism experienced by users of touch interfaces.

Restoring Physicality to Touch Interaction with Programmable Friction
Vincent Lévesque, Louise Oram, Karon MacLean, J. Edward Colgate, and Michael A. Peshkin
Proc. International Conference on Consumer Electronics (ICCE) 2011, Las Vegas, IEEE, 2 pages, January 2011.

Abstract

Rich tactile interaction with control surfaces has been compromised in the transition to touch interfaces. This paper discusses new haptic effects that restore physicality to touch interaction by dynamically altering a touchscreen's frictional properties.

Laterotactile Displays and Rendering: from Illusion to Application
Vincent Lévesque and Jerome Pasquero
Invited Talk, IEEE VR 2008 Tutorial, Integration of Haptics in Virtual Environments: from Perception to Rendering, March 9, 2008, Reno, Nevada.

Abstract

This tutorial will discuss the process through which seemingly minor observations about haptic phenomenon have led to the design of simple yet powerful interfaces in the McGill Haptics Laboratory. The focus will be on a tactile display technology that relies on lateral skin deformation to produce a variety of tactile sensations, including some that are typically associated with normal indentation of the skin. This work, which was initially inspired by a simple tactile illusion produced by running a pen along the teeth of a comb, has resulted in the design of a family of practical laterotactile displays and the development of tactile rendering algorithms. Applications of this technology for mobile interaction and for the display of refreshable Braille and tactile graphics for visually impaired persons will also be covered.

Laterotactile Devices and Rendering
Vincent Hayward and Vincent Lévesque
Invited Talk, EuroHaptics 2006, Workshop II - Tactile Display Design, Paris, France, July 2006
Experimental Evidence of Lateral Skin Strain During Tactile Exploration
Vincent Lévesque
Online presentation, International Society for Haptics, May 2004

Theses

Virtual Display of Tactile Graphics and Braille by Lateral Skin Deformation
Vincent Lévesque
Ph.D. Thesis, McGill University, August 2009

Abstract

Graphical content is increasingly pervasive in digital interfaces and documents yet it remains accessible to visually impaired persons almost exclusively on media with limited flexibility such as embossed paper. Textual content is more accessible but nevertheless limited by the cost and functionality of refreshable Braille displays and voice synthesis. This thesis explores the use of a novel tactile stimulation approach that relies on lateral skin deformation for the computerized display of virtual Braille and tactile graphics.

Tactile synthesis by lateral skin deformation is initially explored in the context of Braille. The feasibility of producing virtual Braille by laterotactile stimulation is first demonstrated by creating the illusion of brushing against a line of Braille dots through the synchronization of a travelling wave of skin deformation with the displacement of a tactile array of eight actuators. This principle is then extended to complete 6-dot Braille cells by distributing lines of virtual dots onto the rows of actuators of a general-purpose STReSS2 tactile array. Reading the resulting virtual Braille is shown to generally be feasible but demanding, suggesting that a specialized laterotactile Braille display should be devised or that dots be rendered for contrast rather than realism.

Tactile rendering by lateral skin deformation is then further explored with the gradual development of a virtual tactile graphics framework that emulates conventional features such as raised lines and areal textures through a coherent set of patterns that includes grating textures, stroked and dotted shapes, bitmap-based masks, and composite patterns. Dynamic rendering is also exploited to produce novel effects such as tactile flow, reactive textures dependent on the exploration behaviour, and interactive content with alternate views. The usability of the framework is informally evaluated with visually impaired volunteers and early tactile patterns studied through formal experiments. The tactile patterns are presented on the Tactograph, a haptic interface redesigned specifically for the display of tactile graphics that combines a STReSS2 display with an instrumented planar carrier.

This thesis demonstrates the potential of lateral skin deformation for the display of Braille and tactile graphics, and explores in the process the ways in which this novel approach to tactile stimulation can be applied to produce meaningful tactile sensations.

News

Measurement of Skin Deformation Using Fingerprint Feature Tracking
Vincent Lévesque
M. Eng. Thesis, McGill University, November 2002
Dean's Honour List

Abstract

This thesis describes an experimental platform for the study of skin stretch of the human fingerpad during tactile exploration tasks. A digital camera records the sequence of patterns created by a fingertip as it slides over a transparent surface with simple geometrical features. Skin deformation is measured with high temporal and spatial resolution by tracking anatomical landmarks on the fingertip. Techniques adapted from the field of online fingerprinting are used to acquire high-contrast fingerprint images and extract salient features (pores, valley endings, and valley bifurcations). The performance of the method is evaluated with surfaces embossed with a ridge or valley and flat surfaces. This work is motivated by the need to provide meaningful 'tactile movies' for a tactile display using distributed lateral skin stretch.

Patents

Granted Patents (8)

Simulation of Tangible User Interface Interactions and Gestures Using Array of Haptic Cells
Ali Modarres, Juan Manuel Cruz-Hernandez, Danny Grant, and Vincent Lévesque
US Patent 9,405,369, filed April 25, 2014, issued August 2, 2016.

Abstract

A user interface device includes a flexible layer comprising a touch surface configured to receive a touch by a user, a plurality of haptic cells covered by the flexible layer, each haptic cell comprising a haptic output device, a sensor configured to sense an amount and/or rate of deformation of the flexible layer when a user touches the touch surface, and a processor configured to receive an output signal from the sensor, generate a haptic control signal based on the output signal from the sensor, and output the haptic control signal to at least one haptic output device of the plurality of haptic cells to cause the haptic output device to deform an associated haptic cell in response to the sensed deformation of the flexible layer.

Method and Apparatus of Converting Control Tracks for Providing Haptic Feedback
Vincent Lévesque, Jamal Saboune and David Birnbaum
US Patent 9,401,079, filed August 29, 2014, issued July 26, 2016.

Abstract

Disclosed are systems and methods for converting a control track designed for use with a number and/or type of haptic output devices to be used with other numbers and/or types of haptic output devices. For example, a computing device may convert the control track into another control track that can be applied to other types and/or numbers of haptic output devices. The converted control track may be compatible for use with a smartphone or other system that includes a different number and/or type of haptic feedback devices than the system for which the haptic track was originally designed. In this manner, the user of the smartphone or other system may experience haptic feedback using a device that is different from another haptic feedback system for which the control track was originally designed for use. The conversion may occur locally at the smartphone or other system and/or remotely at another device.

System and Method for Simulated Physical Interactions With Haptic Effects
Vincent Lévesque, Juan Manuel Cruz-Hernandez, Amaya Weddle, and David Birnbaum
US Patent 9,330,544, filed March 14, 2013, issued May 3, 2016.

Abstract

A system of the present disclosure may include a sensor configured to detect user interaction with a touch surface and transmit a sensor signal associated with the user interaction; a processor in communication with the sensor, the processor configured to: determine a position of the user interaction based on the sensor signal, determine a feature associated with the position of the user interaction, control a device associated with the feature, modify a display signal based in part on the user interaction, select a haptic effect to generate based at least in part on user interaction and the position, the haptic effect selected to simulate the feature, and transmit a haptic signal to generate the haptic effect, and a haptic output device in communication with the processor and coupled to the touch surface, the haptic output device configured to receive a haptic signal and output a haptic effect.

Systems and Methods For Haptics In Vibrating Environments and Devices
Vincent Lévesque and Amaya Weddle
US Patent 9,202,351, filed March 11, 2013, issued December 1, 2015.

Abstract

Systems and methods for haptics in vibrating environments and devices are disclosed. For example, one described system includes: a haptic output device; a processor coupled to the haptic output device, the processor configured to: determine that a haptic effect should be generated; receive a signal associated with a parasitic vibration; determine a haptic effect based in part on the parasitic vibration; and output a haptic signal associated with the haptic effect to the haptic output device.

Automatic Haptic Effect Adjustment System
Vincent Lévesque
US Patent 9,202,352, filed March 11, 2013, issued December 1, 2015.

Abstract

A system is provided that automatically adjusts a haptic effect. The system generates a haptic effect based on one or more haptic parameters. The system measures an affective state of a user that experiences the haptic effect. The system adjusts at least one haptic parameter of the one or more haptic parameters based on the measured affective state. The system generates a new haptic effect based on the one or more haptic parameters, where the at least one haptic effect parameter causes the new haptic effect to be different from the haptic effect.

Method and Apparatus for Simulating Surface Features on a User Interface with Haptic Effects
Vincent Lévesque and Juan Manuel Cruz-Hernandez
US Patent 9,196,134, filed October 31, 2012, issued November 24, 2015.

Abstract

A haptic effect enabled device for simulating a tactile sensation on a surface. In some cases, the haptic effect enabled device may be a user interface device, and the tactile sensation may be simulated on a surface of the user interface device. The interface device may include a haptic output device configured to generate a haptic effect, such as a periodic haptic effect, at the surface. The interface device may include a drive module configured to generate a periodic drive signal based on a touch input at the surface of the interface device and based on the tactile sensation to be simulated at the surface. The interface device may include a drive circuit operatively coupled to the drive module and the haptic output device and configured to apply the periodic drive signal to the haptic output device. In some cases, the surface may be separate from the device.

User Interface Device Provided with Surface Haptic Sensations
Danny Grant and Vincent Lévesque
US Patent 9,041,647, filed March 15, 2013, issued May 26, 2015.

Abstract

A user interface device includes a housing, a user input element supported by the housing, and a haptic output device supported by the housing. The haptic output device is configured to generate a haptic effect at a surface of the user interface device. The surface is part of the housing and/or the user input element. A processor disposed within the housing. The processor is configured to receive an input command from the user input element, communicate the input command to a host computer, receive an output command from the host computer, and output a haptic signal based on the output command to the haptic output device to generate the haptic effect at the surface.

Apparatus to Reproduce Tactile Sensations
Vincent Hayward, Jerome Pasquero and Vincent Lévesque
US Patent 7,077,015, filed May 29, 2003, issued July 18, 2006.

Abstract

A tactile sensing transducer provides as its output a skin deformation signal for storage or transmission to a tactile stimulation display transducer. A skin surface imaging means operating optically, acoustically or otherwise is positioned for viewing and providing an output image signal corresponding to said deformed skin surface. An electronic processing means connected to said imaging means provides an output skin deformation signal corresponding to the deformation of said skin. The display transducer may incorporate a plurality of individually actuatable, bendable cantilevered arms mounted in line on a base support, each of the arms having tip ends which are displaceable, upon actuation, to provide a tactile display.

Pending Patents (40)

Systems and Methods for Force-Based Object Manipulation and Haptic Sensations
Vincent Lévesque, Juan Manuel Cruz-Hernandez, Danny Grant, Jamal Saboune, Liwen Wu, Kurt Eerik Stahlberg, and Abdelwahab Hamam
US Patent Application 14/663,406, filed March 19, 2015, pending.

Abstract

Systems and methods for force-based object manipulation and haptic sensations are disclosed. One disclosed method includes the steps of receiving a first signal indicating a location of a user interaction and receiving a second signal indicating a first force. The method also includes, if the location of the user interaction corresponds to an object displayed on a display screen: outputting a first haptic signal to a haptic output device to cause a first haptic effect; and outputting a second haptic signal to the haptic output device to cause a second haptic effect if the first force meets or exceeds a first force threshold.

Systems and Methods for Haptically-Enabled Curved Devices
Vincent Lévesque, Danny Grant, Ali Modarres, and Jamal Saboune
US Patent Application 14/663,405, filed March 19, 2015, pending.

Abstract

One illustrative system disclosed herein includes a curved device that includes a curved outer housing, The illustrative system also includes a sensor configured to detect a user interaction with the curved device and transmit a sensor signal associated with the user interaction. The illustrative system additionally includes a processor in communication with the sensor, the processor configured to: receive the sensor signal from the sensor; determine a user interaction based on the sensor signal, determine a first haptic effect based at least in part on the user interaction, and transmit a haptic signal associated with the first haptic effect. The illustrative system also includes a haptic output device configured to receive the haptic signal and output the first haptic effect.

Systems and Methods for Providing Enhanced Haptic Feedback
Vincent Levesque, Abdelwahab Hamam, Juan Manuel Cruz-Hernandez, and Vahid Khoskava
US Patent Application 14/587,135, filed December 31, 2014, pending.

Abstract

A system includes a haptic output device constructed and arranged to generate a haptic effect to a user of the system, and a chemical delivery device constructed and arranged to deliver a chemical to the user of the system.

Deformable Haptic Wearables with Variable Physical Properties
Juan Manuel Cruz-Hernandez, Jamal Saboune, Abdelwahab Hamam, and Vincent Levesque
US Patent Application 14/585,429, filed December 30, 2014, pending.

Abstract

A haptically-enabled device has an interface for receiving an instruction to provide haptic information. The device includes a tangible element with a physical property including length, stiffness, or texture. A haptic output device is attached to the tangible element and a haptic response module provides haptic information to the haptic output device. The haptic output device causes the tangible element to be altered from a first state to a second state of the physical property.

Systems and Methods for Recording and Playing Back Point-of-View Videos with Haptic Content
Vincent Lévesque, Jamal Saboune and David Birnbaum
US Patent Application 14/585,752, filed December 30, 2014, pending.

Abstract

A system includes a video recorder configured to record a point-of-view video of an event, a sensor configured to sense vibrations associated with the event, a processor configured to synchronize the recorded point-of-view video and the sensed vibrations, and a playback device that includes a display and a haptic output device. The playback device is configured to play back the synchronized point-of-view video and vibrations, and the haptic output device is configured to generate haptic effects based on the vibrations.

Systems and Methods for Providing Haptic Notifications
Vincent Lévesque, David Birnbaum, Amaya Weddle, Robert Lacroix, Juan Manuel Cruz-Hernandez, Danny Grant, Neil Olien, Erin Ramsay, and Ali Modarres
US Patent Application 14/586,012, filed December 30, 2014, pending.

Abstract

A system includes a sensor configured to sense relative movement between a surface and a user's skin; a haptic output device configured to generate a haptic effect at the surface; and a processor configured to receive an input signal from the sensor indicating the sensed movement between the surface and the skin of the user, determine whether a notification is to be sent to the user, and output an output signal, based on the notification, to the haptic output device to generate the haptic effect if it is determined that the notification is to be sent to the user.

Friction Augmented Controls and Method to Convert Buttons of Touch Control Panels to Friction Augmented Controls
Vincent Lévesque, Neil Olien, Christopher Ullrich, David Birnbaum, and Amaya Weddle
US Patent Application 14/585,898, filed December 30, 2014, pending.

Abstract

A system includes a touch control panel configured to receive an input from a user, a controller in signal communication with the touch control panel and configured to control at least one operational setting of a powered apparatus, and a haptic output device in signal communication with the controller and configured to simulate an input button of the touch control panel by outputting a friction effect to the user as the user provides the input.

Systems and Methods for Generating Haptic Effects Based on Eye Tracking
Vincent Levesque, Juan Manuel Cruz-Hernandez, Abraham Dauhajre, Paige Raynes, and Satvir Singh Bhatia
US Patent Application 14/584,098, filed December 29, 2014, pending.

Abstract

A system includes an eye-tracking device configured to track a user's eye gaze while looking at media content comprising an image, and a processor configured to execute one or more computer program modules, including a content determination module that, when executed by the processor, analyzes the media content to identify the image, an eye gaze determination module that, when executed by the processor, determines a gaze location of the user's eye gaze while looking at the image, and an event determination module that, when executed by the processor, determines an event to trigger based on the identification of the image and the gaze location.

Systems and Methods for Haptically-Enabled Holders
Vincent Levesque, Juan Manuel Cruz-Hernandez and Abdelwahab Hamam
US Patent Application 14/582,457, filed December 24, 2014, pending.

Abstract

One illustrative system disclosed herein includes a processor configured to: receive a signal; determine a haptic effect based at least in part on the signal; and transmit a haptic signal associated with the haptic effect. The system further includes a haptic output device in communication with the processor and coupled to a holder, wherein the holder is configured to mechanically couple with an electronic device. The haptic output device is configured to receive the haptic signal and output the haptic effect.

Systems and Methods for Haptically-Enabled Interactions with Objects
Vincent Levesque, Wei Zhu, Eric Gervais, Fengtian An, Eric Lajeunesse, and Johnny Maalouf
US Patent Application 14/577,565, filed December 19, 2014, pending.

Abstract

One illustrative computing device disclosed herein includes a sensor configured to detect a user interaction with a physical object and transmit a sensor signal associated with the user interaction. The illustrative computing device also includes a processor in communication with the sensor, the processor configured to: receive the sensor signal; determine a characteristic of the physical object based on the sensor signal; and determine a function based at least in part on the user interaction and the characteristic. The processor is also configured to determine a haptic effect associated with the function; and transmit a haptic signal associated with the haptic effect. The illustrative computing device further includes a haptic output device in communication with the processor, the haptic output device configured to receive the haptic signal and output the haptic effect.

Systems and Methods for Object Manipulation with Haptic Feedback
Vincent Levesque, Wei Zhu, Eric Gervais, Fengtian An, Eric Lajeunesse, and Johnny Maalouf
US Patent Application 14/577,461, filed December 19, 2014, pending.

Abstract

One illustrative computing device disclosed herein includes a sensor configured to detect a user interaction with a physical object and transmit a sensor signal associated with the user interaction. The illustrative computing device also includes a processor in communication with the sensor, the processor configured to: receive the sensor signal; determine a characteristic of the physical object based on the sensor signal; determine a haptic effect associated with the characteristic; and transmit a haptic signal associated with the haptic effect. The illustrative computing device further includes a haptic output device in communication with the processor, the haptic output device configured to receive the haptic signal and output the haptic effect.

Systems and Methods for Deformation-Based Haptic Effects
Vincent Levesque, Danny Grant, Juan Manuel Cruz-Hernandez, Ali Modarres, and William Rihn
US Patent Application 14/552,987, filed November 24, 2014, pending.

Abstract

One illustrative system disclosed herein includes a deformation sensor configured to detect a deformation of a deformable surface and transmit a first sensor signal associated with the deformation. The system also includes a sensor configured to detect a user interaction with a user input device and transmit a second sensor signal associated with the user interaction. The system further includes a processor configured to: receive the first sensor signal; receive the second sensor signal; execute a function based at least in part on the first sensor signal and the second sensor signal. The processor is also configured to: determine a haptic effect based at least in part on the first sensor signal or the second sensor signal; and transmit a haptic signal associated with the haptic effect to a haptic output device configured to receive the haptic signal and output the haptic effect.

Haptically-Enabled Deformable Device with Rigid Component
Vincent Levesque, Danny Grant and Yiting Wang
US Patent Application 14/512,719, filed October 13, 2014, pending.

Abstract

A device includes a flexible component, such as a display and a rigid component coupled to the flexible component. An input circuit coupled to the flexible component can detect deformation of the flexible component. A response module can provide haptic feedback or another response based on the deformation. In some devices, the rigid component can be made of rigid members attached by a flexible connection and coupled to a motor. The flexible connection can be altered by the motor to cause the flexible component to deform from one state to another.

Systems and Methods for Shape Input and Output for a Haptically-Enabled Deformable Surface
Vincent Lévesque, Danny Grant, Jamal Saboune, Abdelwahab Hamam, and William Rihn
US Patent Application 14/465,005, filed August 21, 2014, pending.

Abstract

One illustrative computing device disclosed herein includes a first sensor configured to detect a position associated with a deformable surface and transmit a sensor signal associated with the position; and a processor in communication with the sensor, the processor configured to: receive the sensor signal; determine a haptic effect based at least in part on the sensor signal; and transmit a haptic signal associated with the haptic effect. The illustrative computing device also includes a haptic output device in communication with the processor, the haptic output device configured to receive the haptic signal and output the haptic effect.

Systems and Methods For Determining Haptic Effects For Multi-Touch Input
Vincent Lévesque and Juan Manuel Cruz-Hernandez
US Patent Application 14/336,548, filed July 21, 2014, pending.

Abstract

A system of the present disclosure may include a haptic output device configured to output a haptic effect to a touch surface; a touch sensitive input device configured to detect a first user interaction and transmit a first sensor signal, the touch sensitive input device further configured to detect a second user interaction and transmit a second sensor signal, wherein at least part of the first user interaction occurs at the same time as the second user interaction; a processor in communication with the sensor, the processor configured to: receive the first sensor signal and the second sensor signal; determine a haptic effect based in part on the first user interaction and the second user interaction; transmit a haptic signal associated with the haptic effect to the haptic output device.

Systems and Methods to Generate Haptic Feedback for Skin-Mediated Interactions
Vincent Lévesque and Juan Manuel Cruz-Hernandez
US Patent Application 14/331,592, filed July 15, 2014, pending.

Abstract

A system includes a sensor configured to sense an input at a skin surface, a processor configured to receive an output signal from the sensor and generate a haptic control signal based on the output signal, and a haptic output device configured to generate a haptic effect based on the haptic control signal.

Systems and Methods for Multi-Output Electrostatic Haptic Effects
Ali Modarres, Juan Manuel Cruz-Hernandez, Christopher Ullrich, Danny Grant, Neil Olien, and Vincent Lévesque
US Patent Application 14/322,701, filed July 2, 2014, pending.

Abstract

One illustrative system disclosed herein includes a processor configured to determine a haptic effect, wherein the haptic effect includes a static ESF effect or a confirmation ESF effect; and transmit a haptic signal associated with the haptic effect. The illustrative system also includes an ESF controller in communication with the processor, the ESF controller configured to receive the haptic signal, determine an ESF signal based at least in part on the haptic signal, and transmit the ESF signal. The illustrative system further includes an ESF device in communication with the ESF controller, the ESF device including an ESF cell and configured to receive the ESF signal and output the haptic effect.

Mobile Device with Motion Controlling Haptics
William Rihn, David Birnbaum, Yiting Wang, Vincent Lévesque, and Danny Grant
US Patent Application 14/306,600, filed June 17, 2014, pending.

Abstract

A haptically enabled device includes a haptic output device used to control motion. The haptically enabled device determines a desired motion, and then generates a haptic effect on the haptic output device to cause the desired motion.

Haptic Notification Manager
Vincent Lévesque, Danny Grant, Jean-Francois Blanchard-Dionne, Amaya Weddle, and Juan Manuel Cruz-Hernandez
US Patent Application 14/291,735, filed May 30, 2014, pending.

Abstract

A notification manager is provided that manages notifications for a user. The notification manager detects a notification event and determines a context of the notification event. The notification manager determines a target device for the notification event based on the determined context and one or more available displays. The notification manager then generates a notification on one or more of the one or more available displays to direct the user's attention to the target device.

Systems and Methods for Providing Haptic Feedback for Remote Interactions
Vincent Lévesque, Ali Modarres, Danny Grant, Jean Francois Dionne, and David Birnbaum
US Patent Application 14/275,216, filed May 12, 2014, pending.

Abstract

A system includes a first electronic device and a second electronic device. The first electronic device includes a sensor configured to sense or property experienced by the first electronic device, and a transmitter configured to transmit a signal based on output from the sensor. The second electronic device is in signal communication with the first electronic device. The second electronic device includes a receiver configured to receive the transmitted signal, a detector configured to determine an object that a user of the second device is focusing on, a processor configured to generate a haptic signal representative of the transmitted signal if it is determined that the object the user is focusing on corresponds with a location of the first electronic device, and a haptic output device configured to receive the haptic signal and generate a haptic effect to the user.

Systems and Methods for Viewport-Based Augmented Reality Haptic Effects
Vincent Lévesque and Danny Grant
US Patent Application 14/269,357, filed May 5, 2014, pending.

Abstract

One illustrative system disclosed herein includes a display configured to receive a display signal and output an image, and an image capture device configured to capture an area image and transmit an image signal. The illustrative system also includes a processor in communication with the image capture device and the display, the processor configured to: receive the image signal; determine a virtual object based in part on the image signal; determine the display signal based in part on the image signal, wherein the display signal includes data associated with the virtual object; determine a haptic effect based at least in part on the virtual object; and transmit a haptic signal associated with the haptic effect. The illustrative system further includes a haptic output device configured to receive the haptic signal and output the haptic effect.

Wearable Device with Flexibly Mounted Haptic Output Device
Danny Grant, Vincent Lévesque, Elena Redelsheimer, and Daniel Gregory Parker
US Patent Application 14/243,313, filed April 2, 2014, pending.

Abstract

A system includes a wearable device configured to be worn by a user. The wearable device includes a flexible mounting, and a haptic output device connected to the flexible mounting. The system includes a signal generator configured to generate a haptic signal and communicate the haptic signal to the haptic output device. The haptic signal includes a short duration control pulse configured to create a small deformation sensation to the user.

System and Method for a Haptically-Enabled Deformable Surface
Juan Manuel Cruz-Hernandez, Danny Grant, Vincent Lévesque, and Ali Modarres
US Patent Application 14/222,518, filed March 21, 2014, pending.

Abstract

One illustrative system disclosed herein includes a first haptic output device configured to receive a first haptic signal and output a first haptic effect to a deformable surface and a second haptic output device configured to receive a second haptic signal and output a second haptic effect to the deformable surface. The illustrative system further includes a processor coupled to the first haptic output device and the second haptic output device, the processor configured to: determine an event, determine a first haptic effect and a second haptic effect based at least in part on the event, transmit a first haptic signal associated with the first haptic effect to the first haptic output device, and transmit a second haptic signal associated with the second haptic effect to the second haptic output device.

Haptic Feedback for Interactions with Foldable-Bendable Displays
Ali Modarres, Vincent Lévesque, Danny Grant, and Juan Manuel Cruz-Hernandez
US Patent Application 14/222,303, filed March 21, 2014, pending.

Abstract

A flexible device includes a bendable-foldable display that has bendable flaps connected by a hinge. The display has sensors for detecting a folding characteristic between the at least two flaps and for detecting a bending characteristic in at least one flap. The display has a haptic system with haptic output devices, where the haptic system receives input from the sensors indicating deformation of the bendable-foldable display device. A flexible device also includes bendable, foldable, or rollable displays that have sensors and actuators to augment user interaction with the device. Based on one or more measurements provided by the input, the haptic system interprets the input to determine deformation characteristics of the bendable-foldable display device. The haptic system generates haptic feedback based on the deformation characteristics.

Method and System For Providing Haptic Effects Based on Information Complementary to Multimedia Content
Vincent Lévesque, Ali Modarres, Juan Manuel Cruz-Hernandez, and Jamal Saboune
US Patent Application 14/145,650, filed December 31, 2013, pending.

Abstract

The present disclosure is generally directed to systems and methods for providing haptic effects based on information complementary to multimedia content. For example, one disclosed method includes the steps of receiving multimedia data comprising multimedia content and complementary data, wherein the complementary data describes the multimedia content, determining a haptic effect based at least in part on the complementary data, and outputting the haptic effect while playing the multimedia content.

Systems and Methods for a Haptically-Enabled Projected User Interface
Vincent Lévesque, Ali Modarres, Amaya Weddle, and David Birnbaum
US Patent Application 14/143,675, filed December 30, 2013, pending.

Abstract

One illustrative system disclosed herein includes a sensor configured to detect a user input associated with a projected user interface and transmit a sensor signal associated with the user interaction, and a processor in communication with the sensor, the processor configured to: receive the sensor signal, determine a remote haptic effect based at least in part on the user input, generate a haptic signal based at least in part on the remote haptic effect, and transmit the haptic signal. The illustrative system further includes a haptic output device in communication with the processor, the haptic output device configured to receive the haptic signal and output the remote haptic effect.

Method and Apparatus of Body-Mediated Digital Content Transfer and Haptic Feedback
Vincent Lévesque, Ali Modarres, Amaya Weddle, David Birnbaum, Juan Manuel Cruz-Hernandez, and Henry Da Costa
US Patent Application 14/106,275, filed December 13, 2013, pending.

Abstract

The disclosure relates to systems and methods of transferring/storing digital content, and/or providing haptic feedback via wearable devices directly or indirectly responsive to an event such as a communication, an event occurring in an electronic environment and/or physical environment, the transfer of digital content, and/or other events. Wearable devices may be worn at various locations on a body of the user. Each location may be associated with a wearable device. Different haptic feedback may be provided at different locations based on the event such that the events may be distinguished or otherwise identified based on the location at which haptic feedback is provided. The locations may be stored in a configuration of wearable devices that act as a mapping of the wearable devices. The locations and/or haptic feedback itself may be varied based on the context and/or properties such as size of digital content or status of the transfer.

Systems and Methods for Generating Friction and Vibrotactile Effects
Vincent Lévesque, Ali Modarres, Juan Manuel Cruz-Hernandez, Amaya Weddle, David Birnbaum, and Danny Grant
US Patent Application 14/090,922, filed November 26, 2013, pending.

Abstract

A system includes a sensor configured to sense an input received from a user at a surface; a first haptic output device configured to generate a first haptic effect at the surface based on the input received from the user, the first haptic effect comprising a friction effect at the surface; a second haptic output device configured to generate a second haptic effect, the second haptic effect comprising vibrotactile feedback; and a processor configured to receive an input signal from the sensor and output a first drive signal to the first haptic output device to generate the first haptic effect and to output a second drive signal to the second haptic output device to generate the second haptic effect.

Systems and Methods for Generating Haptic Effects Associated With Audio Signals
Juan Manuel Cruz-Hernandez, Jamal Saboune, Vincent Lévesque, and Ali Modarres
US Patent Application 14/078,445, filed November 12, 2013, pending.

Abstract

Systems and methods for generating haptic effects associated with audio signals are disclosed. One disclosed system for outputting haptic effects includes a processor configured to: receive an audio signal; determine a haptic effect based in part on the audio signal by: identifying one or more components in the audio signal; and determining a haptic effect associated with the one or more components; and output a haptic signal associated with the haptic effect.

Systems and Methods for Haptic Fiddling
Danny Grant, Vincent Lévesque, Amaya Weddle, David Birnbaum, Juan Manuel Cruz-Hermandez, and Jamal Saboune
US Patent Application 14/010,973, filed August 27, 2013, pending.

Abstract

Systems and methods for haptic fiddling are disclosed. In one embodiment, a sensor signal indicating an interaction with an electronic device is received by the electronic device. In response to receiving the sensor signal, the electronic device may output an output signal. The output signal may be output to a display, a haptic output device, an audio device, and/or another output device. The output signal can correspond to a modality. After outputting the output signal, another sensor signal indicating another interaction with the electronic device may be received by the electronic device. This interaction may be associated with the previously-received interaction. In response to receiving this interaction, the electronic device may output another output signal to the same output device(s) and/or different output device(s). The second output signal may correspond to one or more different modalities than the previous output signal.

Usable Hidden Controls with Haptic Feedback
Vincent Lévesque
US Patent Application 13/945,498, filed July 18, 2013, pending.

Abstract

A system configured to activate a hidden control includes a sensor configured to sense an input by a user. The input includes a gesture. The system includes a haptic output device configured to output a haptic feedback as the input is being sensed, and a processor configured to determine if the sensed input corresponds to a predetermined gesture associated with activation of a control, and if the sensed input corresponds to the predetermined gesture to activate the control.

Systems and Methods For Perceptual Normalization of Haptic Effects
Vincent Lévesque, Ali Modarres, Neil Olien, Danny Grant, Erin Ramsay, David Birnbaum, and Amaya Weddle
US Patent Application 13/933,915, filed July 2, 2013, pending.

Abstract

Systems and methods for perceptual normalization of haptic effects are disclosed. One system may include a first sensor configured to detect a user interaction with a touch surface and transmit a first sensor signal associated with the user interaction; a second sensor configured to detect a feature associated with the touch surface and transmit a second sensor signal associated with the feature; a processor in communication with the first sensor and the second sensor, the processor configured to: determine a first haptic effect based on the user interaction; determine a modified haptic effect based in part first haptic effect and on the feature; output a haptic signal associated with the modified haptic effect; and a haptic output device in communication with the processor and coupled to the touch surface, the haptic output device configured to receive the haptic signal and output the modified haptic effect to the touch surface.

Systems and Methods For Providing Mode or State Awareness With Programmable Surface Texture
Vincent Lévesque, David Birnbaum, Juan Manuel Cruz-Hernandez, and Amaya Weddle
US Patent Application 13/830,125, filed March 14, 2013, pending.

Abstract

Systems and methods of mode or state awareness with programmable surface texture are disclosed. For example, in one embodiment, a system of the present disclosure may include a sensor configured to detect an interaction with a touch surface and transmit a sensor signal associated with the interaction; a processor in communication with the sensor, the processor configured to: determine a mode of operation; control at least one feature of a system based on the mode of operation and the interaction; determine a simulated texture associated with the mode of operation; output a haptic signal associated with the simulated texture; and a haptic output device in communication with the processor and coupled to the touch surface, the haptic output device configured to receive the haptic signal and simulate the texture on the touch surface based in part on the haptic signal.

System and Method For Feedforward and Feedback With Haptic Effects
Vincent Lévesque and Juan Manuel Cruz-Hernandez
US Patent Application 13/830,162, filed March 14, 2013, pending.

Abstract

Systems and methods for feedforward and feedback with haptic effects are disclosed. One such system may include a sensor configured to detect an interaction with a touch surface and transmit a sensor signal associated with the interaction; a processor in communication with the sensor, the processor configured to: determine an operation available on a device, the operation associated with a first user interaction; determine a simulated texture associated with the operation; output a haptic signal associated with the simulated texture; determine whether to perform the operation based on a second user interaction; and a haptic output device in communication with the processor and coupled to the touch surface, the haptic output device configured to receive a haptic signal and simulate a texture on the touch surface based in part on the haptic signal.

Contactor-Based Haptic Feedback Generation
Vincent Lévesque
US Patent Application 13/803,103, filed March 14, 2013, pending.

Abstract

A system that generates haptic effects senses contact on an interface by a contactor/object. The system then determines one or more properties of the contactor and generates a type of haptic effect in response to the contact. The type of haptic effect is based at least on the one or more properties.

Systems and Methods for Enhanced Television Interaction
Danny Grant, Pamela Pimentel, Eric Gervais, Vincent Lévesque, Aaron Kapelus, Mina Chang, Daniel Parker, David Birnbaum, Li Jiang, and Stephen Rank
US Patent Application 13/827,850, filed March 14, 2013, pending.

Abstract

Systems and methods for enhanced television interaction are disclosed. For example, one disclosed method includes receiving notification information, the notification information indicating an event associated with video content displayed by a television device; determining a haptic effect associated with the notification information; generating and transmitting a haptic signal to a haptic output device, the haptic signal configured to cause the haptic output device to output the haptic effect.

Friction Modulation for Three Dimensional Relief in a Haptic Device
Robert Lacroix and Vincent Lévesque
US Patent Application 13/793,843, filed March 11, 2013, pending.

Abstract

A haptic device includes a display configured to display an image, a haptic output device configured to generate a haptic effect to a user when the user interacts with the display, and a processor configured to receive information related to the image displayed on the display. The processor is also configured to create a friction based haptic effect map associated with the image displayed on the display, and generate a signal to the haptic output device to output the haptic effect when the user interacts with the display when the image is displayed on the display, the haptic effect being configured to simulate a feel of the image in three dimensions.

Haptic Sensations as a Function of Eye Gaze
Danny Grant and Vincent Lévesque
US Patent Application 13/793,966, filed March 11, 2013, pending.

Abstract

Systems, electronic devices, and methods provide haptic sensations as a function of eye gaze. A system may include a detector configured to determine a direction of an eye gaze of a user of the system, a processor configured to generate signal representative of a haptic effect based on the direction of the eye gaze, and a haptic output device configured to receive the signal from the processor and output the haptic effect to the user. A method for providing a haptic effect to a user of a system may include determining a direction of an eye gaze of the user of the system, generating a haptic effect based on the direction of the eye gaze, and outputting the haptic effect to the user.

Method and Apparatus for Providing Haptic Cues for Guidance and Alignment with Electrostatic Friction
Vincent Lévesque and Juan Manuel Cruz-Hernandez
US Patent Application 13/782,771, filed March 1, 2013, pending.

Abstract

A haptic effect enabled device for producing a haptic effect. In some cases, the haptic effect may represent a component of a spatial pattern represented on a surface of the haptic effect enabled device. In some cases, the haptic effect enabled device may comprise a haptic output device, a drive module, and a drive circuit. The drive module may receive information indicative of a location of a touch input at the surface and determine whether the touch input's location corresponds with a location of one of multiple components of the spatial pattern. The drive module may generate a drive signal that the drive circuit then applies to the haptic output device to generate the haptic effect.

Mobile Device with Instinctive Alerts
David Birnbaum, Vincent Lévesque and Danny Grant
US Patent Application 13/772,682, filed February 21, 2013, pending.

Abstract

A haptically enabled system receives information and determines that a user of the system should be notified of the information. The system further receives and analyzes inputs to determine a context of the user. The system then generates an instinctive alert based on the context, where the instinctive alert is configured to impart a desired emotion on the user. The system then displays the instinctive alert based on the context.