2010 Second International Conference on Games and Virtual Worlds for Serious Applications

Augmented Reality Games for Upper-Limb Stroke Rehabilitation

J. W. Burke, M. D J. McNeill, D. K. Charles,

P. J. Morrow

School of Computing and Information Engineering, University of Ulster, Coleraine,

Northern Ireland burke-j5@email.ulster.ac.uk

{mdj.mcneill, dk.charles, pj.morrow}@ulster.ac.uk

J. H. Crosbie, S. M. McDonough

School of Health Sciences, University of Ulster, Jordanstown, Northern Ireland

{j.crosbie, s.mcdonough}@ulster.ac.uk

Abstract— Stroke is the number one cause of severe physical disability in the UK. Recent studies have shown that technologies such as virtual reality and imaging can provide an engaging and motivating tool for physical rehabilitation. In this paper we summarise previous work in our group using virtual reality technology and webcam-based games. We then present early work we are conducting in experimenting with desktop augmented reality (AR) for rehabilitation. AR allows the user to use real objects to interact with computer-generated environments. Markers attached to the real objects enable the system (via a webcam) to track the position and orientation of each object as it is moved. The system can then augment the captured image of the real environment with computer-generated graphics to present a variety of game or task-driven scenarios to the user. We discuss the development of rehabilitation prototypes using available AR libraries and express our thoughts on the potential of AR technology.

Keywords: serious games, rehabilitation, stroke, augmented reality, video-capture.

BACKGROUND ON?STROKE

Stroke is a major problem in the UK. The British Heart Foundation estimate that there are 1.1 million people living in the UK who have had a stroke and that 111,000 first-time strokes occur each year [1]. Stroke can often cause severe physical disability such as attention deficiency, pain, weakness and paralysis, often on one side of the body. Such impairments can result in loss of ability to carry out typical day-to-day activities [2]. It has been reported that the upper limb of the affected side can remain weak in up to 66% of stroke survivors [3].

Post-stroke rehabilitation aims to help people with stroke restore movement by carrying out exercises on a regular basis. It has been shown [4] that?early,?intensive?rehabilitation therapy which incorporates?active functional tasks?in an?enriched environment?results in more positive outcomes. It is not uncommon for stroke survivors to experience depression and therefore they may find it difficult to concentrate on a therapy programme. People with stroke commonly report that traditional rehabilitation tasks can be mundane and boring due to their repetitive nature.

978-0-7695-3986-7/10 $26.00 ? 2010 IEEE DOI 10.1109/VS-GAMES.2010.21

In recent years, studies have been investigating how technology such as virtual reality and motion detection systems could address some of the problems associated with traditional stroke rehabilitation (see [5, 6] for reviews of the field). In an ongoing project, we are investigating how video game design can be applied to rehabilitation systems and how it can potentially be beneficial to people with stroke. We have been incorporating these game design principles with technology such as virtual reality and imaging to create rehabilitation gaming systems. More recently, we have focused on the use of low-cost webcams as the interaction technology for rehabilitative games and are investigating the potential benefits which augmented reality may offer.

GAME?DESIGN?PRINCIPLES FOR?REHABILITATION?Video games are well known as a means of providing

excitement, stimulation, motivation, challenge and enjoyment to those who play them. Such motivation and enjoyment could prove highly beneficial in a rehabilitation context, where tasks are often mundane and repetitive by nature. It has been suggested in the literature that the designers of rehabilitation tasks could benefit from examining the principles and formulas of video game design [6]. Our group is currently working to develop guidelines for optimising engagement in games for stroke rehabilitation. We have identified three aspects of game design which are important for optimising the user experience in a rehabilitation context:?meaningful play,?challenge?and conservative handling of?failure:

Meaningful play?emerges from a game through the relationship between a player’s actions and the system’s outcome [7]. A good game should provide clear, consistent and meaningful feedback in response to the player’s actions. Feedback can be communicated aurally (sound effects, speech), visually (ability to see arm/hand in the game, numerical scores, progress bars) and through haptic technology (vibration). Meaningful play is important to rehabilitation as it is crucial that a person with stroke playing the game is aware of their goals, what actions they need to take to achieve those goals and whether or not they are achieving those goals (both

75

short-term and long-term) in order for them to engage effectively with the rehabilitation game.

Due to a number of factors such as poor motor control and possible unfamiliarity with playing video games, the likelihood of people with stroke experiencing?failure?in a game, certainly initially, is high. Handling failure in a positive way by encouraging and rewarding all engagement with the game will make it more likely that players will not feel discouraged should they not perform as well as they had hoped.

Finally, games for rehabilitation can be designed to dynamically adapt the level of?challenge?depending on the performance of the player. This is important to ensure that the person with stroke is presented with a level of challenge suitable to their skills. If the game is too difficult, the player could become frustrated and give up; similarly, if the game is too easy, the player could quickly become bored [8].

A more detailed analysis of these principles can be found in Burke et al [9].

STROKE?REHABILITATION AT THE?UNIVERSITY OF

ULSTER

Our research group at the University of Ulster comprises people from both Computing and Rehabilitation Sciences. We have developed several systems for upper limb stroke rehabilitation, using both electromagnetic sensor and imaging technologies. These systems incorporate our ideas on utilising the principles of game design to improve user engagement. This section gives a brief overview of a few of these rehabilitation systems.

Early work in our group involved the development of a 3D virtual reality system which used electromagnetic sensors to track real-world user movement. Both functional and game-like scenarios were designed to provoke reach, grasp, move and release – movements which are relevant to everyday tasks. Details of this work can be found in [10]. While the VR system may be appropriate for clinical use it is unlikely that it would be appropriate for home use, due to cost and level of technical expertise required to configure the system ready for use.

Following on from our VR rehabilitation system, a low-cost rehabilitation gaming system was developed which uses a standard webcam to acquire user movement data. This offers the potential for home-based stroke rehabilitation, being low- cost and requiring little technical knowledge to set up. The system was developed using Microsoft’s XNA platform (C#) and works with any DirectShow-compatible webcam. Games were designed by taking the game design principles covered in Section 2 into account. Details of the system can be found in [11]. Results from an initial phase of these studies can be found in [9] – further studies with these games are currently underway.

MARKER-BASED?AUGMENTED?REALITY FOR?STROKE?REHABILITATION

Following the design of our low-cost webcam-based gaming system, we have recently started to investigate the potential of using augmented reality (AR) techniques for upper-limb rehabilitation games. AR typically refers to the viewing of a real world scene which has been augmented with virtual (computer-generated) elements (Figure 1). In marker- based AR techniques, markers are attached to real objects which enable the system (via the webcam) to track the position and orientation of each object as it is moved. The system can then augment the captured image of the real environment with computer-generated graphics to present a variety of game or task-driven scenarios to the user.

(b)

Figure 1. (a) An AR marker in the real world. (b) A virtual mug overlaid using the data from the AR tracking algorithm to determine position and orientation.

AR offers the advantages of being able to use real objects (which have AR markers attached) to interact with virtual environments/games. These real objects can vary in size, shape and mass, which may result in players acquiring muscle strength and motor skills which are more transferable to everyday life than those associated with activities in purely virtual environments, where the user is typically not holding any physical object. Although earlier work in our group experimented with the user using physical objects (tracked with VR sensors) to interact with a virtual environment [10], augmented reality allows users to see a live view of a real world environment, augmented by computer-generated imagery.

There are several open-source libraries available online which provide marker-based tracking for augmented reality applications, such as?ARToolKit?[12],?ARToolKitPlus?[13] and?ARTag?[14]. We selected?ARToolKitPlus?as the most appropriate library for our needs due to restrictions or licensing issues with the other libraries. We ported an OpenGL C# version of the library to the Microsoft XNA platform to allow us to use coding assets from our previous XNA-based webcam games. Several AR stroke rehabilitation game prototypes are currently in development.

Brick’a’Break

The first prototype developed is a game similar to Atari’s?Breakout. The player is presented with a row of bricks at the top of the playfield which they must clear by rebounding a ball with their paddle, which they control by moving a real-world physical object (a cube, in this instance, although this could be any object) with an AR marker attached (Figure 2). The prototype begins by using an A4-sized marker to give the

system a ‘profile’ (relative position and orientation) of the player’s desk surface. The virtual representation of the game is then shown on the on-screen image on top of the player’s desk. The object which the player is using as their paddle is augmented on-screen with a virtual cube to aid in showing the position of the object in the virtual world. A ball moves around the playfield which the user must rebound with their paddle. When the ball collides with one of the bricks at the top of the field, the brick breaks and disappears, rebounding the ball in the opposite direction of the collision. The goal of the game is to break all of the bricks in each level. Unlike in the original Breakout game, when a life was lost when the ball was missed, here, the player does not lose a life for missing a ball – rather, there is a short pause while the ball is re-spawned. Also in line with the principle of handling?failure?conservatively, there is no time limit for the player to clear a row of bricks; however the player’s score is determined by the time taken, with shorter times resulting in higher scores. Future versions of the game will feature multiple levels with differing brick patterns, different ball speeds and different playfield sizes to create a suitable level of?challenge, as well as including effective visual and auditory feedback for the player, aiding to create?meaningful play. This game may help improve reach and grasp motions.

(a)

Figure 2.?Brick’a’Break: (a) The real world scene, with the player holding the cube (AR marker visible). (b) Real world image augmented with the virtual scene in the game.

Shelf Stack

In this game, the player can have several real world objects of different shape, size and weight to which AR markers are attached. Each of these real world objects has a corresponding virtual object which can be seen on-screen; this virtual object may look different to the real world object, for example a real- world cube may be represented as a virtual teapot. The player is also presented with a set of virtual shelves in the scene, each holding several rings (Figure 3). In the initial levels, following a profiling of the user’s desktop (as in the previous game), one of the objects will be highlighted with a red ring. One of the rings on the shelf will also be highlighted in red, indicating the designated target (Figure 3a). The player must pick up the designated object and move it to the highlighted ring. Once the object is placed inside the red ring, the ring turns green, indicating success (Figure 3b). The player must then move the object to a new position on the desk, which the system will indicate with another red ring. Once the object is placed at this new position (Figure 3c), another object will be selected at random. The player must again correctly select and position the respective object accordingly. Each correct action is rewarded with a point; additional points are awarded for the speed at which the player carries out the task. The player must try and score as many points as possible within a designated time limit. This method of scoring means that the player will never?fail; rather they will be encouraged to increase their score over multiple play sessions. The level of?challenge?can be increased through the addition of more shelves, placing the shelves higher, shortening the time limit and requiring a higher score before allowing progression to the next level. Future versions of this game will include additional visual and auditory feedback which aim to invoke?meaningful play. This game may help rehabilitate reach, grasp, lift and release motor functions, as well as requiring cognitive skills to discriminate the correct object and placement of the object.

A number of issues have been identified which must be addressed in order for AR to provide an effective platform for user-friendly rehabilitation games:

Camera position – To support good hand-eye coordination, the ideal position of the webcam is as close to the eye position of the player as possible. Since stroke typically results in one side of the body having motor problems the camera can be placed on the opposite side. For games which require the involvement of both upper limbs placement can be an issue.

Depth perception – Our initial evaluation showed that it can be difficult to perceive depth in the scene. We intend to attempt to improve this problem by including shadows or visual aids.

Occluding a marker or having insufficient lighting in a room can cause issues with the AR tracking. Improving prediction and smoothing algorithms may help to minimise this issue.

Further work on these prototypes is ongoing and we hope to evaluate the games in the near future with people with stroke through single patient case studies, followed by a larger study.

shape and weight in an augmented reality world offers the potential for developing high quality motor skills which may be more transferable to activities of daily living than other techniques. The prototypes developed and presented here have raised a number of issues which require further work.

Augmented reality has the potential to bring about interesting and rewarding outcomes for upper limb stroke rehabilitation. The technology will almost certainly improve in the future to allow low-cost markerless tracking, opening up the potential for high quality user-friendly home stroke rehabilitation.

(a)

(b)

Figure 3.?Shelf Stack: Each visible virtual object is augmented on top of a physical object. These screens show: (a) the initial selection of an object which the player must pick up and move to the highlighted ring on the shelf; (b) the correct positioning of the object on to the shelf; (c) the correct placement of the object on the designated area of the desk.

CONCLUSION

In this paper we have discussed our ideas for incorporating the principles of game design for games for motor rehabilitation. We have also reported on early work evaluating augmented reality technology for upper limb stroke rehabilitation. Enabling users to handle real objects of different

ACKNOWLEDGMENT

This work was supported by a Department for Employment and Learning (DEL, Northern Ireland) research studentship.

REFERENCES

British Heart Foundation and Stroke Association. Stroke statistics, August 2009. Available:?http://www.heartstats.org/datapage.asp?id=8615.

R. Anderson, The Aftermath of Stroke: The Experience of Patients and their Families, 1st ed., Cambridge University Press, 1992, pp. 262.

J. H. Van der Lee, R. C. Wagenaar and G. J. Lankhorst, "Forced use of the upper extremity in chronic stroke patients," Stroke, vol. 30, pp. 2369-2375, 1999.

D. T. Wade, V. A. Wood and R. L. Hewer, "Recovery after stroke - the first three months," Journal of Neurology, Neurosurgery, and Psychiatry, vol. 48, pp. 7-13, 1985.

J. H. Crosbie, S. Lennon, J. R. Basford and S. M. McDonough, "Virtual Reality in Stroke Rehabilitation: Still More Virtual Than Real," Disability & Rehabilitation, vol. 29, pp. 1139-1146, 2007.

A. Rizzo and G. J. Kim, "A SWOT Analysis of the Field of Virtual Reality Rehabilitation and Therapy," Presence: Teleoperators and Virtual Environments, vol. 14, pp. 119-146, 2005.

K. Salen and E. Zimmerman, Rules of Play: Game Design Fundamentals. Cambridge, Massachusetts: MIT Press, 2003, pp. 688.

M. Csikszentmihalyi and I. S. Csikszentmihalyi, Optimal Experience: Psychological Studies of Flow in Consciousness. Cambridge University Press, 1988.

J. W. Burke, M. D. J. McNeill, D. K. Charles, P. J. Morrow, J. H. Crosbie and S. M. McDonough, "Optimising engagement for stroke rehabilitation using serious games," The Visual Computer, vol. 25, issue 12, pp. 1085-1099, 2009.

M. Ma, M. McNeill, D. Charles, S. McDonough, J. Crosbie, L. Oliver and C. McGoldrick, "Adaptive virtual reality games for rehabilitation of motor disorders," in Lecture Notes in Computer Science , vol. 4555/2007, Springer Berlin / Heidelberg, 2007, pp. 681-690.

J. W. Burke, M. D. J. McNeill, D. K. Charles, P. J. Morrow, J. H. Crosbie and S. M. McDonough, "Serious games for upper limb rehabilitation following stroke," in IEEE International Conference in Games and Virtual Worlds for Serious Applications, 2009, pp. 103-110.

H. Kato. ARToolKit. Available:?http://www.hitl.washington.edu/artoolkit/.

D. Wagner. ARToolKitPlus. Available:http://studierstube.icg.tu- graz.ac.at/handheld_ar/artoolkitplus.php.

M. Fiala. ARTag. Available:http://www.artag.net/.