Let’s Get Non-verbal, Electronically

Digital communication hasn’t the richness of face-to-face conversation because it cannot communicate the non-verbal cues and contextual information that are so important to us all.

But that will not be a problem for much longer. The PASION project, an EU-funded research effort, has developed a suite of tools to communicate precisely this kind of information.

And even more remarkable than the technological advance are the applications that the technology enables, including new kinds of online gaming, new forms of groupware, even tele-psychiatry.

One of the most compelling applications is “augmented social networking”. Over the last couple of years, services like Facebook or MySpace have been growing enormously but there are still things they can’t do. The tools PASION is developing offer new ways of showing your friends where you are, what you’re doing, how you feel; new ways of looking at your friends list, new ways of communicating.

Non-verbal office

But PASION is not just for play. “Many problems in collaborative work derive not from the need to communicate and share explicit information, already met by current groupware and communication tools, but from higher-level problems in coordinating these processes,” explains Richard Walker, chief dissemination officer of the PASION project.

“For example, misunderstandings, or the erroneous attribution of intentions can cause problems for groups.” PASION believes that augmented communication will limit or reduce this sort of problem.

The project has developed a prototype that runs on a mobile phone and provides information that users can exploit to coordinate their work. It can show information like user availability, or indicators and visualisations that illustrate the social position of a user in the group.

It also offers information on user mood. Currently, this works via self-reporting, but soon it will be generated automatically. It can even tell you what the user is doing at a particular moment.

PASION has already incorporated augmentation in many desktop tools, typically used by knowledge workers, such as MS Outlook, iGoogle, Skype, and Thunderbird. More are on the way.

Games people play

Gaming, too, is a rich area for exploitation, with non-verbal cues augmenting games in the same way as the successful Nintendo Wii uses motion sensors to turn video games into a social event.

For example, the Familiars game developed within the project tracks user status within a social network and gives them a score for each improvement in their status. This could be really big. Already people on social networking websites like Facebook and MySpace suffer from artificial ‘friendship inflation’ as they add contacts pell-mell to their friends list.

But few of those friends are ‘real’ – many may be people the user never really meets. Often users gain prestige because of the quantity of friends they have, not the quality. Familiars tracks the quality. For example, is the user a significant node for many others? Do members in the group contact the user regularly? These are the factors used to calculate the final score.

Facial communication and the tele-shrink

There are vertical applications, too. One of the most compelling is tele-psychiatry. Using special software to analyse facial expressions, and a voice link, psychiatrists could work with their patients at a distance, helping them to deal with their problems.

This is not the only vertical application, either. Non-verbal cues about learners’ arousal or about the way a group is interacting could be enormously useful to online moderators or teachers in e-learning applications.

The range and variety of applications are impressive, but when will we see them on the high street?

“There are, of course, [parts] that could use more work, but we have developed elements of all these systems, and different partners will take them forward in different ways,” explains Walker.

For example, simple sensors that indicate physiological state, like the heart rate, are simple, cost little to produce, are well accepted among younger users, and could be deployed cheaply and easily for social networking or gaming applications; these are tools we could see deployed in the near term.

Similarly, there are software modules almost ready to deploy in groupware, and that is an area that could be exploited relatively quickly. PASION’s telecom partner is considering deploying some of the research for mobile gaming, and this work could also appear quickly.

Other applications will take longer, or be reserved for specialist applications. PASION’s facial recognition system, for example, is currently very expensive but there are already applications where it could be applied – such as the tele-psychiatry one.

In all, we should soon see the sound of silence become part of our digital life.

The PASION project received funding from the Future and Emerging Technologies (FET) strand of the Sixth Framework Programme for research.

This is the second of a two-part special feature on the PASION project.