Over the last decades, embedded computer systems have become powerful and widespread with remarkable success. Besides traditional computers, such as desktops, laptops,
smartphones, and servers, such systems have become part of nearly every technical appliance, for example, cars, televisions, and washing machines and thereby, an essential part
of our lives. A common phrase for such appliances is “smart device”, a term which encompasses equipment to which one can digitally connect in order to exchange information
and commands. Further, it can potentially sense its environment and process as well as act
upon this measurement.
Although computers and the devices containing them play such an important role,
the way in which we interact with them has not changed much since the early days.
Humans are required to control them in a manner very different from human-to-human
communication. In particular, the possibilities to provide input to a smart device are
mostly limited to traditional interfaces, such as buttons, knobs, or keyboards; or graphical
representations thereof on displays.
Technological progress in recent years in hardware, as well as in algorithmic methods, e.g. in machine learning, enables novel solutions for human-computer (or in fact
human-device) interaction. Only recently, the use of speech has become practical and
is used on smartphones as well as for home devices, incorporating a modality in the
interaction process that is innate to humans and rich in expressiveness. However, for
natural communication, other modalities, such as gestures, complement speech and may
do so in human-computer communication, enabling simple and spontaneous interactions
and avoiding the known social awkwardness of having to talk to devices. The adoption
of speech and also other commonly used interaction methods, such as touch input on
smartphones, indicate the relevance of considering further modalities in addition to the traditional ones. This dissertation contributes to further bridging the interaction gap between
humans and smart devices by exploring solutions in the following areas:

(i) Wearable gesture recognition based on electromyography (EMG): The touch of
our fingers is widely used for interaction. However, most approaches only consider binary
touch events. We present a method, which classifies finger touches using a novel neural
network architecture and estimates their force based on data recorded from a wireless EMG
armband. Our method runs in real time on a smartphone and allows for new interactions
with devices and objects, as any surface can be turned into an interactive surface and
additional functionality can be encoded through single fingers and the force applied.

(ii) Wearable gesture recognition based on sound and motion: Besides other signals, gestures might also emit sound. We develop a recognition method for sound-emitting
gestures, such as snapping, knocking, or clapping, employing only a standard smartwatch.
Besides the motion information from the built-in accelerometer and gyroscope, we exploit
audio data recorded by the smartwatch microphone as input. We propose a lightweight
convolutional neural network architecture for gesture recognition, specifically designed to
run locally on resource-constrained devices. It achieves a user-independent recognition
accuracy of 97.2% for nine distinct gestures. We find that the audio input drastically
reduces the false positive rate in continuous recognition compared to using only motion.

(iii) Device representations in wearable augmented reality (AR): While AR technology is becoming increasingly available to the public, ways of interaction in the AR
space are not yet fully understood. We investigate how users can control smart devices in
AR. Connected devices are augmented with interaction widgets representing them. For
example, a widget can be overlaid on a loudspeaker to control its volume. We explore
three ways of manipulating the virtual widgets in a user study: (1) in-air finger pinching
and sliding, (2) whole-arm gestures rotating and waving, (3) incorporating physical objects
in the surroundings and mapping their movements to interaction primitives. We find
significant differences in the preference of the users, the speed of executing commands,
and the granularity of the type of control. While these methods only apply to control of
a single device at a time, in a second step, we create a method which also takes potential connections between devices into account. Users can view, create, and manipulate
connections between smart devices in AR using simple gestures.

(iv) Personalizable user interfaces from simple materials: User interfaces rarely
adapt to specific user preferences or the task at hand. We present a method that allows
the quick and inexpensive creation of personalized interfaces from paper. Users can cut
out shapes and assign control functions to these paper snippets via a simple configuration
interface. After configuration, control takes place entirely through the manipulation of the
paper shapes, providing the experience of a tailored tangible user interface. The shapes,
which are monitored by a camera with depth sensing, can be dynamically changed during
use.

The proposed methods aim at a more natural interaction with smart devices through
advanced sensing and processing in the user’s environment or on his/her body itself. As
these interactions could be made ubiquitously available through wearable computers, our
methods could help to improve the usability of the growing number of smart devices and
make them more easily accessible to more people.