EX-SITU: Interaction Située Extrême

A set of prominent designers embarked on a research journey to explore aesthetics in movement-based design. Here we unpack one of the design sensitivities unique to our practice: a strong first person perspective—where the movements, somatics and aesthetic sensibilities of the designer, design researcher and user are at the forefront. We present an annotated portfolio of design exemplars and a brief introduction to some of the design methods and theory we use, together substantiating and explaining the first-person perspective. At the same time, we show how this felt dimension, despite its subjective nature, is what provides rigor and structure to our design research. Our aim is to assist researchers in soma-based design and designers wanting to consider the multiple facets when designing for the aesthetics of movement. The applications span a large field of designs, including slow introspective, contemplative interactions, arts, dance, health applications, games, work applications and many others.

Machine learning is one of the most important and successful techniques in contemporary computer science. It involves the statistical inference of models (such as classifiers) from data. It is often conceived in a very impersonal way, with algorithms working autonomously on passively collected data. However, this viewpoint hides considerable human work of tuning the algorithms, gathering the data, and even deciding what should be modeled in the first place. Examining machine learning from a human-centered perspective includes explicitly recognising this human work, as well as reframing machine learning workflows based on situated human working practices, and exploring the co-adaptation of humans and systems. A human-centered understanding of machine learning in human context can lead not only to more usable machine learning tools, but to new ways of framing learning computationally. This workshop will bring together researchers to discuss these issues and suggest future research questions aimed at creating a human-centered approach to machine learning.

A common challenge faced by many domain experts working with time series data is how to identify and compare similar patterns. This operation is fundamental in high-level tasks, such as detecting recurring phenomena or creating clusters of similar temporal sequences. While automatic measures exist to compute time series similarity, human intervention is often required to visually inspect these automatically generated results. The visualization literature has examined similarity perception and its relation to automatic similarity measures for line charts, but has not yet considered if alternative visual representations, such as horizon graphs and colorfields, alter this perception. Motivated by how neuroscientists evaluate epileptiform patterns, we conducted two experiments that study how these three visualization techniques affect similarity perception in EEG signals. We seek to understand if the time series results returned from automatic similarity measures are perceived in a similar manner, irrespective of the visualization technique; and if what people perceive as similar with each visualization aligns with different automatic measures and their similarity constraints. Our findings indicate that horizon graphs align with similarity measures that allow local variations in temporal position or speed (i.e., dynamic time warping) more than the two other techniques. On the other hand, horizon graphs do not align with measures that are insensitive to amplitude and y-offset scaling (i.e., measures based on z-normalization), but the inverse seems to be the case for line charts and colorfields. Overall, our work indicates that the choice of visualization affects what temporal patterns we consider as similar, i.e., the notion of similarity in time series is not visualization independent.

Ultra-high resolution wall-sized displays (“ultra-walls”) are effective for presenting large datasets, but their size and resolution make traditional pointing techniques inadequate for precision pointing. We study mid-air pointing techniques that can be combined with other, domain-specific interactions. We first explore the limits of existing single-mode remote pointing techniques and demonstrate theoretically that they do not support high-precision pointing on ultra-walls. We then explore solutions to improve mid-air pointing efficiency: a tunable acceleration function and a framework for dual-precision (DP) techniques, both with precise tuning guidelines. We designed novel pointing techniques following these guidelines, several of which outperform existing techniques in controlled experiments that involve pointing difficulties never tested prior to this work. We discuss the strengths and weaknesses of our techniques to help interaction designers choose the best technique according to the task and equipment at hand. Finally, we discuss the cognitive mechanisms that affect pointing performance with these techniques.

Recent advances in materials science research allow production of highly stretchable sensors and displays. Such technologies, however, are still not accessible to non-expert makers. We present a novel and inexpensive fabrication method for creating Stretchis, highly stretchable user interfaces that combine sensing capabilities and visual output. We use Polydimethylsiloxan (PDMS) as the base material for a Stretchi and show how to embed stretchable touch and proximity sensors and stretchable electroluminescent displays. Stretchis can be ultra-thin (≈ 200μm), flexible, and fully customizable, enabling non-expert makers to add interaction to elastic physical objects, shape-changing surfaces, fabrics, and the human body. We demonstrate the usefulness of our approach with three application examples that range from ubiquitous computing to wearables and on-skin interaction.

Advances in AI technology affect knowledge work in diverse fields, including healthcare, engineering, and management. Although automation and machine support can increase efficiency and lower costs, it can also, as an unintended consequence, deskill workers, who lose valuable skills that would otherwise be maintained as part of their daily work. Such deskilling has a wide range of negative effects on multiple stakeholders -- employees, organizations, and society at large. This essay discusses deskilling in the age of AI on three levels - individual, organizational and societal. Deskilling is furthermore analyzed through the lens of four different levels of human-AI configurations and we argue that one of them, Hybrid Intelligence, could be particularly suitable to help manage the risk of deskilling human experts. Hybrid Intelligence system design and implementation can explicitly take such risks into account and instead foster upskilling of workers. Hybrid Intelligence may thus, in the long run, lower costs and improve performance and job satisfaction, as well as prevent management from creating unintended organization-wide deskilling.

Human movement has historically been approached as a functional component of interaction within human computer interaction. Yet movement is not only functional, it is also highly expressive. In our research, we explore how movement expertise as articulated in Laban Movement Analysis (LMA) can contribute to the design of computational models of movement's expressive qualities as defined in the framework of Laban Efforts. We include experts in LMA in our design process, in order to select a set of suitable multimodal sensors as well as to compute features that closely correlate to the definitions of Efforts in LMA. Evaluation of our model shows that multimodal data combining positional, dynamic and physiological information allows for a better characterization of Laban Efforts. We conclude with implications for design that illustrate how our methodology and our approach to multimodal capture and recognition of Effort qualities can be integrated to design interactive applications.

Although Human–Computer Interaction research has developed various theories and frameworks for analyzing new and existing interactive systems, few address the generation of novel technological solutions, and new technologies often lack theoretical foundations. We introduce Generative Theories of Interaction , which draw insights from empirical theories about human behavior in order to define specific concepts and actionable principles, which, in turn, serve as guidelines for analyzing, critiquing, and constructing new technological artifacts. After introducing and defining Generative Theories of Interaction, we present three detailed examples from our own work: Instrumental Interaction, Human–Computer Partnerships, and Communities & Common Objects. Each example describes the underlying scientific theory and how we derived and applied HCI-relevant concepts and principles to the design of innovative interactive technologies. Summary tables offer sample questions that help analyze existing technology with respect to a specific theory, critique both positive and negative aspects, and inspire new ideas for constructing novel interactive systems.

We consider kinesthetic awareness, the perception of our own body position and movement in space, as a critical value for embodied design within third wave HCI. We designed an interactive sound installation that supports kinesthetic awareness of a participant's micro-movements. The installation's interaction design uses continuous auditory feedback and leverages an adaptive mapping strategy, refining its sensitivity to increase sonic resolution at lower levels of movement activity. The installation uses field recordings as rich source materials to generate a sound environment that attunes to a participant's micro-movements. Through a qualitative study using a second-person interview technique, we gained nuanced insights into the participants' subjective experiences of the installation. These reveal consistent temporal patterns, as participants build on a gradual process of integration to increase the complexity and capacity of their kinesthetic awareness during interaction.

Wall-sized displays support small groups of users working together on large amounts of data. Observational studies of such settings have shown that users adopt a range of collaboration styles, from loosely to closely coupled. Shared interaction techniques, in which multiple users perform a command collaboratively, have also been introduced to support co-located collaborative work. In this paper, we operationalize five collaborative situations with increasing levels of coupling, and test the effects of providing shared interaction support for a data manipulation task in each situation. The results show the benefits of shared interaction for close collaboration: it encourages collaborative manipulation, it is more efficient and preferred by users, and it reduces physical navigation and fatigue. We also identify the time costs caused by disruption and communication in loose collaboration and analyze the trade-offs between parallelization and close collaboration. These findings inform the design of shared interaction techniques to support collaboration on wall-sized displays.

Recent work identified a concerning trend of disproportional gender representation in research participants in Human–Computer Interaction (HCI). Motivated by the fact that Human–Robot Interaction (HRI) shares many participant practices with HCI, we explored whether this trend is mirrored in our field. By producing a dataset covering participant gender representation in all 684 full papers published at the HRI conference from 2006–2021, we identify current trends in HRI research participation. We find an over-representation of men in research participants to date, as well as inconsistent and/or incomplete gender reporting, which typically engages in a binary treatment of gender at odds with published best practice guidelines. We further examine if and how participant gender has been considered in user studies to date, in-line with current discourse surrounding the importance and/or potential risks of gender based analyses. Finally, we complement this with a survey of HRI researchers to examine correlations between who is doing with the who is taking part, to further reflect on factors which seemingly influence gender bias in research participation across different sub-fields of HRI. Through our analysis, we identify areas for improvement, but also reason for optimism, and derive some practical suggestions for HRI researchers going forward.

Couples exhibit special communication practices, but apps rarely offer couple-specific functionality. Research shows that sharing streams of contextual information (e.g. location, motion) helps couples coordinate and feel more connected. Most studies explored a single, ephemeral stream; we study how couples' communication changes when sharing multiple, persistent streams. We designed Lifelines, a mobile-app technology probe that visualizes up to six streams on a shared timeline: closeness to home, battery level, steps, media playing, texts and calls. A month-long study with nine couples showed that partners interpreted information mostly from individual streams, but also combined them for more nuanced interpretations. Persistent streams allowed missing data to become meaningful and provided new ways of understanding each other. Unexpected patterns from any stream can trigger calls and texts, whereas seeing expected data can replace direct communication, which may improve or disrupt established communication practices. We conclude with design implications for mediating awareness within couples.

Our goal is to understand how dancers learn complex dance phrases. We ran three workshops where dancers learned dance fragments from videos. In workshop 1, we analyzed how dancers structure their learning strategies by decomposing movements. In workshop 2, we introduced MoveOn, a technology probe that lets dancers decompose video into short, repeatable clips to support their learning. This served as an effective analysis tool for identifying the changes in focus and understanding their decomposition and recomposition processes. In workshop 3, we compared the teacher's and dancers' decomposition strategies, and how dancers learn on their own compared to teacher-created decompositions. We found that they all ungroup and regroup dance fragments, but with different foci of attention, which suggests that teacher-imposed decomposition is more effective for introductory dance students, whereas personal decomposition is more suitable for expert dancers. We discuss the implications for designing technology to support analysis, learning and teaching of dance through movement decomposition.

Motor skill acquisition inherently depends on the way one practices the motor task. The amount of motor task variability during practice has been shown to foster transfer of the learned skill to other similar motor tasks. In addition, variability in a learning schedule, in which a task and its variations are interweaved during practice, has been shown to help the transfer of learning in motor skill acquisition. However, there is little evidence on how motor task variations and variability schedules during practice act on the acquisition of complex motor skills such as music performance, in which a performer learns both the right movements (motor skill) and the right time to perform them (timing skill). This study investigated the impact of rate (tempo) variability and the schedule of tempo change during practice on timing and motor skill acquisition. Complete novices, with no musical training, practiced a simple musical sequence on a piano keyboard at different rates. Each novice was assigned to one of four learning conditions designed to manipulate the amount of tempo variability across trials (large or small tempo set) and the schedule of tempo change (randomized or non-randomized order) during practice. At test, the novices performed the same musical sequence at a familiar tempo and at novel tempi (testing tempo transfer), as well as two novel (but related) sequences at a familiar tempo (testing spatial transfer). We found that practice conditions had little effect on learning and transfer performance of timing skill. Interestingly, practice conditions influenced motor skill learning (reduction of movement variability): lower temporal variability during practice facilitated transfer to new tempi and new sequences; non-randomized learning schedule improved transfer to new tempi and new sequences. Tempo (rate) and the sequence difficulty (spatial manipulation) affected performance variability in both timing and movement. These findings suggest that there is a dissociable effect of practice variability on learning complex skills that involve both motor and timing constraints.

Traditional graphic design tools emphasize the grid for structuring layout. Interviews with professional graphic designers revealed that they use surprisingly sophisticated structures that go beyond the grid, which we call graphical substrates. We present a framework to describe how designers establish graphical substrates based on properties extracted from concepts, content and context, and use them to compose layouts in both space and time. We developed two technology probes to explore how to embed graphical substrates into tools. Contextify lets designers tailor layouts according to each reader's intention and context; while Linkify lets designers create dynamic layouts based on relationships among content properties. We tested the probes with professional graphic designers, who all identified novel uses in their current projects. We incorporated their suggestions into StyleBlocks, a prototype that reifies CSS declarations into interactive graphical substrates. Graphical substrates offer an untapped design space for tools that can help graphic designers generate personal layout structures.

Software tools for generating digital sound often present users with high-dimensional, parametric interfaces, that may not facilitate exploration of diverse sound designs. In this article, we propose to investigate artificial agents using deep reinforcement learning to explore parameter spaces in partnership with users for sound design. We describe a series of user-centred studies to probe the creative benefits of these agents and adapting their design to exploration. Preliminary studies observing users’ exploration strategies with parametric interfaces and testing different agent exploration behaviours led to the design of a fully-functioning prototype, called Co-Explorer, that we evaluated in a workshop with professional sound designers. We found that the Co-Explorer enables a novel creative workflow centred on human–machine partnership, which has been positively received by practitioners. We also highlight varied user exploration behaviours throughout partnering with our system. Finally, we frame design guidelines for enabling such co-exploration workflow in creative digital applications.

Information visualization research has developed powerful systems that enable users to author custom data visualizations without textual programming. These systems can support graphics-driven practices by bridging lazy data-binding mechanisms with vector-graphics editing tools. Yet, despite their expressive power, visualization authoring systems often assume that users want to generate visual representations that they already have in mind rather than explore designs. They also impose a data-to-graphics workflow, where binding data dimensions to graphical properties is a necessary step for generating visualization layouts. In this paper, we introduce StructGraphics, an approach for creating data-agnostic and fully reusable visualization designs. StructGraphics enables designers to construct visualization designs by drawing graphics on a canvas and then structuring their visual properties without relying on a concrete dataset or data schema. In StructGraphics, tabular data structures are derived directly from the structure of the graphics. Later, designers can link these structures with real datasets through a spreadsheet user interface. StructGraphics supports the design and reuse of complex data visualizations by combining graphical property sharing, by-example design specification, and persistent layout constraints. We demonstrate the power of the approach through a gallery of visualization examples and reflect on its strengths and limitations in interaction with graphic designers and data visualization experts.

International audience

We introduce BIGFile, a new fast file retrieval technique based on the Bayesian Information Gain framework. BIGFile provides interface shortcuts to assist the user in navigating to a desired target (file or folder). BIGFile's split interface combines a traditional list view with an adaptive area that displays shortcuts to the set of file paths estimated by our computationally efficient algorithm. Users can navigate the list as usual, or select any part of the paths in the adaptive area. A pilot study of 15 users informed the design of BIGFile, revealing the size and structure of their file systems and their file retrieval practices. Our simulations show that BIGFile outperforms Fitchett et al.'s AccessRank, a best-of-breed prediction algorithm. We conducted an experiment to compare BIGFile with ARFile (AccessRank instantiated in a split interface) and with a Finder-like list view as baseline. BIGFile was by far the most efficient technique (up to 44% faster than ARFile and 64% faster than Finder), and participants unanimously preferred the split interfaces to the Finder.

Machine Learning models can output confident but incorrect predictions. To address this problem, ML researchers use various techniques to reliably estimate ML uncertainty, usually performed on controlled benchmarks once the model has been trained. We explore how the two types of uncertainty—aleatoric and epistemic—can help non-expert users understand the strengths and weaknesses of a classifier in an interactive setting. We are interested in users’ perception of the difference between aleatoric and epistemic uncertainty and their use to teach and understand the classifier. We conducted an experiment where non-experts train a classifier to recognize card images, and are tested on their ability to predict classifier outcomes. Participants who used either larger or more varied training sets significantly improved their understanding of uncertainty, both epistemic or aleatoric. However, participants who relied on the uncertainty measure to guide their choice of training data did not significantly improve classifier training, nor were they better able to guess the classifier outcome. We identified three specific situations where participants successfully identified the difference between aleatoric and epistemic uncertainty: placing a card in the exact same position as a training card; placing different cards next to each other; and placing a non-card, such as their hand, next to or on top of a card. We discuss our methodology for estimating uncertainty for Interactive Machine Learning systems and question the need for two-level uncertainty in Machine Teaching.