This paper introduces Workflow graphs, or W-graphs, which encode how the approaches taken by multiple users performing a fixed 3D design task converge and diverge from one another. The graph's nodes represent equivalent intermediate task states across users, and directed edges represent how a user moved between these states, inferred from screen recording videos, command log data, and task content history. Read More The result is a data structure that captures alternative methods for performing sub-tasks (e.g., modeling the legs of a chair) and alternative strategies of the overall task. As a case study, we describe and exemplify a computational pipeline for building W-graphs using screen recordings, command logs, and 3D model snapshots from an instrumented version of the Tinkercad 3D modeling application, and present graphs built for two sample tasks. We also illustrate how W-graphs can facilitate novel user interfaces with scenarios in workflow feedback, on-demand task guidance, and instructor dashboards. Read Less

When watching how-to videos related to physical tasks, users’ hands are often occupied by the task, making voice input a natural fit. To better understand the design space of voice interactions for how-to video navigation, we conducted three think-aloud studies using: 1) a traditional video interface, 2) a research probe providing a voice controlled video interface, and 3) a wizard-of-oz interface. Read More From the studies, we distill seven navigation objectives and their underlying intents: pace control pause, content alignment pause, video control pause, reference jump, replay jump, skip jump, and peek jump. Our analysis found that users’ navigation objectives and intents affect the choice of referent type and referencing approach in command utterances. Based on our findings, we recommend to 1) support conversational strategies like sequence expansions and command queues, 2) allow users to identify and refine their navigation objectives explicitly, and 3) support the seven interaction intents. Read Less

For cooking professionals and culinary students, understanding cooking instructions is an essential yet demanding task. Common tasks include categorizing different approaches to cooking a dish and identifying usage patterns of particular ingredients or cooking methods, all of which require extensive browsing and comparison. However, no existing system provides support for such in-depth and at-scale analysis. Read More We present RecipeScape, an interactive system for browsing and analyzing the hundreds of recipes of a single dish available online. We also introduce a computational pipeline that extracts cooking processes from recipe text and calculates a procedural similarity between them. To evaluate how RecipeScape supports culinary analysis at scale, we conducted a user study with cooking professionals and culinary students with 500 recipes for two different dishes. Results show that RecipeScape clusters recipes into distinct approaches, and captures notable usage patterns of ingredients and cooking actions. Read Less

The diversity and the scale of available online instructions introduce opportunities but also user challenges in currently used software interfaces; Users have limited computational resources, and thus often make strategic decisions when browsing, navigating, and understanding instructions to accomplish a task. These strategic user interactions possess nuanced semantics such as users' interpretations, intents, and contexts in which the task is carried out. Read More My dissertation research introduces techniques in constructing data structures that capture the diverse strategies users employ in which the collective nuanced semantics across multiple strategies are preserved. These computational representations are then used as building blocks for designing novel interactions that allow users to effectively browse and navigate instructions, and provide contextual task guidance. Specifically, I investigate 1) structure of instructions for task analysis at scale, 2) structure of collective user task demonstrations, and 3) structure of object uses in how-to videos for tracking, guiding and searching task states. My research demonstrates that the user-centered organization of information extracted from interaction traces enables novel interfaces with contextual task support. Read Less

In this position paper, I present a set of data-driven techniques in modeling the learning material, learner workflow and the learning task as graphical representations, with which at scale can create and support learning opportunities in the wild. I propose the graphical models resulting from this bottom-up approach can further serve as proxies for representing learnability bounds of an interface. Read More I also propose an alternative approach which directly aims to "learn" the interaction bounds by modeling the interface as an agent's sequential decision making problem. Then I illustrate how the data-driven modeling techniques and algorithm modeling techniques can create a mutually beneficial bridge for advancing design of interfaces. Read Less

Learner-driven subgoal labeling helps learners form a hierarchical structure of solutions with subgoals, which are conceptual units of procedural problem solving. While learning with such hierarchical structure of a solution in mind is effective in learning problem solving strategies, the development of an interactive feedback system to support subgoal labeling tasks at scale requires significant expert efforts, making learner-driven subgoal labeling difficult to be applied in online learning environments. We propose SolveDeep, a system that provides feedback on learner solutions with peer-generated subgoals. Read More SolveDeep utilizes a learnersourcing workflow to generate the hierarchical representation of possible solutions, and uses a graph-alignment algorithm to generate a solution graph by merging the populated solution structures, which are then used to generate feedback on future learners' solutions. We conducted a user study with 7 participants to evaluate the efficacy of our system. Participants did subgoal learning with two math problems and rated the usefulness of system feedback. The average rating was 4.86 out of 7 (1: Not useful, 7: Useful), and the system could successfully construct a hierarchical structure of solutions with learnersourced subgoal labels. Read Less

We conducted a series of exploratory studies on sensemaking behaviors people exhibit while watching how-to-cook videos. The three different scenarios we examined are a) when people seek for alternatives in ingredients, tools and actions, b) when people seek for explanations or more detail on certain instructions, and c) when people use text search and when people use video when learning how to cook a dish. Read More We found a) people often make arbitrary decisions on substituting ingredients, cooking tools, or cooking actions while following instructions, b) people satisfice by verifying knowledge with little data and not wanting to deviate from the initially chosen video, and c) people use text search for definitions and confirmation of substitutions while they use video search for explanations and precise details for instruction steps Read Less

In culture analytics, it is important to ask fundamental questions that address salient characteristics of collective human behavior. This paper explores how analyzing cooking recipes in aggregate and at scale identifies these characteristics in the cooking culture, and answer fundamental questions like ”what makes a chocolate chip cookie a chocolate chip cookie?”. Read More Aspiring cooks, professional chefs and cooking hobbyists share their recipes online resulting in thousands of different procedural instructions towards a shared goal. However, existing approaches focus merely on analysis at the ingredient level, for example, extracting ingredient information from individual recipes. We introduce RecipeScape, a prototype interface which supports visually querying, browsing and comparing cooking recipes at scale. We also present the underlying computational pipeline of RecipeScape that scrapes recipes online, extracts their ingredient and instruction information, constructs a graphical representation, and computes similarity between pairs of recipes. Read Less