1. INTRODUCTION
Ever since the invention of small sticky notes such as the Post-It™, designers have been plastering them everywhere. They appear on walls and billboards, cluster around computer monitor frames, and are embedded within documents. They are used for gathering informal feedback and annotating discussions, for personal notes, asynchronous discussion among groups or between designer and client, and for error correction. Posterboards covered with many colors of sticky notes are used to document tasks as diverse as brainstorming sessions, project planning, and design presentations.
Studies on work process for many kinds of designers show that sticky notes are ubiquitous, used for a variety of purposes. Architects and product designers scribble notes and stick them onto blueprints or even onto physical models. New media or industrial product designers can use them to represent nonlinear or sequential content units, enjoying the ease with which events can be rearranged in a storyboard, for example.
The tangible ease of use of sticky notes is one of their principal affordances. However, in the digital world of modern design practice, the ability to save, track, and search the information on sticky notes is both important, and unfortunately, unwieldy. Designers resort to camera capture of whiteboards or posterboards covered with sticky notes, or to postsession computer entry by hand. How can we capture the ease of use and tangible affordances of sticky notes (lightweight, portable, sortable, stickable anywhere) and still retain the affordances of digital information (save, search, filter, versions, edit)? How can we enable designers to capture “digital scribbles” in different types of media (images, video, animation, color or texture swatches) and attach them to real-world objects?
In this article we describe Post-Bits, a design prototype of a tangible sticky-note tool that supports both real-world design practice and digital information processing. The basic premise of Post-Bits seems simple: small sticky notes made not from regular paper, but from electronic paper (e-paper). However, behind the apparent simplicity of the idea lies a complex set of questions: what kinds of content are effective on e-paper stickies? How can designers use them not only in known ways but new ways? How do they integrate with current software and design practice? Practical questions about the Post-Bit system itself also abound: how does digital information move onto and off of each Post-Bit? What sensors and systems need to be in place to handle editing, playback, capture, and other interactions? How are stacks of multiple notes handled? What about power and communication systems? This project aimed to answer these questions through designing and building a conceptual prototype of a Post-Bit.
The Post-Bits system is partly conceptual design and partly real prototype: we built a working prototype of the system, but its display was a standard small LCD screen, not e-paper. Video-rate color e-paper was not commercially available at the time, and standard bi-stable e-paper was not easily obtainable in the small sizes we needed for our prototype. Video-capable e-paper and its kin, such as organic LED (OLED) screens, and reliable flexible circuitry are both still in development; but it is clear that small flexible displays will soon be available in the consumer market from firms like Sony, Fuji Xerox, Fujitsu, and Philips (Sony and Philips both showed working prototypes in 2007). We found that designing and building the Post-Bits prototype with off-the-shelf technologies was extremely useful in understanding the interaction modalities and software support needed to create an e-paper sticky-note technology. It also helped us understand the kinds of design practices and processes that Post-Bits can support, and to explore the design space of tangible tools for design practice.
1.1. Tangibility: Physical affordances of paper, with digital information handling
Experiments in tangible computing have shown the utility of small distributed displays, allowing free arrangement of information in space (Gorbet et al., Reference Gorbet, Orth and Ishii1998; Merrill et al., Reference Merrill, Kalanithi and Maes2007). Post-Bits combine the affordances of physical tiny sticky memos and the digital handling of visual information, enabling people to arrange multimedia content in embodied physical spaces.
The Post-Bit system's functionalities were based on the use of small sticky notes as paper tags as noted in interaction design literature (Moran et al., Reference Moran, Saund, Melle, Gujar, Fishkin and Harrison1999; Gill, Reference Gill2003). We felt it was important to keep the interaction design simple and natural, retaining only fundamental functions that seem like a natural interaction with small papers: flipping, sorting, stacking, sticking. This informed the choice of sensor types and sensor placement on a Post-Bit (what people want to do, what feels natural in the hand) as well as placement for data and power connectivity in the environment (e.g., on a PC monitor frame, just like many paper stickies).
To further refine our focus, we researched design methods using sticky notes (e.g., brainstorming sessions using visual collages) as well as other pieces of paper (color swatches, magazine clippings, photos). We also tracked how designers use sticky notes in their physical design environments, and how sticky notes form part of the brainstorming and iterative design chain. Finally, we looked at the diverse uses of digital design tools to extrapolate tasks and functions that mapped well to the functions of e-paper sticky notes. In particular, we investigated design tasks in interactive systems, such as design for multimedia and product design for interactive systems and devices (although many of the tasks we examined such as swatching, sorting, and annotating also apply in more traditional design domains).
1.2. Paper sticky notes and their digital analogs: Finding common ground
In an effort to aid migration of data from sticky notes into the computer, paper sticky notes are sometimes found clustered around the frame of a computer monitor. This approach is fairly limited, as it relies strictly on visual/spatial proximity of the notes to the computer screen, with no integration with whatever may happen to be on the screen at the time. Other limiting factors are monitor frame space, and the inability to store and track the sticky note information over time. Digital sticky notes seem an obvious way to address this, and indeed, many such systems exist.
One digital analog of paper sticky notes is the plethora of sticky note helper applications found as freeware online (a few examples are StickyPad, Jjot, or MyStickies). Sticky note widgets are often found as well as in applications like Adobe Photoshop or browser extensions (such as Internote for Firefox). Another example is WebStickies, an inline Google module that allows you to put digital sticky notes on your iGoogle homepage. Although these aptly replicate the functions of sticky notes inside the computer as well as sometimes adding useful functionalities such as search, notification, or alarm functions, they cannot migrate out of the computer to be applied to spaces and surfaces in the real world.
Much research underlies the creation of digital analogs for paper notes (and other documents). Older examples of digital memo systems such as the NoteCards work at Xerox PARC (Halasz et al., Reference Halasz, Moran and Trigg1987) as well as some early card-based hypertext systems (HyperCard) were designed to support workflow or idea flow (in NoteCards, a “semantic network” of information), with built-in hierarchies and embedded tools. However, these were never designed to capture the lightweight usability of the physical sticky. Other paper-note studies focused on understanding the use of spatial organization in paper-based information, including the use of sticky notes (Shipman et al., Reference Shipman, Marshall and Moran1995; Bellotti & Rogers, Reference Bellotti and Rogers1997). These kinds of studies led to the restructuring of information in new digital systems such as the hyperbolic browser from PARC (Lamping & Rao, Reference Lamping and Rao1996), but again, this system focused on overall workflow support, rather than on lightweight notes.
In a more tangible realm, several researchers have experimented with physical/digital interplay using paper sticky notes. The “Quickie” sticky note system (Mistry & Maes, Reference Mistry and Maes2008) allows for digital capture of the content of paper sticky notes, focused on handwriting capture using a commercially available digital pen. This system focuses on handwritten notes that are processed through handwriting recognition software; important digital affordances include the ability to browse through images of the notes and to search them using keywords. Quickie notes also have onboard radio frequency identification (RFID) tags that serve as finder beacons, making them easy to locate within a home or office space. Another system, WebStickers, uses sticky notes with printed bar codes that users can use to associate a URL with that sticky note; the sticky note can then be applied to a real-world object. To see the Web site associated with the physical object, the user swipes a barcode reader over the sticky note. Systems designed for contextual awareness often track users to deliver digital notes on ambient or peripheral displays (Want et al., Reference Want, Fishkin, Gujar and Harrison1999; MacIntyre et al., Reference MacIntyre, Mynatt, Voida, Hansen, Tullio and Corso2001). The StickySpots system (Elliot et al., Reference Elliot, Neustaedter and Greenberg2007) uses digital notes, in this case as messages, but they are displayed in a variety of embedded locations in a home, such as a digital tabletop or wall screen as well as a standard computer monitor. The notes migrate to different displays depending on the location of the person for whom the message is intended.
Another approach to spatial/physical handling of digital information focuses on sets of small devices with onboard displays, data communication and computation, such as (among others) the Triangles system (Gorbet et al., Reference Gorbet, Orth and Ishii1998) or the more recent Siftables (Merrill et al., Reference Merrill, Kalanithi and Maes2007). These systems focus on multimedia content handling but do not take the affordance of paper or of sticky notes as their primary interaction model, which leads to a different set of functionalities. However, both Triangles and Siftables do demonstrate the collection, sorting, and reordering of media clips, showing the usefulness of multiple small displays for the physical handling of multimedia. Camera phones and smart phones also have some functionalities in this category, although it is not their primary focus. All of these systems feature devices, however, that are inherently heavier and less manipulatable than paper; hence, our interest in e-paper as a medium for a new kind of digital note.
2. PROPERTIES OF E-PAPER USEFUL FOR TANGIBLE INTERFACES
Describing the various technologies behind e-paper is outside the scope of this paper; we are primarily interested in the affordances of e-paper for use in a tangible tool for design. However, a brief discussion of the properties of current e-paper technologies is germane, as the Post-Bit system is a working design prototype that uses today's technology to posit behaviors and forms for a tangible design tool made of e-paper. Three aspects of e-paper are most important in this design. First, e-paper is a reflective display; that is, it does not use backlighting or light-produced pixels like an LCD or LED screen, and it can be read in bright light or sunshine. Second, it can be built to have a flexible substrate (although many use a solid one), which allows us to construct a device that has the physical bendability of paper (Fig. 1). Third, e-paper has the ability to retain an image without power. This means that, for example, a video keyframe could be retained as the image on an e-paper Post-Bit indefinitely, without draining the device's small battery.
Fig. 1. A flexible e-paper display holding its image without power. Photo courtesy of Fujitsu Inc. press release. [A color version of this figure can be viewed online at journals.cambridge.org/aie]
Video-rate color e-paper has not yet been released commercially (as of late 2008). However, in 2007, Philips and Sony both demonstrated flexible color e-paper achieving switching speeds capable of supporting video (Fig. 2). Coupled with new developments in flexible circuitry such as that from Plastic Logic (Fig. 3), the technologies for building the Post-Bit system currently exist, but are not yet commercially available. However, other types of e-paper are currently available commercially. Fuji Xerox has shown e-paper document readers on a solid substrate, and E-ink offers card-sized e-paper. Other flexible displays such as flexible transparent OLED or Philips' flexible LCD may also serve for small-size multimedia playback on flexible screens.
Fig. 2. Sony's video-rate flexible color e-paper in a 2007 demonstration. Photo courtesy of Sony Corp. press release. [A color version of this figure can be viewed online at journals.cambridge.org/aie]
Fig. 3. Plastic Logic from E Ink Corporation: a flexible, high resolution, printed active-matrix backplane driving an electronic paper frontplane. Photo courtesy of E Ink Corp. press release. [A color version of this figure can be viewed online at journals.cambridge.org/aie]
Many designs for e-paper technologies have been restricted to the uses of e-paper either for commercial signage, or as a paper document, retaining the familiar form and feel of the letter-sized page. Sony and Amazon both have e-paper books commercially available: the Reader Digital Book and the Kindle, respectively. These follow and improve upon a few e-book launches featuring e-paper over the past decade (notably an earlier Sony e-book, LIBRIe, for Japanese language books only). However, these restrict themselves to a solid backplane behind the e-paper display, losing the affordance of flexibility. We believe that it is important to exploit new functionalities in display technologies such as memory function (allowing information retention without power) and flexibility (allowing an integrated interface) to create dynamic natural interfaces for rich digital media in physical space.
2.1. Physical handling of paper sticky notes: Informal observation
We started the design of our tangible sticky notes system with informal user observations of the use of paper sticky notes, a popular and useful paper item in most workplaces. We are interested in both content and in physical dispersion and handling of information. Thus, we took a look at what kind of activities and information people use sticky notes for in our own workplace, a multimedia/computer science research laboratory. We watched how people physically handled the sticky notes they used, and where they tended to place them in their surroundings. We needed to understand where to begin with placement of our tangible controllers on the notes themselves, and what kinds of information would be most valuable in conjunction with those controllers.
We created a small paper prototype of a multimedia Post-Bit, approximately 2 inches wide and 4 inches long, with an image printed on one side. We handed this to people and watched how they handled it, first without any prompting. Then we asked them to imagine that the image was a keyframe for an embedded video they could watch. Common finger placements and ways of handling the paper prototype would inform the choice of sensor types and sensor placement on a Post-Bit note. Next, we created a small paper prototype of a container for several notes, observing how people held and used it and the notes along with it, again with the idea that these represented video keyframes. Sorting through, rearranging, and restacking these keyframes was one of the most common actions. Pulling out a single note to play with and then reinserting it into the pack was another common action. Based on these observations we set the sensor placement for our first prototype.
2.2. Common uses of paper sticky notes
From the literature as well as informal observations, we derived three common functions in the way people use sticky notes (although others, of course, occur): storing information, sharing information, and sorting information. First and foremost, people use tiny sticky notes to store information, either as a reminder or as a place to keep small notes. They scribble some key words or short notes on sticky notes and stick these on their PC, desk, notebook, or doorway. When they move from one task to another, they notice the information and are reminded; or if they need the information and look deliberately for the memo, the search task is aided by spatial memory. Second, sharing information between people includes asynchronous communications like “Call me when you get back to your desk” and warnings like “Do not touch the red button!” Third, for sorting information in situations such as brainstorming, people often list a data element on each sticky note and then stick them on a big physical desktop, wall, or other surface. Thus, the sticky notes become movable markers used to create a structure for seeing an overall “big picture” of the data at hand. One useful observation was that people tend to put paper sticky notes on a PC display frame—deliberately placing it at the edge of their visual field, as a peripheral reminder (Fig. 4). This resulted in a design for the primary connection of a Post-Bit to be similarly located on the display frame when it is connected to a PC. This, in turn, resulted in some unique interface functions based on the Post-Bit's location on the display frame (see drop-beyond-drag and squeeze-it functions below).
Fig. 4. People often populate their workspaces with paper sticky notes, in this case, around the frame of a computer display. [A color version of this figure can be viewed online at journals.cambridge.org/aie]
3. POST-BITS SYSTEM DESCRIPTION
Each Post-Bit is a small e-paper device for handling multimedia content, combining interaction control and display into one unit. Post-Bits can download and show multimedia content via a tangible display and are also controllable via embedded sensors mounted behind the e-paper. Tangible properties of paper such as flipping, flexing, scattering, and rubbing are mapped to control aspects of the content. This allows interaction with the multimedia content such as speeding up or slowing down frames, or rearranging them by restacking several Post-Bits in a different order.
Because the standard graphical user interface (GUI) remains a primary method of authoring and managing multimedia, a method of connection and communication is needed between Post-Bits and the host computer. This is the Post-Bit Base, a thin physical surface with data connections and power supply. A Post-Bit Base mounted on a PC display frame enables one to drag-and-drop files directly from the GUI window onto the Post-Bit. In the other direction, information from the Post-Bit is returned via a squeezing action (rather like squeezing a ketchup packet). Both interactions are supported by animated feedback.
The Post-Bit integrated system includes four main components. First is the Post-Bit Player, which is the e-paper sticky memo itself, with integrated sensors and connectors. Second, the Post-Bit Binder is a small container that a few Post-Bits can be stacked and stuffed into, for ordering arrangement and multiple connections to the host computer. Third, the Post-Bit Base is the data and power port that allows communication with the host computer. Fourth, the Post-Bit software and GUI reside on the host PC and manage multimedia transfer and serial communication.
3.1. Post-Bit Player design
A Post-Bit is envisioned as an e-paper device measuring about 2 × 4.5 inches (Fig. 5). The display is flexible, about the thickness and stiffness of a paperback book cover. (For another take on flexible displays, see Schwesig et al., Reference Schwesig, Poupyrev and Mori2003.) Each Post-Bit has a detachable bond on its back for random placement mounting, mechanically mimicking the sticky back of sticky notes. Behind the display, a flexible (resistive bend) sensor and a touch slider sensor are placed for easy operation by the fingers. Data collected from the sensors is used to step through multimedia content. Each Post-Bit also uses a battery, a tilt sensor, a microcontroller, a tact switch, and a display driver.
Fig. 5. A sketch of the Post-Bit player. [A color version of this figure can be viewed online at journals.cambridge.org/aie]
3.2. Post-Bit Binder
The Post-Bit Binder is a small container that a few Post-Bits can fit into (Fig. 6). It connects to the Post-Bit Base in a manner similar to the Post-Bits themselves. On the edge that fits into the Post-Bit Binder, a Post-Bit has embedded terminals that are used to download content from the host PC (or from other devices such as cell phones). One terminal is an interface to connect a Post-Bit to the mounting on a PC frame. The others are pass-through connectors for the next Post-Bit in the binder stack. As terminals are connected straight inline, it works regardless of the number of displays. The Post-Bit Binder keeps the connections aligned properly, serves as primary connector to the PC frame, and serves as an easily handled physical container for a pack of Post-Bits. The Binder itself has an e-paper cover that displays either metadata about its content, or simply mirrors the static content of the top Post-Bit in the stack.
Fig. 6. A sketch of the Post-Bit Binder components. [A color version of this figure can be viewed online at journals.cambridge.org/aie]
3.3. Post-Bit Base (data and power connections)
The Post-Bit Base is the data and power port that allows communication with the host computer (Fig. 7). The connection between Post-Bits and a Base is established through surface contact. The Base's connection to the Post-Bit is wireless, using induction to pass data and to charge the Post-Bit's battery. To enable a user's physical arrangement of Post-Bits, the data and power supply base should be able to be embedded into several places in the workspace environment and the system should know which Post-Bit is where and when. Ideally, users can choose ad hoc physical placements to suit their own mental maps: for example, “I keep the To Do Today list on the shelf to my left.”
Fig. 7. The structure of the data connection of a Post-Bit Base.
To solve this problem, the Post-Bit Base uses a combination of three technologies. First is a power supply technology, preferably a contact-free (inductive) one such as that marketed by Splash Power, consisting of a thin film shaped to cover physical surfaces. Magnetic or metal velcro data plus power systems such as described by Gorbet et al. (Reference Gorbet, Orth and Ishii1998) are also a possibility. Second is a data communications protocol to deliver content and sensor information. Third is an ID technology such as swept-frequency RFID (Paradiso & Hsiao, Reference Paradiso and Hsiao1999) or an onboard ID system that keeps track of each Post-Bit and its content. Figure 7 shows all in one structure how these three technologies work for the Base. The Post-Bit Base has an embedded coordinate grid. Each Post-Bit has a unique onboard ID. Thus, the system can recognize location of a Post-Bit on a Base. The Post-Bit Base can also be used for other mobile devices, for example, to synchronize calendars on mobile devices while they recharge on the Base.
3.4. Software: GUI with a Post-Bit Base on a PC display frame
The host computer's display frame is the primary location for the Post-Bit Base PC interface. Post-Bits applied to this Post-Bit Base are recognized and their location computed to enable the animations connected with the drag-and-drop and squeeze functions. These authorizations of Post-Bits are coordinated with the system of extended GUIs on a PC (see below, functions: drop-beyond-drag, squeeze). Location information is important to cope with the asymmetry problem discussed in Section 4.1.
4. SENSORS AND FUNCTIONS FOR TANGIBLE INTERACTION
New sensor geometries and associated interaction functions were developed to deal with the tangible properties of the flexible Post-Bit. An example of the sensor geometry problems presented can be seen in the location of the Post-Bit Base, which connects the small e-paper device to the host computer. The Post-Bit Base is intended to be located on the edge of the display screen, so the design has inherent left–right asymmetry. One edge of the backside of the Post-Bit has velcro and connectors and the other side has a slider sensor; this adds more asymmetry. The Post-Bit may be mounted anywhere around the frame of the display, either toward the right or toward the left; and there is also the problem of the handedness of users, which adds even more asymmetry. To solve this problem, a tilt switch was added as a third sensor on board the Post-Bit. This sensor manages the asymmetry problem by, for example, making sure that content appears right side up even if the Post-Bit is turned upside down. Z-agon is a recent application that shows a similar approach to managing asymmetrical mappings of interfaces to contents (Ueda et al., Reference Ueda, Matsumoto and Okude2004). The Post-Bit system, however, uses the tilt sensor to address asymmetry caused both by handedness and by the double edges of the display.
4.1. Drop-beyond-drag: Data transportation from GUI to physical objects
Drop-beyond-drag is a function for copying multimedia content from a device (PC, PDA, cellphone) to Post-Bits. When a Post-Bit is placed on the frame of the computer monitor or device screen, it can act as an extension of the main display (Fig. 8). A user drags and drops the file from a PC display directly to a Post-Bit. If the driver is not available or enabled to allow this, then drop-beyond-drag comes into play: the Post-Bit lights up to acknowledge that the mouse has arrived in its vicinity and shows key information such as a title, an icon and a key frame. This is a similar concept to pick-and-drop (Rekimoto, 1997). However, the Post-Bit directory transports the files without disengaging from the device and the Post-Bit displays the content on itself, which is a more direct mapping.
Fig. 8. GUI animation of the drop-beyond-drag interaction function. [A color version of this figure can be viewed online at journals.cambridge.org/aie]
Where the GUI displayed on the computer monitor and the physical Post-Bit intersect, an animated visual lets the user know that the Post-Bit has acquired the data. When the dragged item comes close to the mounted Post-Bit, a hand icon appears and “grabs” the file. Although this action seems somewhat like Apple's “Dock” icons (Ludolph et al., Reference Ludolph, Norman and Spiegel1998), the metaphor in this case is attached to the physical object outside the GUI frame of reference, the Post-Bit. The animation shows data acquisition by “reaching” out onto the GUI to grab the data.
4.2. Squeeze-it: Data transportation from physical objects to GUI
Getting data from the host computer's GUI onto the Post-Bit is a simple extension of the GUI metaphor. Getting data from the Post-Bit back onto the main screen, however, does not have such an obvious mapping. The squeeze-it function uses a ketchup packet metaphor. By putting a Post-Bit on the display frame and applying pressure to the end of it, a sensor is triggered that transfers data back into the GUI environment, rather like squeezing ketchup from a packet out onto a dish (Fig. 9). In our prototype, we use a tact switch set between the Post-Bit prototype display and the circuit board. Using a variable force sensor instead of a switch would allow us to create more realistic splash animations related to the amount of force (or “squeeze”) applied. In related research, Ikeda et al. (Reference Ikeda, Sato and Kimura2003) created haptic devices using a sponge metaphor, with squeezing as the action driving a display; and Music Shapers (Orth & Weinberg, Reference Orth and Weinberg2001) used variable pressure on a soft object as a mapping for audio parameters.
Fig. 9. A squeeze-it interaction function: squeezing the Post-Bit “squirts” a purple icon out onto the desktop; clicking the icon then brings up the movie clip. [A color version of this figure can be viewed online at journals.cambridge.org/aie]
4.3. Flip-and-find: Stack manipulation
This function enables the finding of target content from multimedia files. When movie data is burned (using drop-beyond-drag) to a stack of Post-Bits, each e-paper Post-Bit shows a key frame of the content. Users can then flip through the Post-Bits to look through key frames (or presentation slides, or any set of sequential or nonsequential data). In the other direction, when uploading content from a stack of Post-Bits in a Binder to the desktop, the sorting aspect of the flip-and-find function creates one file combining the scenes in the same order the Post-Bits are stacked in.
4.4. Pick-and-play: Automatic switching
A loaded Post-Bit can act like a static note or label when it is posted somewhere (the e-paper does not require power to keep one screen of data showing). But it can be set to “wake up” when the Post-Bit is picked up and the tilt sensor is activated. When set to pick-and-play mode, a Post-Bit becomes active as a multimedia player as soon as it is picked up and becomes dormant again upon being put down.
4.5. Flex-and-rub: Tactile interface for video content
Each Post-Bit can show video or other sequential image or text content. The content is controlled by manipulating the physical form of the Post-Bit. A flex-and-rub interaction uses two flat, onboard sensors that are placed in order to be easily reached with the fingers: first, a slider sensor for fast forwarding is mounted along one edge of the Post-Bit's back, and it can be used by stroking the back of the Post-Bit with one finger. Bending the Post-Bit warps the flex sensor, which speeds or slows the frame rate of the content (Fig. 10). This allows careful inspection of, or speedy cruising through, any section of the content. It also provides a quick glimpse of content when the user flips through the Post-Bits in quick succession. Previous work on graspable device manipulation can be found (Rekimoto, Reference Rekimoto1997; Harrison et al., Reference Harrison, Fishkin, Gujar, Mochon and Want1998), but support for the functionality for these systems is distributed across two or more devices. A Post-Bit, however, combines the display, movie controls, and the tactile interface all in one unit on the device itself.
Fig. 10. Flex-and-rub: tangible interface to interact with video. [A color version of this figure can be viewed online at journals.cambridge.org/aie]
4.6. Sort-and-edit: Tangible sorting of display objects
Sort-and-edit enables sorting sticky notes with multimedia content that are randomly laid out on a physical desktop (covered with a Post-Bit Base Pad), by sorting the Post-Bits that contain the video (Fig. 11). This is similar to the Tangible Bits approach (Ishii & Ullmer, Reference Ishii and Ullmer1997), to position aware e-paper (Ichimura et al., Reference Ichimura, Nelson and Pedersen2000), and to Palette (Nelson et al., Reference Nelson, Ichimura, Pedersen and Adams1999), a system operated by tangible manipulation for presentation slides. Post-Bits' sort-and-edit function enables the tangible manipulation and placement of visual content, and can be used to edit video scene-by-scene, or to sort presentation slides.
Fig. 11. Tangible sorting of Post-Bits as multimedia tags. [A color version of this figure can be viewed online at journals.cambridge.org/aie]
5. PROTOTYPES: DESIGNING A TANGIBLE DESIGN TOOL
We made two prototypes: a paper mockup and a microcontroller-based sensor/LCD system prototype (discussed below). The first, a paper prototype, gave us a sense of people's reactions to the Post-Bit concept, showed us how people tended to handle small sticky memos in this physical form, and helped us decide on its detailed functions. The paper prototype consists of a binder (physically similar to the Post-Bit Binder) made from thick paper, and mock Post-Bits that showed printed key frames from a news video. Each paper had stickum on its back so it could stick to a PC display frame. Interacting with and handling this prototype led us to adjust our ideas of sensor placement on the back or the Post-Bit Players, and gave us needed insight into connectivity between Players in the Post-Bit Binders.
5.1. Integrated microcontroller prototype
Next, a simple electronic prototype of Post-Bits was designed to test the interaction design and embedded sensor operation, using an LCD character display instead of e-paper. An Atmel AT90S4433 microcontroller drives the LCD screen and also handles digital and analog input–ouput (Fig. 12). The sensors currently in use include a button to enable electronic docking, a resistive bend sensor to change the information speed (e.g., frame rate), and a slider that scrolls content back and forward. Although we use a resistive sensor for our prototype, piezoelectric sensors are also a possibility, as are other sensor types like capacitive sensors and pressure address carpets.
Fig. 12. (Right) A microcontroller prototype (Atmel, with LCD screen); (left) mounted to a display frame. [A color version of this figure can be viewed online at journals.cambridge.org/aie]
This prototype is mounted to the PC monitor with velcro (Fig. 13). Two metal contacts are embedded in the velcro to provide the serial connection to the computer. Serial cables are connected to a PC via a RS232C interface tip. Using a virtual desktop built with Flash animation, each object on the desktop can talk to the Post-Bit using a standard network/serial port interface.
Fig. 13. A structural schematic and the data flow of the integrated microcontroller prototype. [A color version of this figure can be viewed online at journals.cambridge.org/aie]
In the current prototype, when we drag-and-drop a file on the desktop into the Post-Bit on the edge, the desktop shows a hand-grabbing icon and transports text content to a serial port. The prototype gets a text line via serial terminals embedded in the velcro and draws it on the LCD module. The prototype continues to show the text even when it is detached from the computer's display frame. The prototype has a flex sensor on board that will control the display of the content, and a small tact switch to trigger the squeeze-it effect for downloading data from the Post-Bit (Fig. 14).
Fig. 14. The embedded tact switch for the squeeze-it function. [A color version of this figure can be viewed online at journals.cambridge.org/aie]
5.2. Animation prototype combining functions: Sort-and-edit plus pick-and-play
Figure 15 shows a Flash prototype we built to try out a video segment sorting process with the Post-Bit system. It shows nineteen key frames from a home video distributed across a Post-Bit Base Pad surface. The surface understands the location of each Post-Bit and can use that information to reorder the files or edit a video stream if desired. Using the pick-and-play function, each Post-Bit plays a movie when it is picked up. Movie identification is achieved by associating a still key frame with its video, which remains on the surface of the Post-Bit until it is picked up.
Fig. 15. An animated prototype: sorting video clips on a surface. [A color version of this figure can be viewed online at journals.cambridge.org/aie]
6. DESIGNING WITH POST-BITS: IDEATION AND STORYBOARDING
This section sketches some of the uses of the Post-Bit system in design. Two of these common design tasks (ideation, swatch-based iteration) apply in many areas of design: architecture, fashion, print media, interactive media, industrial design. The third is more specifically focused on interactive devices or systems or sequential media, where designers must provide alternate pathways through a set of menus, or arrange a coherent storyline (Gill, Reference Gill2003).
We looked at: design ideation and brainstorming; design color/texture/shape/behavior iteration through digital swatching; and interactive storyboarding, useful for designing interactive systems, creating sequential medialike video, or product prototyping for interactive devices.
6.1. Ideation: Visual collages, multimedia swatching
Designers have many methods for ideation that involve collecting images to mix and match. Famously, IDEO uses a set of cards showing contexts and evocative images to jumpstart discussion. Others use brainstorming through informal image collection, bringing magazines, clippings, cards, and even toys or other objects, finding richness in unexpected visual or conceptual juxtapositions. Using Post-Bits as part of this visual collage allows the addition of video clips, active textures (e.g., shimmering water, or trees moving in a breeze), and animated graphics to the collective pool. Digital clips like these have already become part of the modern design vocabulary via device-based image capture like camera phones; useful physical/digital analogs of the sticky note are the ubiquitous image-capture devices with color LCD screens at 2 × 3 or 2 × 4 in. Camera phones and high-quality digital cameras often have color LCD screens of reasonable size and resolution for design note taking or video textures grabs for idea sharing. However, these devices often do not allow for easy uptake of other image types, or simple connection to PCs or software applications. Post-Bits are designed to enable a much more direct connection to computer-based digital design tools like Photoshop or Illustrator; or to other image or graphics-oriented applications like Flickr or even Second Life. Squeezing an image or a video clip into a Post-Bit to use as a design swatch uses the drop-beyond-drag function described above. It is like cutting a small, colorful slice off of your computer screen to lay in among a visual collage, or to stick on a posterboard.
Post-Bits can also be used for multimedia-based color swatching in design: for example, in choosing colors using a physical architectural model. A Post-Bit might contain a video clip that simply consists of a range of colors from blues through greens. Using the flex-and-rub function, a designer can scrub back and forth through the color set to find an appropriate shade, perhaps for a design interior or fabric. Having found a desired shade, the designer then “squeezes” the shade directly into a graphics or computer-aided design (CAD) application on a computer, for use in updating an existing design. The same principle applies for architectural or landscape textures such as brick, stone, grass, or water. Used as a multimedia swatch, the Post-Bit helps to close the digital gap between the physical model in the real world and the CAD model in the computer.
6.2. Storyboarding for video editing or for interactive system and device design
Using the sort-and-edit and flip-and-find stack manipulation features of the Post-Bits system eases the task of storyboarding: whether for a film or video, or for designing the interactive pathways of a system or device. Here's an example of a Post-Bit storyboarding scenario.
In its simplest form, each Post-Bit note initially contains a keyframe, representing a scene or a part of a scene of a film. When arranged on a wall, the Post-Bit notes form a traditional storyboard. Each element can be removed or moved around. Using the Post-Bit Binder, segments can be stacked together and uploaded to a nearby monitor to produce a rough cut by playing them back sequentially according to the order of the stack. A completely different rough cut can then be produced by spreading out the Post-Bits and restacking them in a new order.
Similarly, an interaction pathway for a new device can be storyboarded through video clips or animated illustrations onboard each Post-Bit. For example, a simple on/off choice can be shown by using the Post-Bit's flex function to change its color from red to green, or from an empty radio button graphic to a filled one.
7. CONCLUSION AND FUTURE WORK
The next stages of the Post-Bits project will involve further system development: building another prototype to use video-rate color e-paper when it becomes available, as well as flexible circuitry, and refining application software. We need to extend our application research, to further our understanding of what areas are most fruitful for the use of Post-Bits. We will also revisit the affordances of standard paper sticky notes. For example, one major aspect of the usefulness of handwriting on paper sticky notes seems to lie in connecting the flow of the task in physical space. Finding a way to enable easy handwriting on Post-Bits to mimic this functionality would add a desirable feature (current e-paper technologies cannot accept direct pen input). In addition, as with any new interface device or system, we expect to uncover new insights through managed experiments and limited deployments, along the development pathway.
In this paper, we built and demonstrated a design prototype of the Post-Bit multimedia memo system and discussed some applications for it in design ideation, iteration, and storyboarding. The Post-Bit system's functionalities were based on the use of sticky notes as paper tags, with influences from digital sticky notes systems. We feel it is important to keep the interaction design simple and natural, retaining only fundamental functions that seem like a natural interaction with small papers: flipping, sorting, stacking, sticking. We believe that continuing research in e-paper and in flexible circuitry means that a large number of possibilities exist for design applications of small multimedia sticky memos.
ACKNOWLEDGMENTS
We thank Lynn Wilcox, Jonathan Foote, and the members of the Integrated Media and Immersive Conferencing groups at Fujito Xerox Palo Alto Laboratory (FXPAL). We also thank Eleanor Rieffel and John Doherty for their help in visualizing e-paper technologies.
Maribeth Back is a Senior Research Scientist at FXPAL where she leads the Mixed and Immersive Realities Group, which looks at how the interplay of virtual environments with tangible user interfaces, mobile systems, and sensor-fusion networks can be useful in enterprise settings. Maribeth holds a doctorate from the Harvard Graduate School of Design. Dr. Back's earlier research included smart environments (real and virtual), multimodal interface design, ubiquitous computing, new forms of reading and writing, and interactive audio systems design and engineering.
Takashi Matsumoto is an Interaction Designer working primarily on tangible objects and ubiquitous computing technologies. He earned a PhD in media and governance (media design) in 2008 from Keio University. He designs digital devices exploring the user contexts of urban life in a networked era. Dr. Matsumoto is currently developing hardware and media structure for Pileus, a mobile interaction and display device.
Anthony Dunnigan is a Multimedia Design Specialist at FXPAL, where he has worked on numerous interactive design projects for secure systems, smart environments, and multimedia interfaces. He is interested in developing simple to use interfaces that allow people to make use of complex systems.
