OLPC Sugar Pie Menu Discussion
Excerpts from the discussion on the OLPC Sugar developer discussion list about pie menus for PyGTK and OLPC Sugar.
Discussion with Marco Pesenti Gritti and Eben Eliason about Pie Menus for OLPC Sugar
On February 22, 2007, Don Hopkins wrote:
I’ve translated my fancy OpenLaszlo pie menu component to Python/gtk/cairo, so they work with the OLPC Sugar user interface!
These pie menus have an adaptive layout algorithm, that support linear menus which can lay out in all four directions, pie menus, and hybrid pie/linear menus (automatically limiting the number of pie items to 8, and formatting overflow items in as linear menu items below the pie).
I’m just learning pygtk, so they aren’t as integrated or as fleshed out as they should be yet, and they don’t actually do anything but print debugging messages so far, but this is a good first cut that’s demoable.
I like the cairo drawing module! It’s easy to use, and the graphics and feedback highlighting looks nice.
It’s all written in one file of Python code. You should be able to run the demo, which is included in the file, just by going “python piemenu.py” on an OLPC or Python system with pytgk and cairo installed.
I’d really appreciate it if anyone who knows pygtk would look at the code and give me some feedback about what I’m doing wrong and how to improve them, please.
Next I have to flesh out input tracking and notification, and figure out how to send events during tracking when they pop up, the mouse moves, item changes, when items are selected, sub menus are selected, etc. And then of course there’s a some frosting to implement like keyboard navigation, submenu browsing, screen edge handling, accessibility, popup animation, titles, tooltips, icons, shaped windows, disabled items, etc. Also I’d like to integrate them with glade and the persistence stuff so you can load them in from xml files, edit them with a property sheet, and even edit live menus with direct manipulation, etc.
Here is the source code (which is GPL):
http://www.donhopkins.com/home/code/piemenu.py
I’ve uploaded a picture here:
-Don
—
Subject: [sugar] Pie menus for pygtk and OLPC Sugar!
From: Eben Eliason
Date: 2007–02–23 15:39:29
We discussed the possibility of using pie menus early in the design
stages; I’m not sure why they went out. After considering many of the
interaction design issues we’ve been facing lately, I’m now thinking
that they might have their place on the laptops.The following may require some familiarity with the terms discussed in
the OLPC HIG:
HOW
I think that pie menus are a nice design concept, though their
idiosyncrasies can be a bit tricky. I’ve spent the past day going
over every aspect of the interface with pie menus in mind, and have
come up with a plan for their integration with the current system
which should improve usability. There are many ways to configure pie
menus, but I think the following will handle most cases with the
simplest interaction.1. Primary/Secondary/Tertiary
We can create a format for describing the menus that includes keys
which indicate an item’s level of importance as either primary,
secondary, or tertiary. There will be 4 primaries, 4 secondaries, and
any number of tertiary slices (space allowing, we’ll probably set some
max, perhaps 8). The idea here is that each menu could have 3 levels
levels of detail: in primary mode you would see N, S, W, E; in
secondary mode you would gain NW, NE, SE, SW; and in tertiary mode you
would see all options. The advantages of this will be seen later in
the “WHERE” section.2. Activation methods
There are a few ways that pie menus can be activated. Generally
speaking, the right mouse button will be used to activate pie menus,
though some may appear on rollovers of Objects after short delay.When the menu is explicitly triggered, click and hold is the best
method of interaction. When the mouse remains pressed, the selection
region of the slice can extend beyond the drawn graphic of the menu
right to the edges of the screen. This means that the hit area is
virtually infinite, and lends itself well to gestural selection for
more advanced users. In order to prevent selection, the child could
release the mouse within the interior of the ring, or could press the
esc key to cancel the operation and hide the menu before releasing the
mouse.Unfortunately, when the mouse button is not held, extending the hit
area doesn’t work as well, since the selection will have to happen on
a click event instead of a release. When clicking beyond the rendered
region of the pie, users will expect the menu to simply disappear,
with no selection. Therefore, in cases where the pie menu appears on
instantaneous click or on rollover, the hit area of the slice will be
limited to the graphical region only.3. Visual design
Everywhere in our interface icons are exposed in the first layer and
text is hidden, exposed on rollover as the secondary layer. This
makes things easier for pies, since the biggest problem with them is
dynamically scaling them to fit strings of text. Instead, we can
simply show icons of a fixed size equally spaced around the menu.
This is of great benefit to us, since dynamic rendering of the curves
could be costly.We can therefore reveal the text for a menu item in one of two ways.
It could appear in the center of the pie, adjusting to show the text
for the currently highlighted slice, or it could appear as a tooltip
next to the mouse. The latter might actually be better, since it will
appear next to the cursor, which is the child’s point of focus, and
because it won’t obscure the object in the center that the menu
relates to. If we do this, the rollover does need to be instantaneous
so that the menus are quick at revealing their options.At any point of interaction, the slice that will be selected upon
release of the mouse, or on click (depending on the way the menu was
invoked) will be highlighted.4. (Lack of) Hierarchy
Everywhere in the laptop we are striving to reduce hierarchy, and make
everything that’s important available with as little “digging” as
possible. As such, the pie menus won’t be hierarchical. Instead,
each slice will either be an instantaneous action, or will have a
single secondary panel. When a slice has a secondary panel, it will
have a small arrow which indicates this.The secondary panel can function in one of two ways. First, it can be
a popup list, which is useful when the action needs a target (eg. send
to…(my friend) or open with…(activity)). In these cases, a list
will appear adjacent to the wedge allowing the selection of the
target.A secondary panel can also function as a control window. This means
that a panel will appear, containing any number of necessary controls,
much like a popup window might. This is useful for things like a
preference pane, or a color chooser, etc, where more options and
controls might be needed.WHERE
Pie menus are always tricky around screen edges, and I’m not a fan of
any of the design solutions that aim to fix this. As such, I’m
proposing two very specific yet useful cases where these menus will be
used. They are not a general purpose replacement for all menus in my
mind, but their benefits can be maximized in some cases. Here is a
description of where I think these fit in the Sugar UI.1. Contextual Menus/Rollovers
The current laptop design includes the notion of rollovers for
Objects, which typically exist in two states. A primary rollover
appears to provide a text title for the Object, which is otherwise
rendered as an icon only, and a secondary extension of this appears
after a short delay to provide a preview, summary, and a list of
actions associated with the Object.All contextual rollovers should be replaced with pie menus. The menu
itself would appear, of course, surrounding the object so the icon is
still visible in the center. The text description of the object the
menu relates to can appear with the icon, in the center of the menu
(eg. the child’s name when rolling over an XO).This menu could contain a list of primary actions associated with the
object. For instance, for an XO you could “send message”, “make
friend”, “invite to activity”, etc. Each object type might have a
slightly different list of actions, but there will certainly be
overlap. Where possible, we will define the arrangement of these
actions such that they always appear in the same place, regardless of
the object itself. This is important for spatial recognition, making
gestural activation natural everywhere in the interface. Slices for
actions that don’t apply to an object will be left empty.The only drawback to this over the previous rollover design is that
the actual preview image or summary won’t appear automatically after
delay. Instead, the preview would appear as the secondary panel of a
“preview” action slice in the menu, which would of course be
consistently positioned for every object. The advantage, of course,
is that taking actions on objects is actually quickly exposed, whereas
before they were confined to a secondary rollover state.2. Activity Menu
Until now, we didn’t really have a menu system per se for Activities
at all. Instead, we have a toolbar, where buttons/controls in the
toolbar can have rollover states which provide panels with additional
controls contextual to the button itself. There is a file menu of
sorts in the frame, providing basic features like keep (save), share,
etc, but nothing that the activity developer could extend in any way.I propose that we allow each activity to define its own activity pie
menu. This menu could be invoked by right clicking anywhere on the
screen where there isn’t an object with its own contextual menu. It
could contain any number of items, ranked by the three levels
mentioned above. There could perhaps be a method of selecting which
level of detail you want to see through the interface, but I have
other ideas for this as well.This activity menu can function in two powerful ways. The first is
that it provides a place for activities to define additional
functionality. Until now, with no menu system, everything had to be
exposed on screen. This is a goal we still want to shoot for, and it
should be known to developers that their activities should function
completely without need for the activity menu at all. However, when
additional advanced controls or options would benefit experienced
users, this menu could provide an outlet for some of them without
cluttering the toolbar itself. By design, I would recommend that such
advanced features be ranked as tertiary items.The second and actually more powerful way to use these menus in my
mind is, rather than for advanced functionality, for limited basic
control of the interface. As an example, consider the web browser.
The basic set of controls might be “forward” (right), “back” (left),
“reload” (up), “view source” (down). By defining these as the
primaries, I can instantly invoke these frequently used commands with
gestural ease, without even needing to mess about with toolbar
buttons. This kind of gestural activation of frequent actions could
be fantastic. Similarly, a drawing activity might define “circle”,
“square”, “line”, “brush” as it’s basic set of tools, making selection
of the needed tool a breeze without needing to leave the region of the
drawing being worked on.3. Handheld Mode
Finally, a fantastic side effect of the activity menu is its
application in handheld mode. Previously, we had stated that
activities would have to explicitly implement handheld controls, since
there would be no mouse and keyboard. However, when the activity
menus are designed such that the primary items are the basic key
controls for the activity, any activity can run in handheld mode, at
least in some form.We could make the circle button on the right the menu button. Press
and hold this button to display the pie menu, and use the directional
pad to select the option you want. This is where the ranks for items
also comes into play. Since we can read 8 directions from the d-pad,
we can show either primary or secondary levels of the menu. Of
course, activities could choose to use this button for something other
than the pie menu if they wish — in games, for example — but this
would be a really nice default that would make most activities play
nice with handheld mode “out of the box.”I look forward to hearing what everyone thinks. Also, Marco, I’d like
to hear your thoughts on all of these ideas from an implementation
standpoint.- Eben
—
On February 23, 2007, Marco Pesenti Gritti wrote:
this looks pretty cool!
I just wanted to mention Sugar is not going to use gtk widgets (and glade) as the official/supported way to build activities UI. We are developing a set of controls based on HippoCanvas.
Compared to gtk widgets it’s lighter, it integrates better with cairo, it has more powerful layout support and can also embed gtk widgets. On the down side it’s far from gtk maturity, the only documentation is code at the moment and it miss basic features we will have to implement down the road (mainly focus).
It looks like adapting your pie menus implementation to HippoCanvas would be trivial.
Marco
—
From: Don Hopkins
Subject: Re: [sugar] Pie menus for pygtk and OLPC Sugar!
These sound like great ideas!
The best thing to do is to code up some of these ideas and see how they look and feel, since it’s a visceral thing that is hard to think through in your head, without having something concrete to really play with.
I’ll comment on Eban’s message and Marco’s reply at the same time.
Marco Pesenti Gritti wrote:
On Fri, 2007–02–23 at 15:39 -0500, Eben Eliason wrote:
Eban> Pie menus are always tricky around screen edges, and I’m not a fan of any of the design solutions that aim to fix this. As such, I’m proposing two very specific yet useful cases where these menus will be used. They are not a general purpose replacement for all menus in my mind, but their benefits can be maximized in some cases.
Marco> I fully agree on this, so I’m going to comment on the specific cases.
Screen Edge Handling and Mouse Ahead Display Supression
One solution to the screen-edge problem is to move the menu so it completely fits on the screen, and then to warp the mouse by the same offset. But that has to be done very carefully and synchronously to avoid any problems that would interfere with mouse-ahead. You can defer popping up the menu until ALL events have been processed and the mouse is actually paused, and only then move the cursor instantly after popping up the menu, so the user can mouse ahead without popping up the menu or warping the mouse near the screen edge. I agree that there are a lot of undesirable things about that solution, which is quite complex, violates the principle of least astonishment, and is even impossible to implement in web browsers and other environments that don’t let you warp the mouse (and on hardware like absolute position input devices like touch screens that do not have robot arms to reach out and move the user’s finger on the screen, instead of warping the mouse.)
X10 Pie Menu Window Manager
Demonstration of X10 Window Management Pie Menus implemented as an extension to the “uwm” window manager, incorporating the Sun Forth extension language. Research performed by Don Hopkins under the direction of Mark Weiser. Programmed in Sun Forth implemented by Mitch Bradley. Developed and demonstrated by Don Hopkins. Shows screen edge handling at 3:37.
Another approach to the screen edge problem, which I’ve been using on my Pocket PC implementation of touch screen pie menus, is simply to not use the directions that are off the edge of the screen, for pie menus that pop up along the screen edge or in corners. You can put 1/2 pie menus along the edges (calzone menus?), and 1/4 pie menus in the corners (turnover menus?).
There is of course the disadvantage that fewer directions can fit in partial menus, but there is also the slight advantage that when the user starts the gesture on the edge or in the corner of the screen, they have the whole rest of the screen to move out into, to increase the angular precision of the selection. Edge and corner menus allow you to move a greater distance, for more “leverage”. (This applies more in the case of a small finger-operated touch screen with full-screen pie menus, since most of the screen is very near a corner or edge!) So you might be able to get away with putting a few more items-per-degree on edge and corner menus, than in centrally triggered menus (which only allow you half a screen of leverage).
Pull Out Pie Menus
You can also use the distance independent of the direction, to “pull out” sub-items, so you could have a corner pie menu with three different directions (right, down, and diagonal), or an edge pie with five different directions (left, down/left, down, down/right, right), but each of those distinct directions could “pull out” many different sub-items, because you have the whole width or height or diagonal of the screen to pull out.
Just the Pie Menus from All the Widgets
Pie menu demo excerpts from “All The Widgets” CHI’90 Special Issue #57 ACM SIGGRAPH Video Review. Including Doug Engelbart’s NLS demo and the credits. Tape produced by and narrated by Brad Meyers. Research performed under the direction of Mark Weiser and Ben Shneiderman. Pie menus developed and demonstrated by Don Hopkins. Shows Pull-Out Pie Menus at 2:09.
So one feature I’d like pie menus to support is limiting the start and end direction of the layout, to support fractional pie menus for the screen edges and corners.
Layered Pie Menus
A couple of complementary (but mutually incompatible) features for exploiting the distance of motion are “pull out” slices with sub-items selected by the linear distance from the menu center, and also “layered pies” with rings that have different numbers of items in them, so the inner-most ring can have four items, the next ring 8 items, the next ring 16, and so on.
Eban> 1. Contextual Menus/Rollovers
The current laptop design includes the notion of rollovers for Objects, which typically exist in two states. A primary rollover appears to provide a text title for the Object, which is otherwise rendered as an icon only, and a secondary extension of this appears after a short delay to provide a preview, summary, and a list of actions associated with the Object.
All contextual rollovers should be replaced with pie menus. The menu itself would appear, of course, surrounding the object so the icon is still visible in the center. The text description of the object the menu relates to can appear with the icon, in the center of the menu (eg. the child’s name when rolling over an XO).
Gesture Space
I think it’s important to trigger pie menus on a mouse click (and control them by the instantaneous direction between clicks, but NOT the path taken, in order to allow re-selection and browsing), and to center them on the exact position of the mouse click. The user should have a crisp consistent mental model of how pie menus work (which is NOT the case for gesture recognition). Pie menus should completely cover all possible “gesture space” with well defined behavior (by basing the selection on the angle between clicks, and not the path taken). In contrast, gesture recognition does NOT cover all gesture space (because most gestures are syntax errors, and gestures should be far apart and distinct in gesture space to prevent errors), and they do not allow in-flight re-selection, and they are not “self revealing” like pie menus.
Pie menus are more predictable, reliable, forgiving, simpler and easier to learn than gesture recognition, because it’s impossible to make a syntax error, always possible to recover from a mistaken direction before releasing the button, and “self reveal” their directions by popping up a window with labels, and they “train” you to mouse ahead by “rehearsal”.
NeWS Tab Window Demo
Demo of the Pie Menu Tab Window Manager for The NeWS Toolkit 2.0. Developed and demonstrated by Don Hopkins. Shows mouse-ahead display pre-emption.
Roll-overs that pop up pie menus before you click the mouse could be kind of weird, and could violate the principle of least astonishment. The way some web sites implement roll-over pop up linear menus on web pages is pretty annoying, because the menus often cover up something else that you meant to select, and that is frustrating if you just meant to move the mouse over the menu trigger but not pop up the menu.
The Principle of Least Astonishment
The principle of least astonishment (POLA) (alternatively “principle/law/rule of least astonishment/surprise”) applies to user interface and software design, from the ergonomics standpoint. A typical formulation of the principle, from 1984, is: “If a necessary feature has a high astonishment factor, it may be necessary to redesign the feature.”
So whatever design you use for roll-over feedback, it should be sure to re-center the pie menu on the current cursor position (or god forbid warp the cursor to the menu center where it’s currently displayed), and not get in the way of reliable mouse-ahead. I can imagine a kind of light weight roll-over feedback that previews the pie menu and reveals its directions without actually popping up the menu itself or occluding the view of the object you’re pointing to.
There is an issue (known as the “nulling problem”) that I’ve had to deal with for small touch screen pie menus, because you can’t see through your finger, and the computer can’t forcefully reposition it (hopefully). One solution I’ve developed for my Pocket PC pie menus is for the pie menus to take over the entire screen, and to lay out the items around the edges of the screen (instead of centering them on the place where your finger touched the screen). Then it draws some light-weight feedback that looks like cracks in the glass centered on your finger touch location, that do not occlude the menu labels (the crack feedback is drawn underneath the labels). The cracks delimit the edges of the target regions, and show you where on the screen to touch to select the different items, while the labels are highlighted according to the direction selected by your finger motion, but the labels are always justified in the same place around the screen edge no matter where you touch. It also draws some animated “grippy” lines in the selected slice orthogonal to the slide direction, that slide in and out to show you how it’s tracking the distance inside of the slice.
That’s one solution to the nulling problem for small touch screens (de-couple the touch feedback from the menu label position), which might work with the OLPC’s small screen, or as roll-over feedback.
Pet Rock Remote Control with Touch Screen “Finger Pies” on a PocketPC
The Nulling Problem
Here is some interesting info on the “nulling problem” and other issues, by Bill Buxton:
There’s More to Interaction than Meets the Eye: Some Issues in Manual Input
“Example 3: The Nulling Problem.
One of the most important characteristics of input devices is whether they supply absolute or relative values. Other devices, such as tablets, touch screens, and potentiometers return absolute values (determined by their measured position). Earlier, I mentioned importance of the concept of the “dimension of maximum importance.” In this example, the choice between absolute versus relative mode defines that dimension. […]
From an operator’s perspective, the most annoying part of the above transaction is having to reset the controller before the second parameter can be adjusted. This is called the nulling problem. It is common, takes time to carry out, time to learn, and is a common source of error. Most importantly, it can be totally eliminated if we simply choose a different transducer.” (Buxton 1987)
The stuff Buxton writes about chunking and phrasing of gestural input is also very important for pie menus!
Eban> This menu could contain a list of primary actions associated with the object. For instance, for an XO you could “send message”, “make friend”, “invite to activity”, etc. Each object type might have a slightly different list of actions, but there will certainly be overlap. Where possible, we will define the arrangement of these actions such that they always appear in the same place, regardless of the object itself. This is important for spatial recognition, making gestural activation natural everywhere in the interface. Slices for actions that don’t apply to an object will be left empty.
That’s a good idea! I made a visual PostScript interface that had a standard pie menu that you could pop up on any object, but which had a submenu in a standard direction that always popped up an object-type-specific submenu. So the top level menu was always the same and had common commands, and the object specific menu was always close by in the same direction.
Here’s a paper about it that discusses the pie menu design, and has some pictures:
The Shape of PSIBER Space — October 1989:
The PSIBER Space Deck is an interactive visual user interface to a graphical programming environment, the NeWS window system. It lets you display, manipulate, and navigate the data structures, programs, and processes living in the virtual memory space of NeWS. It is useful as a debugging tool, and as a hands on way to learn about programming in PostScript and NeWS.
And here are some videos:
PSIBER Space Deck Demo
Demo of the NeWS PSIBER Space Deck. Research performed under the direction of Mark Weiser and Ben Shneiderman. Developed and documented thanks to the support of John Gilmore and Julia Menapace. Developed and demonstrated by Don Hopkins.
Pseudo Scientific Visualizer Demo
Demo of the PseudoScientific Visualizer. Research performed under the direction of Mark Weiser and Ben Shneiderman. Developed and documented thanks to the support of John Gilmore and Julia Menapace. Developed and demonstrated by Don Hopkins.
HCIL Demo — HyperTIES Authoring with UniPress Emacs on NeWS
Demo of UniPress Emacs based HyperTIES authoring tool, by Don Hopkins, at the University of Maryland Human Computer Interaction Lab.
Eban> The only drawback to this over the previous rollover design is that the actual preview image or summary won’t appear automatically after delay. Instead, the preview would appear as the secondary panel of a “preview” action slice in the menu, which would of course be consistently positioned for every object. The advantage, of course, is that taking actions on objects is actually quickly exposed, whereas before they were confined to a secondary rollover state.
This is definitely something that’s worth implementing in a few different ways and playing around with to see what feels good.
Marco> I’m not fully convinced about this one.
* As we discussed it would probably not be possible to use it on the frame (since icons are near the edges).
* It would make sense in the Friends view. Though we allow to reposition Xo’s there, so again, they could be near the edges.
* What are the advantages to use them in the mesh? The two primary use cases seem 1 explore and 2 join activities. Pie menus doesn’t seem to make 1 easier (you can just click on the activities) and probably makes 2 a little harder (more indirect preview).
* In the current mesh design you can easily browse through participant names with the little animation you sketched out in the flash demo.
Would that be as easy with pie menus?
Pie Menu Layout and Design
Pie menus are best when the number of items is constant and the menus don’t change and aren’t based on dynamic data. That doesn’t mean you shouldn’t ever do that, just that they’re best when they’re predictable, and a real human being has thoughtfully planned them out (which is impossible for dynamically generated menus).
The Sims uses pie menus whose items are dynamically generated, so they aren’t able to fully exploit the advantages of muscle memory. The game controls the order of the pie menu items which are laid out in a circle clockwise starting from the top, but it does not have any explicit control over the direction of each item.
Some objects have stable pie menus that are predictable (like the fridge and the toilet), where an item in one mode will be replaced by another item in another mode (the fridge lets you fix breakfast/lunch/dinner depending on the time, and the toilet has use, seat up/down, followed by optional flush at the end of the menu).
The pie menus automatically add dummy items to force the number of items to be even (2, 4, 8, and more than 8 go into the linear overflow items). We discussed how to enable the game to control the directions of menu items, but there was no easy solution that didn’t place a burden on the game designers defining the actions on the objects, so we just went with dynamic layout.
The Sims Pie Menus
Layout Parameters and Linear Items
The Python (and OpenLaszlo and JavaScript) pie menus let you specify the layout of each item with parameters like linear (true for linear, false for pie), “max_pie_items” to limit the maximum number of pie items before making items linear, linear_direction (a string specifying the north / up / south / down / east / left / west / right / nw / ne / sw / se linear layout direction for the item) (menus can have mixed pie menus and linear items in different directions), “linear_order_reversed” to reverse the linear layout order (to flip “up” layout so upside-down pull-up menus come out in top to bottom order).
Paging and Scrolling Pie Menus
There is some rudimentary stuff in there for paging pie menus, to support arbitrarily large numbers of items in pages of 8 (or so) at a time, but it’s not hooked up to the user interface or well tested. The idea is that there should be some kind of paging user interface (page up/page down keys, as well as arrow buttons somewhere on the screen, or an inner ring of four pie menu paging items: first/last/next/previous).
ActiveX Pie Menus
Demo of the free ActiveX Pie Menu Control, developed and demonstrated by Don Hopkins. Shows paging pie menus at 4:40.
I’ve tried scrolling menus that let you wind the cursor around the center to “spiral” through a long list of items (demonstrated on Brad Meyer’s “All the Widgets” video he made for ACM SIGCHI), but that turned out to be kind of weird… (But maybe it’s good for scrolling through DNA sequences!)
This clip from All the Widgets demonstrates the scrolling spiral pie menu for selecting country flags, a color wheel pie for changing the text and background color in Emacs, pull-out font family/style/size pie menus, and a “precision pie menu” that lets you precisely select any number from 0 to 360 (by “poking” out through the menu bubble with a flexible rubber band), and also a pie menu window manager supporting mouse-ahead.
Just the Pie Menus from All the Widgets
Pie menu demo excerpts from “All The Widgets” CHI’90 Special Issue #57 ACM SIGGRAPH Video Review. Including Doug Engelbart’s NLS demo and the credits. Tape produced by and narrated by Brad Meyers. Research performed under the direction of Mark Weiser and Ben Shneiderman. Pie menus developed and demonstrated by Don Hopkins.
Eban> 2. Activity Menu
Until now, we didn’t really have a menu system per se for Activities at all. Instead, we have a toolbar, where buttons/controls in the toolbar can have rollover states which provide panels with additional controls contextual to the button itself. There is a file menu of sorts in the frame, providing basic features like keep (save), share, etc, but nothing that the activity developer could extend in any way.
I propose that we allow each activity to define its own activity pie menu. This menu could be invoked by right clicking anywhere on the screen where there isn’t an object with its own contextual menu. It could contain any number of items, ranked by the three levels mentioned above. There could perhaps be a method of selecting which level of detail you want to see through the interface, but I have other ideas for this as well.
Linear Overflow Items and Hybrid Layout
That would be cool! Note that the Python pie menus also support linear layout as well as hybrid pie/linear layout, and the application developer can control which items go into the pie and linear parts, and what directions they’re laid out in.
Of course it would be nice to have even more control of the pie layout, so you can tag pie items with preferred directions and priorities, and a smart layout algorithm could resolve the items into appropriate directions, keep the number of items even by inserting inactive items in unused slots, and apply hints like preferred direction or axis, keeping opposite pairs in opposite directions, etc.
Directional Layout
For example, you could add “on” and “off” as a pair, preferring up for on and down for off with a high priority, then add “next” and “previous” as a pair with no preference for direction, and four other items with no pairing or directional preference but a lower priority, so the “next” and “previous” would be placed left and right because they were more important than the unpaired items, which would go along the diagonals.
If you limited the number of pie items to 8, and added some even lower priority items, they would be relegated to linear items, or to an outer ring.
There should be some way to set up the maximum number of pie items in each ring of a multi-layered pie menu, like the number off electrons allowed in each ring of an atom, and a limit on the maximum number of rings, and also a way to control the linear layout of items that don’t fit in the rings.
JavaScript Pie Menus for Internet Explorer with HTML Behavior Components
Pie menus for JavaScript on Internet Explorer version 5, configured in XML, rendered with dynamic HTML, by Don Hopkins.
Marco> Sort of OT but I really dislike the idea of selecting the level of the detail of the interface. It’s just a question without a good answer. If we decide to go this way we should at least analyze the previous attempts to this kind of interface and figure out why they failed.
Agreed!
Eban> This activity menu can function in two powerful ways. The first is that it provides a place for activities to define additional functionality. Until now, with no menu system, everything had to be exposed on screen. This is a goal we still want to shoot for, and it should be known to developers that their activities should function completely without need for the activity menu at all. However, when additional advanced controls or options would benefit experienced users, this menu could provide an outlet for some of them without cluttering the toolbar itself. By design, I would recommend that such advanced features be ranked as tertiary items.
Agreed!
Marco> This make sense to me but we have to define clearly the use cases. I
don’t want activities to come with dumb interfaces and then put all sort
of “advanced” features in the pie to work around the design
deficiencies.
Agreed!
Mouse Ahead Display Suppression
In the ideal world, all application designers would think carefully about the common use cases, make the interface obvious, self revealing, and usable without menus, and design nice static or smart dynamic pie menus that were easy for novices to learn yet work well as accelerators for expert users (like the X11 SimCity pie menus that are shortcuts for the tool palette).
Here’s a video of the pie menus in the X11 version of SimCity, which shows how they support mouse-ahead, and work as an optional accelerator for the tool palette, which can be tucked away to save screen space once you’re used to the pie menus. Notice the funky “totem pole” layout of the tool palette icons, which are different shapes and sizes according to how they’re used and how much they cost. And how they tool palette is arranged in the same pattern as the pie menus, serving as a map to help you learn the pie menus.
X11 SimCity Demo
Demo of Pie Menus in SimCity for X11. Ported to Unix and demonstrated by Don Hopkins. Shows pie menus saving the day by preventing a train from crashing into a fountain just in the nick of time at 3:30.
Multi Player SimCityNet for X11 on Linux
Demo of the latest optimized Linux version of Multi Player SimCity for X11. Ported to Unix, optimized for Linux and demonstrated by Don Hopkins. Shows pie menu and tool palette demo at 11:52.
User Editable Pie Menus
In another other ideal world (which is ideally the same ideal world as the first ideal world, but it’s too idealistic to expect that), the user should be able to edit the user interface to customize it for their own particular usage patterns (like Maya’s marking menus that the 3D artist can tailor to their workflow).
In a game like World of Warcraft (a real-world virtual universe, but less-than-ideal world), each user has a very particular set of commands they want to use at any particular time (because characters play different roles and have different abilities, spells and items, which changes over time and with different activities), so it’s absolutely necessary to support user-defined interfaces constructed by dragging and dropping command icons around to create your own custom toolbars and pie menus. (I’ve been thinking about how design user-editable pie menus for World of Warcraft, but I’ve been too busy trying to get my character to level 70 to actually implement WOW pie menus yet. But it would be easy, given a good design, since WOW’s user interface is fully programmable in Lua, and already supports dragging and dropping action icons into user defined palettes).
Monster Hunter: World — Radial Menu Guide
Want access to all your items, ammo and gestures at your fingertips? Here’s a quick guide on the Radial Menu. Shows how important customizable user defined pie menus are in real time role playing games.
Eban>The second and actually more powerful way to use these menus in my mind is, rather than for advanced functionality, for limited basic control of the interface. As an example, consider the web browser. The basic set of controls might be “forward” (right), “back” (left), “reload” (up), “view source” (down). By defining these as the primaries, I can instantly invoke these frequently used commands with gestural ease, without even needing to mess about with toolbar buttons. This kind of gestural activation of frequent actions could be fantastic. Similarly, a drawing activity might define “circle”, “square”, “line”, “brush” as it’s basic set of tools, making selection of the needed tool a breeze without needing to leave the region of the drawing being worked on.
Marco> I really like this use case.
HyperTIES Hypermedia Browser and Emacs Authoring Tool
Me too! We used pie menus for paging and navigation in HyperTIES hypermedia browser, and the Emacs authoring tool, which I developed for Ben Shneiderman at the University of Maryland Human Computer Interaction Lab:
HCIL Demo — HyperTIES Browsing
Demo of NeWS based HyperTIES authoring tool, by Don Hopkins, at the University of Maryland Human Computer Interaction Lab.
HCIL Demo — HyperTIES Authoring with UniPress Emacs on NeWS
Demo of UniPress Emacs based HyperTIES authoring tool, by Don Hopkins, at the University of Maryland Human Computer Interaction Lab.
Designing to Facilitate Browsing: A Look Back at the Hyperties Workstation Browser
We present application of pie menus to permit low cognitive load actions that reduce the distraction of common actions, such as page turning or window selection.
3. PIE MENUS TO PERMIT LOW COGNITIVE LOAD ACTION
The Hyperties browser uses pie menus as accelerators, to make commonly used commands quickly and easily available. A pie menu is a type of pop up menu whose selections are laid out in a circle around the menu center (18). The menu pops up centered on the cursor, so that each selection is adjacent to the cursor but in a different direction (Figure 1 and 6). A selection is made by moving the cursor in the direction of the desired selection, and clicking. Experienced pie menu users can make selections from familiar menus quickly and reliably without even having to look at the menu, because the menu selection depends on the direction between the two mouse clicks that invoke and select from the menu. The distance of cursor motion does not effect the selection, but the further away from the center the cursor is, the more precise the control of the selection is.
The browser has a control panel at the bottom of the screen, with buttons showing the names of available commands, to turn the page, return to the previous article, show the index, etc. When users are browsing a document by pointing and clicking on highlighted text links in the main contents window, they move the cursor down to the bottom of the screen to press buttons in the control panel, and back up to continue browsing. The permanent display of those controls is important for the novice and occasional users. On the other hand, pop up menus reduce the distraction of moving the cursor by making these commands available wherever the cursor currently is. This reduces perceptual and motor load. Pie menus are arranged with their items in easy to remember directions. For example the BACK page turning commands are to the left (and the NEXT page is to the right) (Figure 6). This arrangement facilitates gestural input and encourages development of muscle memory. Experienced users can make gestural selections from these menus so comfortably and rapidly that it is often unnecessary to display the menu. This is called “mouse ahead display suppression”, and its point is to reduce the perceptual distraction.
[…]
The more recent NeWS version of Hyperties on the SUN workstation uses two large windows that partition the screen vertically. Each window can have links and users can decide whether to put the destination article on top of the current window or on the other window. The pie menus made it rapid and easy to permit such a selection. When users click on a selectable target a pie menu appears (Figure 1) and allows users to specify in which window the destination article should be displayed (practically users merely click then move the mouse in direction of the desire window) . This strategy is easy to explain to visitors and satisfying to use. An early pilot test with four subjects was run, but the appeal of this strategy is very strong and we have not conducted more rigorous usability tests.
In the author tool, we employ a more elaborate window strategy to manage the 15–25 articles that an author may want to keep close at hand. We assume that authors on the SUN/Hyperties will be quite knowledgeable in UNIX and Emacs and therefore would be eager to have a richer set of window management features, even if the perceptual, cognitive, and motor load were greater. Tab windows have their title bars extending to the right of the window, instead of at the top. When even 15 or 20 windows are open, the tabs may still all be visible for reference or selection, even thought the contents of the windows are obscured. This is a convenient strategy for many authoring tasks, and it may be effective in other applications as well.
Marco writes:
Marco> I have a couple of general concerns about the activity menu.
One is conceptual. The actions are global but both the visual
representation and the controller (mouse button) are contextual. I tried
the RadialContext extension in firefox and this aspect really feel odd
to me.The other is about interaction. We conflict with context menus so the
user would have to be careful to hit a part of the screen where there
are not other context menus to activate them.Eban> 3. Handheld Mode
Finally, a fantastic side effect of the activity menu is its
application in handheld mode. Previously, we had stated that
activities would have to explicitly implement handheld controls, since
there would be no mouse and keyboard. However, when the activity
menus are designed such that the primary items are the basic key
controls for the activity, any activity can run in handheld mode, at
least in some form.We could make the circle button on the right the menu button. Press
and hold this button to display the pie menu, and use the directional
pad to select the option you want. This is where the ranks for items
also comes into play. Since we can read 8 directions from the d-pad,
we can show either primary or secondary levels of the menu. Of
course, activities could choose to use this button for something other
than the pie menu if they wish — in games, for example — but this
would be a really nice default that would make most activities play
nice with handheld mode “out of the box.”
Marco> I really like this.
Absolutely! The Sugar pie menus could display graphics and icons corresponding to the hardware buttons to reinforce the correspondence. A great side-effect of limiting (or raising) the number of pie menu items to four or eight, is that they map so nicely to other input devices like game controllers, joysticks, numeric keypads, arrow keys, etc.
Eban> I look forward to hearing what everyone thinks. Also, Marco, I’d like to hear your thoughts on all of these ideas from an implementation standpoint.
Marco> I think it’s very unlikely we will have a composite manager for generation 1. That means we will have to use shaped windows to implement it, which is possible theoretically, but I think no one actually tested the performance of this on the OLPC. Before putting a lot of design work on this I think it would be wise to try out a prototype on the olpc (especially since we have already Don implementation and we can easily add a shaped window to it).
Marco
I like your suggestion to factor the code so it uses Cairo but is not directly tied to GTK, and is easy to hook into a pure Cairo based rendering environment and whatever input tracking system it uses. That will make it easy to port the GTK based pie menus to the official Cairo based compositing renderer, when it’s ready, while in the mean time developing window system independent features and drawing pretty scalable graphics in Python/Cairo.
-Don
—
