Pie Menu FUD and Misconceptions
Dispelling the fear, uncertainty, doubt and misconceptions about pie menus.
This is the infamous Fanged Apple, a design created in protest of Apple’s lawsuit regarding graphical user interfaces (GUI) software.
“In 1989, Apple’s look and feel lawsuit against Microsoft brought the idea of interface copyright to the attention of the programming community. Apple was trying to claim a monopoly over a broad class of graphical user interfaces. If Apple succeeded, not only Microsoft, but every software developer, would be legally compelled to design gratuitously incompatible software.” In response, John Gilmore and Richard Stallman together commissioned the design of the “fanged Apple”–a button showing the Apple logo with vicious teeth. This was followed by an advertisement placed in The Tech, MIT’s student newspaper, which in turn led to a protest rally at the HQ of Lotus Development Corporation — another “look and feel” lawsuit plaintiff.” -History of the League for Programming Freedom
Huge Problem: Software Patents and FUD
There is a sad history of people using software patents to make misleading claims about obvious techniques that they didn’t originate, and constructing flawed straw man definitions of ersatz pie menus to contrast with their own inventions, to mislead the patent examiners into granting patents.
There is a financial and institutional incentive to be lazy about researching and less than honest in reporting and describing prior art, in the hopes that it will slip by the patent examiners, which it very often does.
Patent Abuse Example: US Patent US5689667A: Methods and system of controlling menus with radial and linear portions
Unfortunately a bad patent that covered an obvious technique, and also made some incorrect misleading claims, was abused by Alias marketing in Bill Buxton’s name to baselessly threaten and discouraged others from using pie or marking menus, by exaggerating its scope and obfuscating its specificity. It’s my strong opinion that the particular technique that it covered (overflow items) was quite obvious.
Gordon Kurtenbach and I discussed pie and marking menus in 1990 before he wrote his paper and filed the patent, and at that time he made it clear that he understood pie menus supported mouse ahead display suppression, and that pie menus enjoyed the same benefits as marking menus have in easing the transition from novice to expert user:
“The the cool thing is that expert can mouse ahead like you’ve talked about but they get an ink trail so they have a better idea what they’ve selected without even bothering to wait for the menu to come up.”
However that contradicts what the paper and the patent implies, and it’s misled other people into incorrectly believing that pie menus don’t support what I call “mouse ahead display preemption” (or “suppression”, a harsher word), and that the patent covers much more than it actually does.
When Gordon applied for the patent on in 1995, which his employment contract with SGI required him to do, the patent had at least two misleading statements, and the “overflow” technique claim was obvious, which should have prevented it from being granted or invalidated it.
Another piece of mistaken but published misinformation about the differences of “typical pie menus” and marking menus is that “typical pie menus” pop up submenus after the cursor has moved a certain distance from the menu center, without clicking the mouse button. However, I have never seen nor implemented such badly designed pie menus in the real world.
“Typical pie menus” (such as those in The Sims, played by hundreds of millions of people) have always selected leaf and submenu items by triggering on a button press or release (or pen or finger tap or release). They also typically support mouse-ahead. Pie menus can seamlessly support both quick press-drag-release gestures, as well as the more leisurely click-move-click gestures.
The patent US5689667A “Methods and system of controlling menus with radial and linear portions” also makes the mistake of claiming that that pie menu selection is based on pointing at the items like linear menus (or PIXIE), instead of the direction of cursor motion, which Kurtenbach and Buxton know very well is simply not the case with “typical pie menus”.
“Radial menus include two types: pie menus and marking menus. Pie menus are typically used in item selection using the location principles of linear menus as discussed above. Marking menus operate on the principle of the direction of cursor or pointer motion as being the basis for item selection.” -US Patent US5689667A
Unfortunately they were able to successfully deceive the patent reviewers, even though the patent references the Dr. Dobb’s Journal article which clearly describes how pie menu selection and mouse ahead work, contradicting the incorrect claims in the patent. It’s sad that this kind of deception and patent trolling is all too common in the industry, and it causes so many problems.
Even today, long after the patent has expired, Autodesk marketing brochures continue to spread FUD to scare other people away from using marking menus, by bragging that “Patented marking menus let you use context-sensitive gestures to select commands.”
The Long Tail Consequences of Bad Patents and FUD
I attended the computer game developer’s conference in the late 90’s, while I was working at Maxis on The Sims. Since we were using 3D Studio Max, I stopped by the Kinetix booth on the trade show floor, and asked them for some advice integrating my existing ActiveX pie menus into their 3D editing tool.
They told me that Alias had “marking menus” which were like pie menus, and that Kinetix’s customers had been requesting that feature, but since Alias had patented marking menus, they were afraid to use pie menus or anything resembling them for fear of being sued for patent infringement.
I told them that sounded like bullshit since there was plenty of prior art, so Alias couldn’t get a legitimate patent on “marking menus”.
The guy from Kinetix told me that if I didn’t believe him, I should walk across the aisle and ask the people at the Alias booth. So I did.
When I asked one of the Alias sales people if their “marking menus” were patented, he immediately blurted out “of course they are!” So I showed him the ActiveX pie menus on my laptop, and told him that I needed to get in touch with their legal department because they had patented something that I had been working on for many years, and had used in several published products, including The Sims, and I didn’t want them to sue me or EA for patent infringement.
When I tried to pin down the Alias marketing representative about what exactly it was that Alias had patented, he started weaseling and changing his story several times. He finally told me that Bill Buxton was the one who invented marking menus, that he was the one behind the patent, that he was the senior user interface researcher at SGI/Alias, and that I should talk to him. He never mentioned Gordon Kurtenbach, only Bill Buxton.
So I called Bill Buxton at Alias, who stonewalled and claimed that there was no patent on marking menus (which turned out to be false, because he was just being coy for legal reasons). He said he felt insulted that I would think he would patent something that we both knew very well was covered by prior art.
I told him that companies try to get bad patents all the time, and that I did not mean to insult him by simply repeating to him the misinformation that his marketing people were spreading around the computer industry, in his name. I told him that should not shoot the messenger, and he should reel in his marketing department and tell them to stop spreading FUD in his name, which he refused.
I tried to explain how Alias’s FUD had adversely affected the user interface design of 3D Studio Max, who refused to implement pie menus in spite of user requests for them. But he did not care about 3D Studio Max, since Kinetix was his competition. I asked him whose side he was on, the user’s or the patent lawyer’s?
He claimed to be on the side of the users, since he is such a well known user interface researcher, but I believe he has sold out to big corporations like Alias, SGI, Microsoft and Autodesk by abusing the patent system for profit, and is in the thrall of corporate lawyers, so he wasn’t being honest. Users beware!
The Sims Pie Menus
This is a demonstration of the pie menus, architectural editing tools, and Edith visual programming tools that I developed for The Sims with Will Wright at Maxis and Electronic Arts.
Electronic Arts released The Sims with pie menus in 2000 to critical acclaim, and never had any problem with them or heard a peep from Alias, and continue to use pie menus to this day, because I was fully confident that Bill Buxton and Alias had no leg to stand on with their bad patent, only FUD and bluster.
Open Sourcing SimCity: Chaim Gingold’s “Play Design” PhD Thesis
Pie menus play a critical role in The Sim’s user interface design, dovetailing perfectly with the object and AI architecture. Objects advertise verbs to character AI, so it is natural for the verbs to be arranged in a radial menu about objects. I can’t imagine an alternate design that would have had the same widespread usability, and therefore appeal, without them. It is difficult to imagine The Sims without pie menus. -Chaim Gingold, Play Design PhD Thesis, Open Sourcing SimCity
The Marking Menu Patent Finally Surfaces
A year or so later, I finally ran across the marking menu patent issued to Alias, which is undoubtedly the one the Alias sales people were spreading rumors about while it was still pending, which explains why Buxton was being so coy about it. Now it all makes a lot more sense in perspective.
Around the time I found out about it from Kinetix, Alias had just applied for the patent on marking menus. The Alias sales people had heard Buxton bragging about it, didn’t understand it, but could not keep their mouths shut about it, even though there were damn well supposed to.
So they did their job and spread Fear, Uncertainty and Doubt by bragging about this PENDING patent that they really didn’t know much about, implying it covered much more than it really did, in order to stifle the competition (like 3D Studio Max) as much as possible. The only reason I ever learned about it, was that their FUD was so successful that it actually affected Kinetix’s plans against the will of their users, then they told me about it, and I followed up on it.
When it got back to Buxton that his marketing team had leaked news of the pending patent to Kinetix and myself, which was supposed to be kept secret, he was quite furious, but certainly wasn’t honest with me about what was really up with the patent, so he took his anger out on me instead.
Since the Alias marketing people claimed that Bill Buxton patented marking menus, I suspect that Bill had been bragging about the patent to his marketing team, and failed to mention that marking menus were Gordon Kurtenbach’s invention, not his. So of course the marketing people were quick to mention Buxton’s name and didn’t know who Kurtenbach was, when they told me about the marking menu patent.
Apparently Buxton wanted to keep me in the dark about the patent they’d applied for, so I didn’t go to the U. S. Patent Office and inform them of the mistakes and misrepresentations of prior art in the patent. But nonetheless he bragged about the patent enough to marketing that they leak FUD about it in his name, which successfully discouraged Kinetix’s plans to put marking menus into 3D Studio Max. It’s a textbook example of successful FUD!
Gordon Kurtenbach agreed that Bill Buxton didn’t handle the situation as well as he should have:
Don, I read and understand your sequence of events. Thanks. It sounds like it was super frustrating, to put it mildly. Also, I know, having read dozens of patents, that patents are the most obtuse and maddening things to read. And yes, the patent lawyers will make the claims as broad as the patent office will allow. So you were right to be concerned. Clearly, marketing is marketing, and love to say in-precise things like “patented marking menus”.
At the time Bill or I could have said to you “off the record, its ok, just don’t use the radial/linear combo”. I think this was what Bill was trying to say when he said “there’s no patent on marking menus”. That was factually true. However, given that Max was the main rival, we didn’t want to do them any favors. So those were the circumstances that lead to those events.
In any case, I did not let that discourage me from my long term plan of incorporating pie menus into mainstream products (like The Sims from Maxis and SimCity for the OLPC) and user interface toolkits (including ActiveX, GTK, Sugar, jQuery, Unity3D, etc). Since I couldn’t afford to challenge the patent myself, that was the only way I had to ensure that many people would be able to use and benefit from pie menus.
When the users of a program like 3D Studio Max demand a feature like pie menus from companies like Kinetix, or game developers like Maxis want to use pie menus in their games, they should not be fooled by FUD spread by big patent-obsessed corporations like Alias, SGI, Microsoft or Autodesk. (Ironically, both 3D Studio Max and Alias are now owned by Autodesk!)
Decades of FUD and Marketing Bullshit Have Taken Their Toll
Gordon confirmed to me that indeed, the FUD about the marking menu patent that Alias marketing was spreading on Buxton’s behalf did have a detrimental effect on 3D Studio Max users who wanted marking or pie menus, including myself and my colleagues at Maxis developing the content for The Sims:
After Autodesk acquired Alias, I talked to the manager who was interested in getting pie menus in Max. Yes, he said he that the Alias patents discouraged them from implementing pie menus but they didn’t understand the patents in any detail. Had you at the time said “as long we don’t use the overflows we are not infringing” that would have been fine. I remember at the time thinking “they never read the patent claims”.
The 3D Studio Max developers heard about the Alias marking menu patent from Alias marketing long before I heard of it from them on the trade show floor.
The reason I didn’t know the patent only covered overflows was that I had never seen the patent, of course. And when I asked Buxton about it, he lied to me that “there is no marking menu patent”. He was trying to be coy by pretending he didn’t understand which patent I was talking about, but his intent was to deceive and obfuscate in order to do as much harm to Kinetix 3D Studio Max users as possible, and unfortunately he succeeded at his unethical goal.
The Alias Marking Menu Patent Discouraged the Open Source Blender Community from Using Pie Menus for Decades
Here is another example that of how that long term marketing FUD succeeded in holding back progress: the Blender community was discussing when the marking menu patent would expire, in anticipation of when they might finally be able to use marking menus in blender (even though it has always been fine to use pie menus).
As the following discussion shows, there is a lot of purposefully sewn confusion and misunderstanding about the difference between marking menus and pie menus, and what exactly is patented, because of the inconsistent and inaccurate definitions and mistakes in the papers and patents and Alias’s marketing FUD:
Hi. In a recently closed topic regarding pie menus, LiquidApe said that marking menus are a patent of Autodesk, a patent that would expire shortly. The question is: When ? When could marking menus be usable in Blender ? I couldn’t find any info on internet, mabie some of you know.
The good news: Decades late due to patents and FUD, pie menus have finally come to 3D Studio Max just recently (January 2018)!
Ersatz Pie Menus
Richard Stallman likes to classify an Emacs-like text editor that totally misses the point of Emacs by not having an extension language as an “Ersatz Emacs”.
In the same sense, there are many “Ersatz Pie Menus” that may look like pie menus on the surface, but don’t actually track or feel like pie menus, or benefit from all of their advantages, because they aren’t carefully designed and implemented to optimize for Fitts’s Law by being based purely on the direction between stroke endpoints instead of the entire path, minimizing the distance to the targets, and maximizing the size of the targets.
Microsoft Surface Dial
Someone on Hacker News asked me: Any thoughts on Microsoft’s Surface Dial radial menu?
Good question — glad you asked! (No, really! ;)
Turning a dial is a totally different gesture than making directional strokes, so they are different beasts, and a dial lacks the advantages pie menus derive from exploiting Fitts’s Law.
Even though the Surface Dial has a round dial, that is nothing like a pie menu, even though it’s round, because you turn it. Pie menus aren’t about turning around the center, they’re about stroking out from the center.
A dial, rotating carousel, or rocker switch that you turn or nudge clockwise or counterclockwise to change between different items, and then click to select an item, are not anything like pie menus, no matter what they’re called or how snazzy they look.
You have to turn a dial or carousel more and more to select each subsequent item. (Just like you have to move the cursor more and more downwards to select each subsequent item of a linear pull-down menu).
The mountain should come to Muhammad, so Muhammad doesn’t have to go to the mountain. So pie menus should always pop up centered on the mouse, so they come to you and you don’t have to move over to them, and each item starts out equidistant, nearby, and in a different direction.
Turning Is Not Like Stroking
In terms of “Big O Notation”, pull down menus, click wheels, and carousel selection is linear O(n), while with a pie menu you only have to perform one short directional gesture to select any item, so selection is constant O(1) (with a small constant, the inner inactive radius of the hole in the middle, which you can make larger if you’re a spaz).
Yucky Pie Menus Recipes
Bedazzling and Confusing Graphics and Animations
Try to avoid gratuitous designs with pointless animations that just bedazzle and confuse users, instead of help them. Animations should never slow down interaction, interfere with mouse-ahead, cause events to be lost, obscure or delay important information, or make the user stop and wait.
The items themselves should never rotate around the menu (unless you’re just doing some quick transient snazzy spin-up animation), they should always stay in the same direction.
Here are some window management pie menus that spin up if you click them up without moving, and tilt up around the axis perpendicular to the direction of motion, if you move away from the center before they pop up:
Rectangular Label Targets Instead of Wedge Shaped Slice Targets
Another way to implement frustrating difficult to use ersatz pie menus that totally miss the point is to only use the small areas of the item labels as targets, not the entire huge wedge shaped pie slices extending out to the screen edge.
Triggering Items and Submenus on Cursor Motion Distance Instead of Clicking
Yet another way to screw them up is to trigger them when you move a certain distance from the center (or pop them up when you roll over a target without clicking, or use time-outs without clicking), instead of when you click or release the mouse (or tap or release your finger, pen, etc). There should always be a kinesthetic delimiter like a click or tap at the beginning and ending of the stroke, but the stroke is free wander around any path or pause for any duration. Only the angle between the delimiters matters, to allow for browsing and reselection and error correction.
If distance or time is the trigger, the user has no way of reliably and directly controlling or sensing at which point the selection happens, or using the menus without looking at the screen, and it terribly interferes with navigating nested pie menus.
Not Starting Pie Menus Centered on the Cursor
Any pie menu that does not initially pop up with the cursor (or your finger) in the center, or that is already showing before you start tracking from anywhere on the screen, is terribly broken, because the whole point is to bring the pie menu center to you, so you can select with quick directional gestures without looking at the screen, not for you to have to first look at the screen and then move to the center yourself. That would totally defeat the purpose of pie menus!
Improperly Handling Screen Edges
Handling the screen edge problem can get tricky, especially when combined with mouse-ahead display suppression. One way of handling that is to “warp” the mouse back to the center of the menu if the menu needs to be moved to fit on the screen. But you must be careful not to cancel out any motion away from the center that’s already happened (which is why you should delay warping the mouse until you actually pop up the menu (if ever), so there is no mouse warping when you mouse ahead and the menu is not displayed).
Web browser don’t directly support mouse warping (i.e. like XWarpPointer), and you can’t tell if the edge of the browser window is actually the edge of the screen, but you can use the “Pointer Lock API” to implement a software cursor that you can warp anywhere in the window you want (but not outside).
Improperly Handling Mouse-Ahead Display Preemption and Quick Gestures on Busy Computers
If implemented properly, pie menus can be extremely reliable and satisfying to use on a slow or busy computer. But it’s hard to get right, it can be quite tricky and system dependent, and there are also some subtle nuances about input handling, polling, interactions between mouse-ahead and screen edge handling that are important to get right.
It’s important to properly handle mouse-ahead display preemption in a way that none of the user’s mouse motion is lots, even when the system is very busy. None of the user’s mouse clicks or motion should ever be ignored or lost between the cracks. Not all user interface systems and web browser have adequate event handling and cursor control to get all the details perfectly right, unfortunately. But it’s important to make them as reliable as possible.
There’s a lot more discussion of screen edge handling and mouse-ahead display preemption here:
Also see the discussion of “gesture space” here:
Yummy Pie Menu Recipes
I’m certainly not saying that pie menus should never be graphically slick or have lots of cool animations. Just that they should be thoughtfully designed and purposefully easy to use first, so they deeply benefit users from Fitts’s Law, instead of just trying to impress users with shallow useless surface features.
Spectacular Example: Simon Schneegans’ Gnome-Pie, the slick application launcher for Linux
I can’t understate how much I like this. Not only is it slick, beautiful, and elegantly animated, but it’s properly well designed in all the important ways that make it Fitts’s Law Friendly and easy to use, and totally deeply customizable by normal users! It’s a spectacularly useful tour-de-force that Linux desktop users can personalize to their heart’s content.
Gnome-Pie is a slick application launcher which I’m creating for Linux. It’s eye candy and pretty fun to work with. It offers multiple ways to improve your desktop experience.
Check out the project’s homepage @ http://gnome-pie.simonschneegans.de
