Applying Gestural Interfaces to Command-and-Control

Last fall, I was fortunate enough to be tasked with a fascinating challenge by the US Army's Command-and-Control Directorate (C2D). Along with Martina Balestra, a co-worker at MITRE and fellow Human Factors Engineer, I had the opportunity to explore the possible application of gesture-based interface technologies to command-and-control (C2) workflows and environments. The work that we produced, entitled "Applying Gestural Interfaces to Command-and-Control (C2)" was subsequently submitted and accepted to be presented at HCI International 2011 in Orlando, Florida this July. Since public release was required for this work to be presented, I now have the ability to share it. 

The following is a short abstract of the paper in addition to some concept sketches developed as part of the project. Please contact me if you're interested in hearing more about our process! I'm proud of this work as it was one of the most rewarding challenges of my professional career. 


Abstract (Approved for Public Release: 10-3988. Distribution Unlimited)

This whitepaper examines the applicability of gesture-based user interfaces in notional Command-and-Control (C2) environments of the United States Army. It was authored by a team of Human Factors Engineers at The MITRE Corporation, a not-for-profit research and development organization funded by the United States Government. Since MITRE resides in a not-for-profit advisory position to their federal sponsors, the research team was able to take an unbiased perspective driven solely by identified issues, the search for improved workflows, and practical opportunities for technology development. The goal of the effort was to inform the US Army community so that it can make responsible, needs-driven decisions regarding gestural interface technologies, and avoid the potential pitfalls that may arise from technology-centered or profit-driven decisions.

The problems focused upon by this research primarily revolved around the collaborative human workflows that occur within Command-and-Control environments. Specifically, the effort targeted US Army-based C2 environments, such as a notional fixed command center, a mobile command center, and the environment of the dismounted soldier in the battlefield. The primary issue is that the currently-implemented technologies, while independently sufficient, present constraints when distributed personnel are collaborating across them. The research team addressed this cross-platform issue by adhering to a Systems Engineering framework that required a holistic approach to the “system” of distributed C2 personnel and their technologies. The goal for the final output was to demonstrate how these technologies may come together as a system to support a more efficient, dynamic, and effective operational workflow than today’s reality.

After carefully examining the field of current and emerging gestural interface technologies, and mapping them against available HCI-related research findings, the team concluded that US Army personnel may indeed benefit from effectively and appropriately implemented technologies from this domain. At a high level, gestural technologies offer C2 personnel an ability to conduct more efficient and collaborative workflows across distributed environments. The exact details of these workflows, including the key users, actions, and technology paradigms, are outlined in the content of the whitepaper. In an effort to be as prescriptive as possible, the research team decided that it would be valuable to include a sizable section within the whitepaper dedicated to instructing the user on how to implement gestural technologies for C2 application. In this section, they outline the key design patterns to selecting proper solutions and developing effective interaction design frameworks. The nature of this instructional portion ranges from high-level design principles and best practices down to detailed visual demonstrations of recommended gestures. 

(Approved for Public Release: 10-3988. Distribution Unlimited)

Ideation-to-Innovation: An Evolving Storyboard

As I've continued my examination of the ideation process this semester, I have become increasingly convinced of the benefit of graphical depictions of complex systems such as this. My most recent work has been done in collaboration with Matt Harper, an MIT System Design & Management fellow who is equally as interested in the product design/innovation process. The following is our in-progress storyboard for our course in System Visualization. The final output will be a more polished and animated version, similar to the RSA Animation series. http://www.thersa.org/. This is very much a draft so please fire away with ideas for improvement. 

The Story

Here you are, taking your daily two mile walk from work to the train station. Today’s like any other day…  that is until a cool wind picks up and a dark sky forms over your head and it begins to rain…really rain. Of course, you forgot your umbrella and are not even wearing a decent raincoat. You continue to walk, thinking about your lousy, soaked situation…

Then, it dawns on you What if there was an application for your phone that monitored the weather, recognized that you were walking, identified your contacts that were driving nearby, and notified them of your situation.

Your kind friends could excitedly rush to your rescue with one lucky companion getting the honor of scooping you up in their warm, day, coffee-ready sedan.  Like that, an idea is formed. Nothing real or tangible was actually created of course, but a notion of a possibility was conceived. 

Circumstances Cause Intent

So where did that idea come from? It certainly didn’t fall from the sky with the rain…or did it?  The unexpected rain combined with your lack of umbrella created a circumstance that served as a catalyst for an idea to be formed. You came up with the idea because you wanted to reduce your discomfort in the environment. You wouldn’t have come up with that idea if you strolling comfortably on a sunny day.  

Without that circumstance, problem, or challenge, that creative spark may never had existed! That may sound foolish when you consider the “pick your friend up” mobile application, but what if the idea had a much more profound contribution, say to a cure for cancer or source of sustainable energy. The beauty is not in the idea itself, but in the concept that the challenge presented by the environment triggered your problem-solving creativity to produce something novel. Circumstances create intent to form ideas. 

Intent Relies on Readiness

Of course, the formation of the idea is not even possible without some form of readiness on the part of idea’s creator. It took you having at least some knowledge of the possibilities of today’s mobile phones. And of course, it took an understanding of the generally accepted ways that we help each other, and how giving a ride to a friend in the rain is a perfectly reasonable gesture. Whatever ideas you come up with are going to be products of your knowledge and experiences. In a sense, the ideas are novel combinations of everything you know, filtered in order to achieve a specific intent.

Readiness Relies on Environment

So the idea has been conceived based upon novel combinations of the information in your head. But where did that knowledge come from? From your environment of course… the experiences you’ve had, the people you’ve known, the books you’ve read, the institutions you’ve attended, and the culture you’ve lived in all play a part in forming your knowledge base and values.

This concept becomes even more powerful when you consider the effect of many people working together to form an idea. After all, if ideas are simply novel combinations, simply adding people will greatly increase the chances of great ideas, right? Well not exactly, but we’ll address that in a minute.

Stepping Back: Ideation as a System

Stepping back from the simple mechanics of a single idea, one could see that this ideation environment is really a system. Each component plays a critical role within it, feeding into each other and contributing to the ever-changing form and overall value of the system. Ideas are how we evolve ourselves and our environment.

Not surprisingly, the Ideation System mimics many of the patterns we see in nature. Ideas, much like seeds, play a fundamental role in the evolution of a society. Ideas are constantly forming and germinating in the environment… some catch on and grow, while others fail to catch on. Much like seeds, an environment needs to foster a high quantity of ideas in order to flourish and evolve.

Much like sunlight and rain for seeds, ideas require proper conditions from which to form… and once they do form, they need the cultivation of a supporting environment and culture in which to grow.  

The Tiers of Innovation
Of course, this process of filtering from environment to a ready idea creator to the idea itself is not as simple or mechanical as this model presents it. Culture plays a critical role… the promotion of education, the exploration of new ideas, the questioning of old ones… all require the proper cultural environment. A culture of learning, communication, and experimentation are all fundamental to the formation and realization of great ideas. Let’s take a closer look to see how that culture influences innovation…  

Consider an environment where there is a strong culture of communication, idea exploration, experimentation, and acceptance of failure. The environment is dynamic and rich with cross-pollination. People learn from institutions and organizations, develop new theories and innovative thoughts, and in turn, contribute back into society.

In a very evolutionary way, organizations and individuals evolve through the discovery and advancement of knowledge. They are constantly consuming and organizing their new information to fit their values and mental models of the world.  Advancement of organizations and individuals is often done through the production of ideas, pulling knowledge and capabilities from the environment in novel or unexpected ways to address emerging circumstances

 

Lenses

This may appear all too chaotic and random, and to some extent it is. There is a great deal of forces at play within an open society, from the free sharing of ideas from people to the deliberate intent of organizations and institutions. If one were to examine the flow of information and ideas in a society, they could look at it from a range of perspectives, from cultural influence to structured regulatory effects. 

Taking this approach may allow one to even examine the story behind an idea, as it travels from the foundational knowledge and capabilities from the environment that made it possible to the individual or organization that consumed that information to the final idea that was conceived and then added back into the system for potential growth.

Interested in seeing how the project turned out? here it is..  

The Russian Doll Theory of Ideation

I've often discussed my research into the ideation and innovation process. This work is most recently being conducted as a "systems visualization" effort in alignment with MIT's Comparative Media Studies program (http://cms.mit.edu/) in support of Professor V. A. Shiva Ayyadurai (http://www.vashiva.com/).  An interesting aspect of this attempt to graphically depict the formation of ideas is that it has given me a new perspective on the process and the results that it produces.

The first part of the new perspective is that ideas are essentially packages of remnants from the knowledge, values, and experiences of the person (or people) that originated it.  In other words, an idea has a "DNA" that gives evidence from where it came from. Since the originator of the idea is also a product of their environment and culture, the idea’s roots can be traced back even further. From this perspective, we developed a series of graphical models that attempt to give a new viewpoint of what an idea is exactly.

You might read this and say "So what? You came up with an abstract way of visualizing ideas. Big deal", and perhaps you're right. However, what these graphical models convey to me is that an idea is more than a novel connection of insights and existing ideas, as recent books on the topic have conveyed. While that concept is accurate, I believe that it makes the process sound more random than it really is. What are graphical models began to convey is that an idea is a set of connections resulting from a filtered version of the stored knowledge of its originator (person or people). Furthermore, the idea is instigated and tightly filtered by intention or circumstances that led to it. This is particularly interesting when you consider that the originator itself is essentially a filtered version of its environment. In other words, the development of ideas mimics a "Russian Doll" paradigm, which I believe has major implications on the development of new ideas

The Importance of Culture
If an idea's elements are a sub-set of its creator's knowledge, and that creator's knowledge is a sub-set of its environment, then by association, the quantity (and therefore, quality) of ideas is greatly influenced by the creator's ability to increase their knowledge from their environment. Of course, there also has to be a culture in place that promotes not only the increase of this knowledge but the experimentation of the ideas that it can produce. Beyond just isolated knowledge increase (e.g. reading a book), you can see the combinatorial benefits of a culture that promotes communication and sharing of knowledge. In other words, a culture of learning, communication, and experimentation is the fundamental to the creation of great ideas. Think of a brainstorm. It's essentially the repeated process of people creating random combinations based on elements from their collective knowledge bases. Of course, the importance of culture on creativity is common sense to anyone who has worked in an innovative environment, but it's nice to see the graphical models re-affirm it.

From Circumstances Comes Intent
As I mentioned, an idea is instigated and filtered by the intent and circumstances of its originator. Let me give a concrete example of what I mean. Say you step out of work to discover that it is pouring rain, you don't have an umbrella, you don't want to get wet, and you have to walk a mile to the subway. These circumstances immediately create an intent for you to minimize how wet you get from the rain. You react to your environment immediately based on your experience in the world and make the knee-jerk decision to look for something to block your head from the rain. You quickly think of all the items in your bag, ruling out some and considering others. You then reach into your bag, grab a newspaper and hold it over your head, shielding off some of the rain.

This was the formation of an idea. It's certainly not an original idea, but it follow the process by which ideas are created. Circumstances led to Intent, Intent led to a filtering of choices and exploration of possible combinations, and finally, a single idea was selected. An important takeaway from this example is that the idea did not just spring from thin air. The process to create an idea was quickly instigated and executed due to the tension or constraints presented by the environment.  Without it, the idea never would have been formed. This example of an idea is trivial of course, but many aren't, and they are formed (or not formed) and essentially the same manner.

Next time you have an idea, give a good hard look at it...I bet you'll recognize where it came from.

Come And Get It!

So it turns out that new experiences, not things, are the keys to personal happiness and satisfaction, according to a 2010 study out of Cornell University. I could do my own review of the study, but I wouldn't do any better than Stephen Messenger of Treehugger.com did last April. If you don't have time to read the short article (link below), the general idea is that experiences are less likely to be compared to the experiences of others, do not degrade over time, and provide one with a more internal sense of ownership than a product which is mearly posessed.

http://www.treehugger.com/files/2010/04/happiness-no-purchase-necessary-says-study.php

I'm left wondering what this means in regards to the development of new products and services. Particularly, I'm curious what this means in the digital domain, where the line between products and services is pretty thin. For instance, I consider Mint.com and its corresponding mobile app to be products, but its notifications and pro-active research algorithms (to help me find better loan rates, for instance) make it a pretty rich service. I guess my distinction is that a mobile or web app is a product because they have a form and take up digital space. In a sense, they are "digitally tangible".  Conversely, services are much more ephemeral and intangible by nature - plus, to be overly simplistic, they serve you. To take this product-service relationship one step further, perhaps a digital service is one that provides value outside of the digital domain? There are lots of mobile apps that do this, from navigational apps to shopping assistance and restaurant reviews, but perhaps we haven't gone far enough into the real world.

This got me throwing ideas around about how our digital products could have an even greater real-world existance. From that, I came up with the concept for "Come And Get It". The basic idea behind the service would be that a person could essentially "unlock" a deal at a real-world establishment and trigger their friends to reap the benefits. Those that arrive fastest will receive the best deal, but a minimum amount of people would have to show up for the deal to go through. Essentially, it would be like a location-triggered, time-based, contact-driven version of Groupon. However, since it must take place in the real world, it could trigger a whole range of unique shopping behaviors, such as friends rushing to join each other in the aisles of the same store, or customers behaving like sales people, trying to get their fellow shoppers to buy a product. Might sound annoying or hectic if you're an uninterested customer, but what if you're an owner of a struggling store?

Interested? Hate this idea? Please comment and we can discuss...

Visualizing Ideation - My Project Proposal

I just completed a quick write-up of what I intend to produce in my Systems Visualization course this semester. Since I'm obsessed with the Customer-to-Ideation-to-Design process, I figure a visualization of Ideation as a system was a worthy goal. Here's my write-up. Please provide feedback if you have any. 

The development of new ideas is traditionally viewed as an unpredictable and ambiguous process best left to creative people and brainstorming sessions. Yet, history has shown us that ideas do not as haphazardly as one may think. In fact, the most well-known ideas often emerge, adapt, and evolve in predictable patterns. More specifically, the majority of “new” ideas are not new at all, but simply existing concepts that are re-purposed, evolved, or merged to create some new instance of the existing idea. As these ideas come together, they collectively behave in a Darwinian manner, slowly evolving, branching, and discarding as necessary with each innovation. Ideas can be as grand or as simple as one would like, from Einstein’s Theory of Relativity down to a homeowner’s clever fix for a creaky floorboard.

 In the course of this semester, I am going to attempt to visually represent the ideation process as a functioning, evolving system of interconnected components. The two primary components of the system are the internal mind of the idea developer and the surrounding external environment. At any given time, each component contains its own intentions, capabilities, knowledge, and problems in need of solutions. When the two components interact, an environment for ideation is developed. To pull a concept from the discipline of system design, ideas are the emergent properties of the internal and external components. I will put particular focus on visualizing the “patterns of ideation”, as I believe that these provide a critical foundation for an improved pro-active ideation framework. A major challenge in this effect will be in the representation of the full spectrum of ideation sources, from accidental discovery of the “adjacent possible” through the pro-active adaption and evolution of existing concepts.

 While I develop this visual model, I will be coordinating with the work of classmate Matt Harper, who is exploring the management and ultimate realization of ideas. One of the outcomes that I hope to achieve from this work is a visual representation of an optimal model for repeatedly developing innovative, high-quality ideas. After all, any organization interested in innovation should model their process after the way ideas emerge, connect, and evolve in nature. The concept of “ideas as connections” is interesting as it suggests value in an organized, patterns-driven, connection-based ideation system over unstructured ideation. Some may suggest that an attempt to structure and constrain the ideation process would only hinder the creative mind. However, it is this tension itself brought forth by constraints that often brings about the most innovative solutions. I will attempt to visualize how this tension emerges across the components of the ideation system.   

Finally, I will attempt to validate and iterate the ideation visualization by mapping some of the most influential ideas in history against my model. The source I will use for this will be Time Magazine’s recent publication: “100 Ideas That Changed the World”. I would like to visually demonstrate this full range of influential ideas within my visualization in hopes that it will reveal insights about the ideation system. In this project, I will leverage insights on ideation developed over the past decade as a professional designer entrenched in the creative process. I will also draw from the readings of leading ideation and innovation experts, including Steven Johnson, W. Brian Arthur, Tim Brown, and many others. 

Concept design from February 9, 2011

An Illustration of Ideation

Where do good ideas come from? I've attempted to visualize it. This isn't perfect, but neither is the process itself! 

Idea: Charitable Gambling & The Unexpected Merger of User Insights

Super Bowl week has arrived and my mind's on football. Despite the fact that my hometown Patriots failed to get there, I'm actually looking forward to this game. The NFL's popularity has skyrocketed in the past decade (Sunday's event is expected to be the largest viewing audience in sports television history), and this is due no small part to the games fans play around the games. I'm talking, of course, about Fantasy Football and sports gambling.

The rise of the internet and expansion of TV coverage has enabled even the casual spectator to get all the information they need on every player in every game. This has been a key to the meteoric rise of Fantasy Football. If you're not familiar, Fantasy Football allows football fans to create and compete with their own notional lineup made up of real players. Performance by a Fantasy Owner's players in a game (Touchdowns, Yards, etc.) lead to points for that person's team. The impact on real-life football? The Fantasy Sports Trade Association reports that "55 percent of fantasy sports players report watching more sports on television since they started playing fantasy sports".

Gambling, meanwhile, has been steadily gaining popularity for quite some time as well. Online gambling sites have enabled the average football fan place bets online in a far more easier and safer way than once possible. Of course, it is illegal, but convenience has a way of encouraging these kinds of activities. Finally, it's worth that there has been a recent emergence other competitions for the "active spectator", including "survivor pools" (attempt to pick winning team every week until you lose) and "pick'em leagues" (pick a series of games every week against competition). These competitions take on the mechanics of gambling (e.g. picking games against the point spread), but they tend to include more of the average fan.

So why all the gambling and game-playing? Well for some, gambling is about the pursuit of money at best, and at worst, it's an addiction that goes much deeper. However, these hardcore gamblers are not the reason for the recent rise in NFL popularity. The rise is more from the casual fan who has picked up all these various competitions as a way of making the actual product that much more exciting. Sure, it's still often about the thrill of risking a few bucks to win a little spending money, but if it's solely about the money, why not just play cards or scratch tickets? The reason is that it's NOT just about the money. People play these games because it makes the actual football action that much more exciting for many people. A random Week 6 Cardinals-Browns game is meaningless and boring unless you're counting on Peyton Hillis to score two touchdowns to win your fantasy football game...then it becomes exciting. (Author's Note: I actually stopped playing fantasy football a few years ago, so don't think I'm selling it here.. I'm getting to my point in the next paragraph)

Here's my point... if you ignore the seedy and illegal aspects of sports gambling, and just focus on the human behavior associated with it, you come away with the conclusion that many people in this country like to play games in order to make actual sports more entertaining. It's a way of taking some ownership and getting more immersed in the action. It's active participation more than passive observation. Now consider the fact that people are willing to lose (ie. SPEND) money in order to experience this higher level of entertainment and you really have an interesting insight.

So the point of this TReil & Error blog is about idea development, not sports gambling, so let's get back to that. This original insight about sports gambling/contests came up a while back during a conversation that a friend and I had while watching the NCAA basketball March Madness tournament. The question we asked ourselves is whether there are ways to leverage this insight in a way that is more positive than illegal sports gambling. What we came to is a completely different domain where people also are willing to give up their money: charity.

While this pairing between charity and gambling is somewhat counter-intuitive (one is risky and self-serving while the other is completely altruistic), there are some commonalities. I have the statistics on it, but most gamblers lose in the end or break even at best. Personally, I always hear of the person who goes to the casino, loses $100 and says "oh well, it's worth the price for a fun night". In other words, losing money is fine if it's equal or less than the cost of a fun time.

Meanwhile, charity has it's own associated behaviors. Some giving is done in traditional or direct ways, but other giving is done through charity events, such as galas or casino nights. On these occassions people are willing to throw their money around more willingly because "it's all in a good cause". In other words, their spending habits are completely uninhibited (and rightly so) due to the fact that the money is all going to charity. Worth noting that Americans give an average of 2% of their post-deduction salary to charities, more than any country in the world.. that's quite a bit of money.

So why not align these two things together? What if we created an online sports competition tied to charitable giving? I realize completely this is a legal mindfield in both domains (and potentially a non-starter), but I think the product itself is worth considering. Users could compete in pay-for-play online competitions, such as Football Pick'Em leagues and Fantasy Sports. Everyone who plays would have to register with their charity of choice, and they would have to pay a small monthly fee, or a per-contest fee, such as $10-$20. The losing players in the leagues or contests would "lose" their money to the charities of the winners. In other words, everyone would be competing to win money for their preferred charities...money that would be more than they could be individually. I would imagine that charitable organizations would be willing to give up prizes and benefits for the ability to get the huge exposure of the sports-watching demographic.

I could write much more about this idea, but I just wanted to focus on the core concept for this entry. If you would like to collaborate on this project, such as determining how this can be completely fair and legal, please let me know. I hope this has proven a good demonstration of two common behavioral insights combined in an unexpected way. Finally, check out the concept design for "Gamble for Good" that I created a while back. Enjoy the game on Sunday!