Work, writing

Voice Interaction

As phones have transitioned to smart phones, our personal technology has graduated from conduits between people to a more sophisticated breed that allows for – even invites – direct control. In tandem, people are getting rid of voicemail, making fewer phone calls, and texting more. In one vein, this seems like a more truncated, efficient behavior, but it also implies greater intimacy with the device.

We’re also growing to expect the similar level of control we have over our phones to expand to the devices of our environment. The “smart home” and “connected” objects are commanded with our phones for the time being. Contrary to the shift in phone use, control of these devices that is buried in a growing library of apps is not efficient.

The technological response to surfacing quick control over these smart objects is the use of voice interfaces. The Xbox’s Kinect allows for voice control of your Xbox apps and access to media. The Xfinity remote control makes “change the channel to HBO” possible. Apple’s Siri, Microsoft’s Cortana, Amazon’s Alexa, and the Google Now services are all serious attempts at broadening voice control to access many services.

While speech-to-text recognition has largely improved, the voice controlled services themselves still lack in the sophistication that people presume exists when communicating through a nuanced medium as speech. Even if this level of sophistication is attained, and the services understand and respond exactly as we expect them to, the challenge of intimacy remains.

When common interaction with phones shifted from calls to text as the interfaces allowed more direct (read: intimate) control, we’ve created this controversial-yet-accepted balance of interacting with people directly and multitasking with our pocket computers. Voice interaction necessitates a more public display of that human computer interaction. One that is so uncomfortable, directly inhibits its use. Think of the times you have used your voice input on a phone: public settings, private settings with people around, or solitary settings?

Although we may not be able to out-design social mores, we can take the first challenge—that of accuracy, intuitive use, and predictable outcome—to the whiteboard and to the APIs.

Design Team:

Kristen Kersh • Niamh Parsely • Rob Brogan

Star Trek Holodeck
Work, writing

Design Considerations for Virtual Reality

Yes, much hype. Much much hype.
Yes, I’m always skeptical, and I’m assuming that VR headsets (e.g. Oculus) will take a few iterations, and price points to catch on. Now even a few years into it, wearables are still getting mediocre traction. At best, Apple has people wearing them for social status or fashion. Nevertheless, new technology is deserving of design consideration even more than existing, common devices. They need to be nurtured, and “done right” in order to have a longer life ahead.

What follows are a few things I would keep in mind if I found myself in a position to design for Virtual Reality. Perhaps with more exposure to VR, I can add to this list in the future.

Sharing the room
Others that are not wearing the headset have no insight to the VR experience; unlike a TV, which can be a shared experience. Devices will either have to become more affordable so that everyone can wear them at the same time, or the solitary device should provide some external feedback to others in the room; such as an outward-facing display that mirrors a 2D version of the virtual experience, distinct audio signals (for the room, not the wearer), or as some currently offer: an optional feed that displays on a TV/monitor.

Accessories to support and enhance
Accessories can enhance the experience, further immersing you into the virtual reality by giving you a great approximation of bodily control. These can range from the more necessary, to nice additions.

The ability to turn in place with ease (and not falling into real world objects) is probably the most important and can be solved with a basic swivel chair or the more expensive 360° treadmills.

Oculus accessories

In concert with existing wrist wearables, or custom-made wristbands, the VR headset would no longer need to be the main point of interaction (click, tap, or toggle). Using accelerometers and Bluetooth that are already included in any fitness wearable, one could wave an arm in front of them and have the action mimicked in VR. Or similarly, a shake or a tap on the wrist could replace the need to tap a button on the headset for making selections.

Keep things out of frame (move the eye)
The same principle that applies to photography, painting, or any kind of visual medium: you want the eye to move across the canvas. In this case, you want heads to turn. Succeeding at this influence are short films that have a rich and beautiful environment, but also play between primary and secondary subjects. At times, both are not within the same gaze and you must turn to see either subject.

This should be used in moderation however, as you can easily tire a VR participant with too many subjects in different directions, and also risk a poor experience that leaves observers feeling they might have missed out on parts of the story because they were forced to follow one subject while another of equal importance remains out of view.

Sound quality is as important as image quality
A truly immersive experience relies on tricking your senses. A well-crafted story also relies on directed attention. Audio quality aides both of these by bringing the observer into the virtual world with realistic ambient sound, and the ability to subtly distinguish voice will help people grasp if there’s a character standing next to them that they need to turn and face, or if the speech is coming from an omnipresent narrator.

Prompt to enable Do Not Disturb when starting the VR
This is a short one, but nothing ruins a virtual experience like a pesky notification pushing its way into view. Before starting a VR experience, there should be some reminder or prompt to enable Do Not Disturb mode for the phone. More aggressively, VR software could just disable notifications, but I prefer to let users make the choice.

Subtitles should remain fixed, detached from video movement
Another specific point is that layered content, like subtitles, should be fixed to an easily legible portion of the screen. In one demo, they were out of view, below the general plane of vision. Although moving around and exploring the setting is a hallmark of VR, some visual elements should be fixed or represented “out” of the virtual space – another plane, or layer, if you will.


Most interesting (recent) read

I haven’t posted here much, and that’s basically because I’m lazy. I’m still here, though!

Quickly now, I’d like to share an article that – in my opinion – has a lot of meat, and all of it is interesting, if you’re a designer.

Chinese Mobile App UI Trends

By Dan Grover

Some highlights include:
Chinese culture doesn’t make a big deal of meeting strangers nearby through social apps.
CAPTCHA utilized on login screens (not just signup flows).
People really do use QR codes!
Moments – Just scroll to this part. I really dig the philosophy.

Reblog, Wireframes

dialog box wisdom
Out of context, some of these one-liners you’ve probably seen in a dialog box have a deeper meaning. The last one is an honorable mention; a classic, non-tech artifact.

Everything not saved will be lost.
Nintendo “Quit Screen” message

Accept Change
Microsoft Word

Wait some time and then reconnect
Canon webcam

Remove all attachments
Microsoft Outlook

Make sure your own mask is secure before assisting others
Airplane safety card

I’m sure you can google for plenty more, but these were my favorites.

Dialog Box Wisdom


Form field patterns for email address

Look, a post about UX! This will be quick.

I’m surprised that I haven’t seen many design patterns on sign-in fields and how to make them as fast as possible. Yes, I’ve seen a few design examples, but no patterns that suggest innovation.

Lo and behold, hidden in the rough, the real estate app Trulia has the best email field I’ve ever seen! No comment on the rest of the app.

The second best has been the Belly iPad app (in-store, not consumer facing). They include an extra row below the iPad keyboard with buttons for common e-mail domains:

belly ipad keyboard

When I first saw that I thought it was incredibly clever. Ever since I’ve noticed the common domain suffixes on the iOS keyboard while holding down the [.com] key for certain inputs, I’ve been expecting Apple to integrate a similar email domain shortcut with the [@] key.

My takeaway

Why not use this? I can’t think of a down-side.

Since they’re using form field input masks (in a clever, auto-complete kind of way) it’s not destructive or inhibiting user input. If your domain is not appearing in the masked (grey) suggestion, then simply keep typing the full address.

If your email is:
Then you can stop at: Name@g

If your email is:
Then you’ll keep typing… (standard everywhere else).

UX is about making life easier for people, not necessarily solving problems. Yes, you make life easier by solving problems, but one solution will rarely fit all cases. Aim for the majority, and if you can design in such a way that the minority aren’t negatively impacted by the design, you get major bonus points.

+2 points to you, Trulia. You go Trulia!


Auto-punctuation and syntactic processing

Once in a while, when I might want to construct a clever piece of SMS text-art, I can be annoyed by the automatic placement of the period (full stop) after entering space two times. We must design for the most common use cases however, and on the whole, I do like this auto-fill logic. The little things are often what delight users the most.

Being too liberal with a well-intended shortcut or assumption can be a UX calamity, but I believe that with some additional linguistic information (and a fair amount of user testing), the double-tap auto-punctuation could be extended to questions with a fair amount of success.

Present functionality

Let’s keep in mind that at present, this feature applies a period any time the space button is hit twice in a row. It could interrupt a sentence, or easily mislabel a question or exclamation. It’s currently up to the user to add alternative punctuation before continuing, or to erase the auto-punctuation and correct it.

Proposed functionality

Imagine double-tapping the space bar and iOS would predictively produce proper punctuation — I couldn’t resist the alliteration. This would be informed with linguist programing about content (question words) and context (companion words that differentiate statement from question).

Do you know how to do this? versus You know how to do this.
Reason: Presence of auxiliary “do” in Do*know. Note that the statesman also has the word “do,” but only as an infinitive (to*).

English can be tricky with its question semantics, and this solution would require localized code for different input languages, unlike the global period insertion.

Limitations, Advantages

For English usage, the limitations are easily apparent. It may require too much precious processing to scan an entire string of text for this kind of semantic context. For some languages it might be pointless if an opening punctuation is used – such as the Spanish ¿?. It would be interesting to survey Spanish users to see how prevalent this convention is during SMS or other mobile communication. My hunch is that like many English vowels, these opening question marks are usually dropped for brevity.

Some languages lend themselves to very “easy” semantic processing; so much so that one might think “why aren’t we doing this automatically?” To reference Spanish again (the only other language I feel qualified to write about), circumstance words are supposed to be spelled differently in the context of a question.

What/how is (she/he/it) like?
¿Cómo es?
What/how (she/he/it) is like.
Como es.
Interrogative circumstance words (what, where, etc.) have an accent, but do not in statements.

This is just a potential shortcut, and there are many question sentences without one of the “five Ws,” but it’s an example of logic that can circumvent contextual rules. Some languages even have question words (Japanese and Mandarin come to mind), but they also tend to not use question marks as far as I know.

This is just a few minutes of brainstorming, but if done properly, it could pretty useful, couldn’t it.

What do you think.
See what I did there.