Running a “Prototalking” Session

In our last VUI post, we talked about scriptwriting for our Alexa Skill called ‘Standup,’ designed to help a team stay on track during daily check-ins. Now we’d like to get into what we call “Prototalking.” 

Prototalking is a way to test voice UI applications that substitutes a human partner for the VUI system. We worked with Carbon Five who let us dial in to one of their own standup meetings both to observe and to roleplay as Alexa. We’ve worked with scripted responses to user input in the past, but the Alexa system has some particularities we were looking out for. We kept our eyes on the ways users perceived affordances and recovered from verbal mistakes.

Here are our biggest takeaways:

1. Real-world testing is best

Testing a design usually means observing one participant at a time as they interact with a prototype. We shape this experience with prompts and follow-up questions to stay on target and to emphasize the aspects of the design that we’re interested in. This is a strong, evaluative approach for testing self-contained Alexa Skills, say a whiskey tasting Skill or a trivia game Skill. 

For a Skill like Standup that relies more on contextual drivers, it felt smarter to test the Skill out in the wild, in an actual standup meeting. Even though we still could have learned a lot from having a scripted meeting, we wouldn’t have been able to measure how well the interactions worked in real life.

2. Don’t over-educate

Don’t cheat by telling your participants every detail about how to use your Skill—you’re only cheating yourself! To get a good feel for how users are going to react to your skill, provide your participants just the right amount of guidance and no more. We wanted our test to simulate interaction with an intermediate Standup user who already knew a few tricks, so we provided the meeting leader with a few simple commands that a moderately-experience user might already know, and we indicated when they might be useful.

The same goes for all participants. In our first test, we gave our team lead a strong overview because he would be the one talking to Alexa, but the other team members got no briefing at all! For later tests, we created a simple explanation to let other participants know what they were helping us do and how to join in. We found that this helped set expectations, reduce awkwardness, and ease the burden on the team lead

3. Get into character

Treat your testing like an audition, and know ahead of time how you want to play the role of Alexa. During generative testing, you might get input that you hadn’t scripted for. It can be helpful here to improvise responses and functionality that you might incorporate later. During evaluative testing, however, it’s important to play Alexa true-to-character to accurately simulate interactions. Don’t go off-script! Here are a few tips to help your acting: 

  • Do an Alexa impression! Speak at her pace, which may be slower than you’re used to. Pause when you know Alexa would be working. Knowing the time it takes to get through tasks is critical to your Skill’s success, so be realistic.
  • Don’t go easy on your participants. Only accept commands that are close enough to the language and format that Alexa would accept. When one of our participants was confused by Alexa’s response and said “Alexa…help!” we replied as Alexa would have for the command: “Alexa, tell standup that I need help” instead of breaking character. Accurately modeling Alexa’s behavior, including communication breakdowns, reveals the user experience.
  • Alexa works hard. To simulate her complex functionality, you might need two people to play the part: one for voice interactions and the other for non-vocalized processing. Splitting up these tasks can help keep the experience seamless for your participants. We had one person play the Alexa voice while the other posted messages to Slack and ran a timer.

Prototalking is similar to traditional UI research with just a few key changes. You can ensure success for your VUI testing by getting clear on what you want to learn and adjusting your research tools and methods appropriately.

Are you designing a voice UI? 

We’d love to hear about your experiences! Join our voice UI channel on the Cooper Friends Slack, send us your favorite articles on Twitter or send a note to [email protected] to chat about your voice UI strategy.

Jiwon Paik

Learn more

Subscribe to our mailing list and stay up to date on our latest thought leadership and learning opportunities.

Connect with us

Want to know how we can help your company drive real business progress?

Let’s talk

Stay up to date on our latest thought leadership and learning opportunities