COM 597: December 2009

Term paper 597

Inge Scheve

Zack Midles

CASE STUDY: WEBSCORER (iApp)

For our term project, we did a case study on a brand-new Apple application, the Webscorer iApp race timing system, which will be released later this year. The Webscorer iApp will be available through the iTunes store, and can be used on both the iPod Touch and iPhone.

Our objective was to determine how user-friendly this application is, by designing, conducting and interpreting a usability study on the product. Based on the study, we wanted to share any problems and concerns that the participants in our study identified with Vesa Suomalainen, the main creator of the application and the CEO of www.webscorer.com, and the other developers of the Webscorer iApp team. We also wanted to recommend any improvements we could see from working with the application through the project. Webscorer is developed and designed right here in Seattle, and it was very interesting and motivating to be a part of a real-world product development process. Being able to work closely with the developers throughout the project was very useful and gave us insight into what they were interested in and the things they considered at the various stages of the process.

Market situation:

With the availability of new iPhone and smartphone applications literally exploding, we believe that Webscorer has a good chance of success. Webscorer is tapping into a whole new segment of applications, which is keeping track of statistics, whether it’s race times, details and progress of a workout program, or other similar applications – there are lots of them out there, but none are really designed to deal with the complexity of a formal, multiple participant race or set up to publish results to the web.

While traditional race timing systems that require extensive training, expensive equipment and take a long time to sort/verify and upload results to the web, the Webscorer application represents a unique niche in the race community by providing an event timing system that is easy to use, inexpensive and works with gadgets a lot of people already own and know how to use.

“We are basically trying to offer some much-better-than stopwatch iPhone application and something close to what you would get with a chip-timed race – minus the expense,” says Brad Kirkpatrick, one of the Webscorer developers, of the company’s vision of their application.

While the idea of a simple race timing system that uses the iPhone technology is a fairly clean-cut and easy to define. the Webscorer developers recognize that there are tricky situations in race timing, and that providing a solution for all scenarios is difficult. However, shy of computer chip timing solutions such as those used on the FIS World Cup and in the Olympics, the Webscorer application represents a huge improvement over current lower-cost race timing options.

We chose this project is because we are both athletes. It means a lot to us to have easy access to results and being able to track our performance. We have both been to several competitions where it takes multiple days before results are posted. Our initial impression is that the Webscorer iApp process is quick and efficient. The iPhone devices are easily taken to all parts of the race course, and even in areas without coverage. The results are stored and can be transmitted between devices within the devices’ walkie-talkie range. Thus, the Webscorer iApp eliminates manual race timing with a stopwatch (except potentially as a worst case scenario backup). The results can instantaneously be posted online without having to edit.

Webscorer has the capability to not only accurately time a race with a variety of entrants, but also posts the results to the Internet for quick viewing access. Furthermore, users will be able to create their own profiles on the Webscorer web site, where they can track their results and progress, share results and status with other users and connect with various social networks.

Target audience:

The target audience for this product is anyone with access to an iPhone or an iPod Touch and a need to time something. However, some major groups include coaches of all levels from club and community to NCAA Division I staff, to national team coaches, physical education teachers, athletes and race organizers. Some are professionals, such as coaches, PE teachers and athletes, and have experience with a variety of timing equipment and methods, while others have limited experience. Often, timing crews at smaller and local events consist of volunteers whose only connection to the event is agreeing to time it. However, the iPhone and iPod Touch devices are used by people from all walks of life, making the Webscorer application easier and more intuitive to use, given that they already know how to work the device. Accordingly, we selected test users from a broad range of backgrounds in order to give us the most extensive data on the usability.

Methods:

We worked closely with Mr. Suomalainen to run a usability study on this product. This pertains directly to our class by the fact that we are studying usability in interaction design. While we are not designing the product, we are studying a newly designed product and observing how the usability tests and feedback will be interpreted before the final product is released. One step of the process, was examining their earlier wire frames and design options. This step helped us better understand the design process and the interface issues the team considered when designing the user interface. There were a variety of color schemes and icons considered along the way (see attached slide). We believe, as the test users also commented, that they came up with a clean, intuitive design for the race timing screen.

We considered a several methods to test the user experience, and decided to go with a formative usability study. We recruited five individuals to use the application over a given period of time to see how they satisfied they are with it. These five included a University of Washington track coach, friends, and professional athletes. On page 118 in Measuring the User Experience, the authors argue that, “five participants is enough,” since the majority of usability issues will be observed with the first five participants (Tullis, 2008). We felt that if there are any problems with the product, they would be addressed with our five applicants. We also recruited two random students to run a mini-version of the test during our presentation in class. The issues these two ran into confirmed what the full-version test users had indicated.

To use the application, the test users needed a list of entrants. When these entrants finished the event or competition, the test user checked them off. The time they get when they pass the finish line goes next to their name and it places them automatically.

We asked the test users to use the Webscorer iApp and determine what they enjoy about it, as well as point out any aspect that are difficult to understand or not intuitive. At the end of the study, we provided our test users with a questionnaire where they explained in more detail what they did/did not like about it.

Results:

The study we conducted was intended to deliver feedback from test users on how easy and intuitive the Webscorer race timing application is, and what kind of problems the test users ran into when using it, as well as which features they liked and disliked with the application. To collect this information, we asked the test users to perform the following tasks:

1. Preload a racer list into the application, and time three different kinds of races:

a. Individual (interval) start.

b. Mass start.

c. Mass start where names are taken at the finish.

2. Post results to the Webscorer web site (they were given access to the developer’s test site, as the product is still not launched commercially).

3. Include race notes or photos (given that the iPhone and the iPod Touch also have cameras and note pad functions).

4. Email the result link to a couple of people.

5. Go to www.webscorer.com to locate the results.

6. Go to the application on the device, locate the results from a completed race, edit race times and repost the results to the Webscorer web site.

Our study shows that 100 percent of the participants were able to preload the names into the iApp. All of the participants were also successful in timing a race. So, in terms of completing the most important tasks, the Webscorer iApp works well, also for individuals with limited exposure to timing applications.

When getting into the more advanced function of the application, some of the test users ran into problems. Half of the users were able to email results from the device. Only half of the test users attempted to post results to the Webscorer web site, and of those, only half of them were able to complete the task. Similarly, half of the users attempted to edit results, and of those, only 50 percent were successful. All of the users tried to locate results on the web site, and 75 percent were able to do so. Finally, half of the test users tried to edit results on the device and repost them to the web site. Of those who tried, half of them managed to edit and repost.

We also asked test users to provide comments and feedback on their user experience. Some of the comments are included below.

“On the first screen, you have to click the arrow to start a race. It would be nice to just click the words to move on, as the arrow is a small button to touch.”

“On the entrant list keypad I would like to see it changed such that instead of hitting return and then the plus button to add more than one entrant, the return button is a ‘add an entrant’ button or something.”

“I like it. You have to be prepared to use it though, tough to use it ‘on the fly.’ I understand that is probably not the vision of the app. But again, it’s a good tool.”

“Some of the phrasing is a little confusing. Maybe use "participants" in the place of "name list" and "mass" race is confusing to me (I still don't know what it means).”

Overall, the test users reported Webscorer to be a fast, efficient and quite intuitive race timing application. They also reported that they think the application fills a niche in the market. Where most race timing systems are expensive, hard to operate, require extensive manuals and specialized equipment, the test users appreciated the simplicity of the Webscorer iApp concept. Users can download the application to a device that is widely used for a variety of tasks (the iPhone and the iPod Touch). They already know how their devices work, so there is not complicated equipment to learn.

Once the iApp is installed and running, the test users generally found it easy to load participants and create a start list, and they said it was easy to start a race. This was confirmed with the mini-race conducted during the presentation in class, where we had two volunteers from the class time two classmates racing up and down the hallway in Savery Hall. But just like the class volunteers, some of the test users reported that it was more complicated to use at the finish of a race, especially short events and when many racers finish together. Sometimes the test users confused tapping the racer name button with the button that lets you complete the race (“finish race,” which concludes all participation). The feedback is that they would like a more clear distinction between those two functions in the layout on the screen.

One common complaint with the Webscorer app is that it is hard to time events when many racers finishing in a short amount of time. We suggest that this could be improved by using bib numbers rather than names on the participant list. Also, timers might not know the names of the participants and recognize which entrant to tap. The Webscorer team is now working on solutions to correct these situations.

“We have a fast-tap feature that allows you to quickly finish a group of racers. This will work in some situations. We are also developing a two-iPhone solution which will allow one timer with the first iPhone to finish racers as they cross the finish line. The second timer with the other iPhone would enter bib numbers of finished racers. These two lists would then be consolidated either locally or on the web site. This is perfect for chute-type finishes,” Kirkpatrick says. “There are going to be situations where it’s just impossible to be 100 percent accurate. For those cases we have features that allow for adding missed racers and for correcting racer times.”

Other potential issues this usability study didn’t touch on include the design of the Webscorer web site, where users can create their profiles, keep track of their events and finish times and exchange results with other users, among other features. Also, there have been reports of difficulty using the tap function with cold/wet fingers – a frequent scenario when timing outdoor events). Webscorer has already addressed some of the touch screens problems, such as the inability to push on-screen buttons with cold and/or wet fingers, since cold/wet, fingers don’t trigger the touch screen. One solution, Suomalainen says, is using a special stylus available from the Apple store to click the touch screen buttons. Finally, iPhone ad iPod Touch screens appear to work very sluggishly in cold temperatures, which is also a common scenario when timing outdoor events. And along the same lines, we also didn’t look at battery capacity of the iPhone and the iPod Touch. Cold temperatures are known to drain the batteries of cell phone and small electronics faster than more moderate climates.

Conclusion:

If correctly designed, a usability study involving five test subjects should be sufficient to discover major problems with the application (Tullis, 2008). According to the theories of measuring the user experience, this kind of sample should reveal about 80 percent of the problems.

However, after analyzing and evaluating the usability study we created for Webscorer, we recognize some fundamental flaws in our study’s design. The questions we asked and the tasks we asked the users to complete were of a summary kind that simply tested whether the application works – which it does – rather than how the process of using the application was perceived and what the experience was for the user.

We tried to capture the usability and user experience aspects of the application by asking the test users to evaluate the application and the experience at the end of the survey, and to provide any input on what could be done to improve the product. However, as discussed in class with the split-brain theory explaining that people will come up with rational-sounding arguments for making bad usability seem better because they like the product, the strategy we chose can backfire. It is possible that the test users have emotional reasons for liking or disliking the application, and that the user could produce rational arguments that the application has or does not have good usability based on their emotional experience with the application or the device. Given this source of error, having the test users compose an argument is not sufficient when testing the usability of the application. It can, however, be a nice and useful addition to a well-designed usability study.

Rather than asking test user whether they were successful in timing a race with yes or no as the possible outcomes, we should have given them a statement to evaluate on a Lickert scale. For example, we could have asked them to evaluate the statement “It was easy to start a race” on a scale from “strongly agree” to “strongly disagree.”

By using the Lickert scale we would have been better equipped to help Webscorer analyze how user-friendly a part of the application is (Tullis, 2008). If, for example, all the study participants “strongly disagreed” to the statement “it is easy to start a race,” Webscorer would know that this part of the application is an area they need to create more user-friendly.

We realize that the design of our usability study was conceptually flawed in terms of testing the user experience of the application. While we failed to thoroughly test the usability of the Webscorer iApp, we did collect a considerable amount of useful information about the product. The project also taught us a lot about the process of conducting a usability study and how to formulate the questions so that they deliver useful information about the usability of the application, not simply if it worked.

Furthermore, based on the comments that our users contributed on the survey, Webscorer has started refining the application. The Webscorer developers have already started implementing some of the changes that the test users suggested in order to make the application more intuitive and user-friendly.