We used paper mockups of all three interfaces as the basis for our usability tests. Since we did not have a program running to generate variations on-the-fly, we created a theme and fifteen variations beforehand. These musical fragments were recorded and digitized and used as the basis for all three interface tests. For the Rating and Choosing interfaces, four random sets of variations were chosen, and appropriate "screens" were created for each set. For the Music Processor interface, the music for the theme and variations were written out so that they could be added quickly to the blank page as necessary.
The interfaces were presented in order from lowest user-control (Rating) to highest (Music Processor). We chose this ordering so that the user would not see "behind the scenes" when using the Rating interface, but would be quite familiar with everything by the time s/he was using the Music Processor.
With Subjects 1 and 3, we performed the tests in the graduate computing lab on the first floor of Sieg Hall. Mike ran the "screen" portion of the computer, and Kevin ran the "speaker" portion, playing back the appropriate digitized variation when requested by the user. Since these two users had little or no ability to read standard music notation, we demonstrated the Music Processor interface for them briefly but did not have them actually attempt to use it.
The tests with Subject 2 (the traditional composer) were performed in Mike's apartment. Consequently we had access to a music synthesizer and sequencing software, which gave us some additional flexibility in generating variations for the Music Processor interface test. It also made it possible for the subject to try out ideas before committing them to paper.
Each subject's session with each interface was generally run uninterrupted. After three or four iterations of the process, we stopped and asked a few interface-specific questions. At the end of the entire session, we asked some broader questions and had a general discussion. Although Subjects 1 and 3 did not use the Music Processor, they each saw a demonstration and commented on it during the discussion.
For each subject, we made an audio recording of the entire session (except that with Subject 2 the recording did not come out). We also tried to take notes during the sessions, though these were scanty as both of us were necessary for running the mockup. As soon as possible after each session, we also discussed our observations informally to solidify them in our minds.
We ended up doing all of our interface testing with low-tech mockups. Despite their limited functionality, these turned out to be extremely useful for checking the usefulness of a range of design ideas. They did not take very long to create, and we did not have to worry about the "trivial details" needed to make the interfaces function correctly. In particular, taking the paper mockup approach let us test the Music Processor interface, which simply would not have been feasible to build within the scope of this course. Had we attempted "real" computer mockups, we probably would not have had time to finish more than one.
The paper mockup approach also led to unexpected discovery of new design ideas. Earlier we mentioned that the music for each variation was written out ahead of time to facilitate the test of the Music Processing interface. During the test of this interface with Subject 2, we had these pieces of paper lying on the floor and within sight of the user. When deciding what operation to do next, she ignored the menus that she was supposed to be using and started looking through the handwritten variations. Her music-reading skills were advanced enough that this approach to finding a good variation was much more useful to her than reading a list of function names. This discovery was a direct result of the quirks in our paper mockup. (Perhaps similar accidental discoveries could be made with a buggy computer mockup as well.)
The low-tech mockups had disadvantages as well. Goldberg is supposed to let you explore a large number of variations quickly by repeatedly generating variations on previous variations. With our paper version, we created several variations ahead of time, but it was impossible to predict which variations the users would want to explore further. Consequently the users' choices and ratings never really affected the next batch of variations, resulting in less convincing performance.
