My role with ScienceVR:
Concept design | Interaction design | Prototyping | Development

Museum Version

The National Taiwan Science Education Center commissioned a 5 minute VR experience to showcase Ada Lovelace, the first computer programmer. In her writings, Ada imagined that one day computers could be used to write poetry or music.

We imagine Ada’s dream world with such a computer and use music to introduce basic coding concepts such as:

• Inputs and outputs
• Loops and functions
• Compiling into a machine-readable language as punch cards

Some implementation decisions to make the virtual piano engaging and easy to play included:

• Creating enhanced sensations for playing piano keys through haptic feedback and visual cues like key animation and highlights
• Constraining the hand configuration near keys to point with the index finger in order to play one note at a time

Design decisions to convey the concept of separate programming functions included:

• Modular separation of function panels
• Representing physical interactions in a way that reflects the functions they achieve (e.g. loops as circular, transposing keys as shifting up and down, single-choice buttons for chords and accompaniment)
• Using symbolic representation of loops and functions that operate on the notes played, as too much text can be difficult to read with the screen resolution

Considerations in simplifying the notation included:

• Minimizing the use of representations that require musical knowledge
• Minimizing the number of different representations of notes (iteratively prototyped with colors, letters, numbers, and musical staff)
• Showing implicitly that new sheets appear with more notes, rather than a continuous scrolling

Home Version

In the next several iterations, we aimed to create an experience that could be played over and over again for a classroom or at-home setting. In this version, a step-by-step tutorial takes the player through each function, and players have more time to explore in depth.

We created 3 modes: tutorial, challenge, and free play. In the challenge mode, Ada provides pieces of music for the player to match using the piano input and functions. The player compiles their code to submit their answer, and if they are correct, they move on to the next challenge. In the free play mode, the player can create and combine as they choose.

In the home version, we created more explicit representations of the code and functions:

• Musical staff and letter-based note representations on the music sheet and keyboard.
• Explicit code and output representations
• Physical representation of code with pipes and dials above the piano during playback

We also included more explicit cues

• An indicator dial to count number of input notes played, showing a max of 8 notes
• A button to clear input and start over at any point
• Red/green lights to indicate lever values
• Red/green feedback for correctness on music sheet notes