Launching Augmented Reality
"Augmented reality is the combining, adding or enhancement of the real world by the virtual."
During this project, I will explore the concept of Augmented Reality and experiment with physicality to create an interactive work. I aim to explore how works of art and design can fit within the spectrum of Augmented Reality and how this technology could inspire new interpretations, experiences and narratives.
Research
ORBITA by PlayTronica
Orbita is a device for creating electronic music, away from screens and complex DAW's.
Aesthetically it resembles a record player but works as a sequencer, allowing you to record and play electronic tracks using singular notes. It uses colour magnets as those notes and lets you construct up to four rhythmic and melodic loops at the same time.
Choose your sounds from the online synth or use your own recordings from music software. This device has been used for performance, education, and play.
Spatial Sound by Will Copps
“I really want people to be able to experience spatial sound as I intended it. Like it’s actually in this immersive space around you and the only way to interact with it is to walk through the space and explore that kind of augmented reality interaction of things.” -Copps
This system is a spatially mapped AR sound generator. Once the person holding the phone enters a target zone the sound will play, Copps has integrated this with 4D soundscaping and now each speaker has been panned to allow a fully immersive experience.
4D SOUND
Initially this is what Will Copps was influenced by and this technology served as a primary influence in his soundscaping prototypes. Using 360 degree pillar speakers with computer software that creates visual on screen 'sound holograms', this technology enhances how we experience sound through skillfully mapping the best placement for the speakers. By mapping sound waves and determining the best position for each pillar in the room the surround sound offers an extremely immersive noise experience for any sound installations.
"4DSOUND is a studio exploring spatial sound as a medium. Since 2007, 4DSOUND has developed integrated hardware and software systems that provide a fully omnidirectional sound environment. These environments enable vivid sonic experiences that blur the boundaries between the real and the imagined: the world of sound we know, and a world beyond." -4D SOUND LA
Chloe Lefort
Chloe Lefort, past student, created this example for augmented reality. Using pages in a book as image triggers she creates audio and visual on screen visuals. I liked this example particularly because of the physicality of the book and how the opening and closing of pages effected the programme.
HUVR
In this WIRED article the virtual reality contraption HUVR, created by University of California techs, was explained. This machine uses what the university describes as 'low cost virtual 3D technology' to allow users to interact with a digital 3D image and physically 'feel' it through VR headset handles which navigate through the object in the image. In this example they used an X-ray as the image and technicians could examine the spine and ribs.
Although this example is more virtual technology than augmented reality, I still find it interesting to see how this technology can be used in physical real world outcomes. It is making me consider more physical components for this project.
Sketchbook
Amy Winehouse Tour
Roy Orbison
Tour
Hologram Tours
I've been looking lately at hologram tours especially after ABBA announced their latest virtual concerts. I think it's great that we are able to still see artists in concert who maybe can't perform anymore because of age or injury, as well as singers who have died who can now continue on their touring legacy.
Using old video footage of live performances the concerts are lively and visually accurate.
I want to continue to explore this idea and make some of my own experiments with old footage.
Looking at all of my research and initial sketches I'm feeling very drawn to sound for this project. Music has always been a huge influence in my life. With nearly all of my family being musicians I learned how to read music from a young age and picked up my first guitar while toddling.
Jazz, blues and funk are some of my favourite genres since the style reminds me of my grandad. He greatly encouraged my trumpet playing and now all I really play on trumpet are the songs of artists we used to
listen to like Sinatra, Como or Armstrong.
We would only listen to these songs on vinyl since he siad that was the only way to listen to Jazz. I think is how I'm going to incorporate physicality into this outcome.
To bring records to life i'd like to use AR to play a song along with footage of a live performance using records as image triggers.
To construct this I'm thinking to build a plywood record player that will use small printed wooden records as the triggers.
I've added some drawings of what I imagine the record player to look like.
I realise now that colour coding will make no difference since Vuforia reads images purely by grey scale. Each record will have to be visually different though variables like text, placement and additional graphics.
This worked much better and I think the stars were my saving grace as apparently the more jagged the shape the better the camera reads it.
These were my first drafts of vinyls in Illustrator however when I uploaded these to the Vuforia image database it only gave them a 2 star rating. I decided to go back to the drawing board and add some more details.
This is the term used for visualising sound and although it is not exlusively linked to AR technology, I think there are ways in which this practice can incorporate aspects of AR. If certain sounds can only be seen through the screen or if patterns linked to sounds formed across objects.
Once I had figured out how to get a better star rating I started making record templates so I could just change the text dependant on what song I wanted to trigger. I individually uploaded these to the Vuforia image data base and every record gave me a 4 star rating.
Song Choices
Process
This was me starting to build a prototype record player and mini records out of recycled easter egg boxes.
The song choices for this project were important to me, they each remind me of painful long jazz rehearsals and agonising trumpet runs.
My trumpet music is piled high in my room and I thought the clef itself might be a cool aspect to add to the physical record. I was thinking that the sheet music along with text and decorative stars could be what differentiates between records.
I started drawing some of the notes out as simple circles and forming patterns in Illustrator.
This was my process of making a prototype. I used old Easter egg boxes and dog food boxes to construct the white base and cut small cardboard circles to layer and paint to resemble records.
After making this I realised I wanted the box itself to be higher and sturdier so wood would be my best option and doubling the height.
-> UNITY
After building the image data base in Vuforia I downloaded the file and tried to drag it into my assets in my Unity build. This part was probably the most frustrating error I had in this whole project as I couldn't for the life of me figure out where I was going wrong.
I found a solution online that involved opening the project on a Windows monitor and changing preferences. This however seemed a bit of a long way to come to a solution and I wondered if it was maybe the system version I was using.
This error continued to appear despite attempting to build 4 new scenes and re-downloading every package. I eventually tried a different version of UNITY with the not latest version of Vuforia and suddenly I could import my database.
I always find working in UNITY can become a bit overwhelming for me in terms of file management and where to find things. To combat this I wanted everything clearly labelled and in order so I knew exactly where to look.
I made a folder for my video files, my audio files and my image targets.
The plan was to have the footage play over the record in the shape of a circle however when the video would play it was wrapping around the 3D sphere as an object and not as a surface the way I wanted. I ended up changing from spheres to planes for an easier life.
For the songs I used I refused to pay for the files so instead I set up an external mic and recorded them from a speaker. At first they sounded exactly like I had been hovering over a speaker and recording (which was not what I was going for) so I used Logic Pro X to edit things like the EQ, volume level and add filters to the voices. In the end after some editing each track sounded much clearer and I actually added a little static so it sounded more like a record player.
Each video I found through old footage across various websites but Youtube had quite a lot of old show videos. I downloaded these and edited aspect ratio to 4k and improved some of the colouring throughout the video. If I was to evolve this however your videos should always be run through a compressor for improved realtime running.
This little window in inspector basically saved my project. I couldn't figure out why my image target wouldn't disappear but it turned out it wasn't understanding when the target was found and when it was lost. To make it play when it comes in view I dragged the video game object and the audio into 'ON TARGET FOUND()' and selected the options video.play and audio.play.
To make these turn off when not in view I repeated the process but with 'ON TARGET LOST()' and made the video and audio stop.
While the targets would now play and stop, the image kept trying to appear when the record wasn't in view. To change this I went into the tracking preferences and made it the most limited tracking available so it wasn't trying to constantly find it when not in view.
The Physical
These were my final image targets. I had my 6 lasercut 6.7cm records and these image targets were cut on and attached.
These were the base components of the record box. All made from plywood I had the box base, wooden needle arm, records disk and some little buttons. To finish, I did some sanding and painted components in blue and white for the background of the disk.
Final Export
