Andrew led the development of an app named SPECTRA at the end of 2016 in his final project after running digital and interactive projects for Mirada Studios for over six years. SPECTRA was a groundbreaking app that used 3D data from a Google Tango-equipped smartphone to make art – unlocking a type of creativity previously limited to a small group of elite creative coders and making it as accessible as Instagram. This is a new kind of photography – one that uses shape and 3D position instead of RGB pixels – with results that are abstract, ethereal, and undeniably unique.
Andrew recognized the potential for an app like SPECTRA while directing the RealSense Dancer project – working with Intel’s hardware led to the realization that Google’s Tango initiative could put 3D depth sensors in the hands of consumers and unlock what had previously required a specific mix of creative and technical skills and access to specialized hardware. With the consumer release of Tango on the near horizon, Andrew and John Fragomeni approached Google in the spring of 2016 to propose the creation of a prototype as part of their program to finance launch content for Tango. Once completed, the prototype was approved by Google to move into production and was fully funded. Andrew led a small team of developers and designers to create the final app and to determine the style of visuals that would be included at launch.
There is a massive difference between creating visuals for a specific purpose and making generalized tools that enable others to create their own imagery. SPECTRA presented a unique challenge in that it required the development of a series of effects that would work under most conditions while offering enough adjustability that each user would feel like their ideas could be expressed without too many limitations.
SPECTRA’s artistry centered around “filters” – unique treatments to convert the 3D data captured by Tango into pulsing geometric shapes, popping confetti, glowing stars, bubbles, and many other looks. Once a look was chosen, there were controls to adjust the size, shape, color, and movement of the visuals, as well as setting the background color and even choosing to view the final creation from an alternate camera angle. Once the user was satisfied with the results of the fine-tuning, SPECTRA turned the result into a shareable video and offered one-click sharing to their preferred social platforms.
Making a tool to easily capture and edit 3D pointcloud data on a consumer mobile device was never possible before Google added depth sensors to smartphones, so the creative and technical challenges faced by the SPECTRA development team were completely unique to this project. The sensors used by Tango were relatively low resolution and low frame rate, so the stylization of the captured 3D data had to be specifically designed to yield pleasing results despite a sparse input in the form of a 3D point cloud and device pose (position and rotation) per frame. Performance on the phones was a constant consideration, as many visual treatments that were explored would not run in realtime and still look good. The UX of manipulating 3D data is something that is foreign to most consumers, so crafting a UI that is playful and easy to use while avoiding too much confusion was an enormous challenge, especially when designing ways for users to change camera movement after the fact.
Tango sensors were included on two smartphone models: the Lenovo Phab Pro 2 and the Asus Zenphone AR. To see samples of the app in action, check out Andrew’s Instagram page or search for videos online from one of these two phones.
Unfortunately, Google ended the Tango program a little over a year after it was launched, choosing to refocus their AR strategies on software that would work with as many smartphones as possible instead of just those with specialized sensors. Google’s ARCore and Apple’s ARKit are beginning to get some Tango-like features, but we are still one or two hardware generations away from having proper depth sensing capabilities on millions of smartphones. But once we have them, apps like SPECTRA will be able to finally reach the masses so that everyone can make art like this on their own phones.