Augmented Reality, if you don’t know, is the combining of digital images (video, animation, 3D objects) with a live video image (think iPhone camera) to create the illusion of the digital objects existing in the real world. If you already downloaded iOS 11 for the iPhone and you have an iPhone 6s or newer, download the app, Housecraft. Go find yourself an empty room, a real one in the real world. One big enough that you could comfortably add a few pieces of furniture and put together a little conversation pit. Here’s where it gets fun! Follow the instructions to scan your floor. Now drop in a sofa, a couple of chairs and a table. Move them around, change the fabric and color. Once you have it all laid out, go ahead and walk around the space. As you change your position, the digital furniture remains planted to the floor. For some game-style fun, make a table drop on a sofa and watch the simulated physics engine bounce objects all around. Voilá! Set design and approval, on location, using your iPhone as a magic director’s viewfinder.
For a bigger catalog of furniture (sans the physics engine) download IKEA Place and start designing your next set. Combine that with their generous 365-day return window and you can go from virtual, to real, to production, to return all with your phone and a pick up truck. Just don’t spill anything on the Ektorp love seat—you break it, you buy it! How is this magic possible? ARKit from Apple on iOS 11 is the answer. To do AR you need four things: a camera, a screen, an object tracker and a real-time graphics processing engine. Your iPhone already has a video camera and screen, and now with ARKit in iOS 11, it has added the software engine to make it work. Real-time tracking and compositing are nothing new. You do it all the time in Snapchat when using a filter to give yourself cat ears. That kind of object tracking is 2D. It looks for the common features of your face (eyes, nose, chin) and uses that to composite and warp its prebuilt graphics to animate in response to grinning or sticking your tongue out. It’s cool, but the illusion breaks as soon as you turn your head too far, or if your face goes off camera. ARKit from Apple is a fully realized 3D volume scanner. Meaning, it uses the camera, accelerometer, compass and a bunch of smart computation to build a model of the space around you (floors, ceiling, distances, angles, etc.). As you move the camera around in the real world, the 3D model moves inside the phone. This has obvious implications for gaming and social media: run around your house shooting zombies, or leave animated gifs up around town for other people to find. But I think it has profound implications for filmmaking. Here’s where I think we’re all going to start seeing the virtual world pop up in our real world sets in the coming year:
SET DESIGN
Ikea set design is just the beginning. Tech-savvy prop houses will start scanning and offering 3D models of their offerings so designers can audition objects on location before ever pulling things from warehouses or off the truck. Sets, complete with furniture, props, wall color and wardrobe on virtual stand-ins will be shared as digital files. Every one in every department will be able to pick up their phone or iPad and then walk around the space, make changes, choices and suggestions before ever setting foot on a stage.
VFX
A day spent on a green screen reacting to things that are there will be transformed, as directors and actors can whip out their phones and see pre-rendered versions of what they’re supposed to be interacting with.
PRODUCTION
Watching and recording a rehearsal of actors with digital elements through the magic director’s viewfinder is exactly what James Cameron did during the production of Avatar. While he did real-time motion capture, video capture rendering and compositing (which required an amazing amount of technical wizardry), the most profound aspect of his system was the freedom to walk around with his actors and his camera inside the virtual set. Here’s a proof-of-concept of ARKit and Unity 3D:
This aspect of that workflow is what ARKit is offering. But on your iOS device in a living room, rather than a massive stage full of technicians. There’s still a ton of animation, compositing and work to be done in post to make your low-budget sci-fi look epic, however now you have the ability to see what everyone is looking at instead of just shooting blindly into a green screen. ARKit is evidence once again of Apple’s interesting style of innovation. It is certainly not the first company to develop Augmented Reality tools. Microsoft and Alphabet have sunk hundreds of millions of dollars into their AR platforms (hardware and software). Like MP3 players and smartphones before it, Apple didn’t need to invent a technology to figure out a really interesting spin on it. This is particularly true in the film community where Apple computers and iPhones are ubiquitous. Having AR as part of the operating system of a tool used daily in film production (iPhones) means AR will soon become a tool used daily by filmmakers. Learn more about the evolution of filmmaking tech, including where we might be headed next, by checking out more of Eric Escobar’s Hacking Film series. To learn how to become a Member of Film Independent just click here.
