Virtual Reality and Augmented Reality

I just finished my first virtual reality project. What a ride. The fun part was that everything had to be invented or designed. The frustrating part was that everything had to be invented and designed. A great opportunity came up when Global Heritage Fund, a non-profit that helps take care of world heritage sites was looking to show their donors the amazing places that they are supporting all over the world. Sometimes these places are extremely hard to access and virtual reality is a great way to fully immerse someone in another location from the comfort of their home.

The chosen location was Ciudad Perdida, a series of terraces on a mountain deep in the Colombian jungle, once inhabited by the Tayrona, who mysteriously disappeared at the end of the 16th century.

While preparations for our trip took place we started testing cameras and building rigs to capture virtual reality. This took a couple of months to research everything from GoPros to Blackmagic 4K minis, to all the 360 camera that are out there. We tried any amount of cameras from 2 all the way up to 12. We tried them in pairs to create the 3D effect and any other configuration you can imagine. There were lots of failures and near misses but we learned a lot. Ultimately for the budget we had and the quality today's smartphones can process we settled on a rig with 8 Kodak Pixpro SP360s 4K in a circle. The cameras come with 235 degree wide angle lenses and while they are consumer grade, they allow for some white balance and light settings. They can be synced by one remote control that keep eight cameras within a one frame sync. The cameras were not genlocked. The rig was first build out of foam core and when everything was working, a local machine shop laser-cut two camera plates out of aluminum, based on a CAD drawing done at Allied, the VFX company at Beast that was going to take care of the post part. The stitching of the VR.

We tested the rig and the post process for a couple of weeks until Michael Lester, Evan Ryan and Gordon Whittmann got it to work even when the cameras where moving. VR in 3D with auto stitching and minimal cleanup to get rid of the tripod. What they achieved is beyond impressive.

The plan was to not only deliver a VR experience in 3D but to augment it with elements in CG to lead the eye and to recreate what the location would have looked like a thousand years ago. Specifically we added butterflies and we added the huts that the Tayrona used to live in. This was a complication that the team at Allied executed beautifully.

The trip itself was pretty tough. A two day hike by foot up one of the highest coastal mountains in the world, through the jungle, with 300 lbs of equipment. We had an expert archeologist, three park rangers, mule drivers and their mules, a cook, as well as a local filmmaker and a platoon of the Colombian army helping us get up the more than fifteen hundred moss-covered steps to get to the site. I have never been so tired in my life. By the end of the journey, I was only doing fifteen steps at a time, with a five minute break in between.

We spent a week up there during which we shot 60 VR scenes in all types of light conditions. We would wake up at 4.30am and go scout to figure out where the most beautiful light would be -- the light in the jungle changes rapidly -- we also figured out where we could hide from the camera. When you shoot in 360, you have to get out of the way, or there's extra 'crew clean up' in post, which we wanted to avoid. Easy in the jungle, but not so easy on a white sand virgin beach.

At night we would download and organize the footage. Data management is huge in VR, and when done properly sets you up for a smooth post production process. We would also spend hours recharging the batteries and getting the cameras ready for the next day. There wouldn't always be electricity so we took tons of batteries.

After two week being bitten by millions of mosquitos, other bugs, nearly stepping on snakes, running through torrential rain to make it past the swelling rivers and being covered in mud from head to toe for days, we came back to do post.

While editorial was done in Premiere to put together the chosen scenes, which were stitched in 2D in Autopano, the team at Allied would take that frame count and do the same using their proprietary software and do the final stitch in 3D. All editorial choices as well as rough temp sound was done in Premiere and shared with the client. Later on all the visuals would be replaced with the stitched scenes from Allied, augmented with VFX and matte paintings. The whole process took a few weeks which was mainly rendering time. All final sound was done Binaurally by Christopher Forest at Handsome Sound.

In the end, we delivered a VR experience that takes viewers on the same path as the Conquistadors in the 15th century, all the way from a white sand beach in the Caribbean, crossing jungle rivers and up to Ciudad Perdida. It was a trip I will never forget and a fun way to test new technology. The client and Oculus like the results and there's more to come.

Some of our phone clips and left overs cut together in a making of.

Early tests with just 2 Pixpros and a foam core plate.

What we carried up the mountain. With the help of a few mules.

On the journey. We hiked some of it at night through a dense jungle with only our headlights on. Pretty spooky.