You could break Playdeo's productions into three broad areas
Concepting and prototyping
Creating new formats, new interactive mechanics, world, character and script development
Mixed Reality production
The production of 3D-tracked video content, production design, costume, locations, audio, editing, post production, VFX and music
Development and release
Developing games, interactive characters and interactions, integrating the video into the game environment, publishing, marketing and business intelligence
From scriptwriting to storyboard, from production design to costume, shoot and post.
Making a new playable space inside video of course requires a significant amount of software and engineering invention.
For example, with Avo, the player draws a path with their finger across the screen, for Avo to follow, and in this way they choose Avo’s direction and story, as well as changing their perspective on the scene. Every time a player moves, they’re engaging a dynamic, rules-based system that responds by choosing from multiple cameras that follow where Avo is directed. This creates a set of camera cuts that frame the character in the most interesting way at any given moment.
On a lower level, we needed to create a system that uses two video decoders to dynamically construct visual edits on-device, with no frame pauses or glitches.
For Mystery Coast, we invented and built Signet, a full 3D volumetric painting system, based on signed distance fields, that has enabled a buttery smooth, satisfying and expressive cleaning mechanic in order to restore 3D treasure.
We use our own camera software built on Apple's ARKit, that allow us to instantly record and experiment with game mechanics mapped into recorded video, as well as making remote production and instant editing possible.
On both Avo and Mystery Coast, the entire set and all props and objects were 3D-scanned, to add depth to video. When brought together in the Unity game engine they allow game characters, mechanics and interactions to sit and operate comfortably and convincingly in the 3D world.
