the Magicverse has you
Magic Leap’s John Gaeta is probably one of the only people on the planet who can say something like ‘dynamic adaptive geospatially aligned applications’, and from him, it makes sense. The former VFX supervisor, perhaps most well-known for overseeing the effects of The Matrix trilogy and helping to instigate Bullet Time into cinematic lore, is now overseeing a different kind of immersive experience for users as senior vice president of creative strategy at Magic Leap.
When you talk to Gaeta – who prior to Magic Leap was the executive director at ILMxLAB – you really feel like you might be talking to someone who can see the future. With Bullet Time and other developments such as Universal Capture and spherical HDR photography, Gaeta kind of did. befores & afters sat down with him at the VIEW Conference in Turin to get his wide-ranging take on the new kind of immersive future he’s now involved with, via what has become known as the ‘Magicverse’.
b&a: I want to jump right in and talk about what people are able to do with Magic Leap right now in terms of augmented reality and mixed reality. One thing is the A.I. assistant, Mica – it feels very different to interact with her – why is that?
John Gaeta (SVP creative strategy, Magic Leap): The reason why Mica is interesting is because she has all these little micro-expressions. Really, really super subtle, and that’s where the Uncanny Valley really crushes almost everything. The one thing, though, that a lot of people rightfully question is, well, how will that virtual human ever feel natural and completely adaptive in any situation? Learning – deep learning – will play a role in some of that, but a person will need to allow themselves to be learned from at that kind of level.
Deep learning aside, one can pour the real person into their direct vessel, so you can drive yourself remotely if you think about it. And so the Uncanny Valley falls away when you have both micro-expressions and explicit subtleties of you coming through your literal one-to-one representation.
b&a: Do you feel everyday people struggle with contemplating the idea of, ‘Well, this is all awesome and everything, but what can I really use something like Mica for,’ and then, in general, ‘what can I use mixed reality for’?
John Gaeta: Sure. Some people will want it for artistic purposes et cetera, but in truth the killer app is communications for mixed reality; holopresence. Co-presence. It’s really going to happen and it’s going to be a medium beyond television at some point and broadcast. Being co-present is going to be everything. And it’s not only just being co-present, but co-present and collaborative around applications, which is a dry word which can mean almost anything. An application can exist in space. It could be entertainment, it could be education, it could be health care, it could be transportation, it could be civics and government, it could be public, it could be manufacture.
So we’re imagining a future world where dynamic adaptive geospatially aligned applications with high fidelity 3D graphics that couple themselves to dynamic real-time things, situations, cognizant of people, aware of the context of the place, right. So that’s spatial computing, and that’s absolutely going to happen and there are going to be a lot of opportunities for developers that can combine things. In fact, the visual effects industry is a good foundation to understand the complexity of integration.
b&a: Yes, because in visual effects things are often solved for one particular reason, entertainment or whatever, but are often applied to other things, aren’t they? I’m thinking, matchmoving, stitching, tracking – all things that have become part of VR and MR.
John Gaeta: That’s right, we have to be aware of the spatial relationship of things. And dynamically, too. You have to be aware of light for flawless integration. You have to be aware of change over time. There’s just so many things that we’ve learned. People can cross over if the tools and systems are in place to allow them. I think it would be naïve to think that the film industry is going to contain its greatest talent for very long. Unless they’re really really driven by the sculptural artistry aspects of it.
b&a: Did it seem like a very obvious segue for you, in fact, to go from VFX to somewhere like ILMxLAB and then Magic Leap? Did that seem like to you – ‘Oh, I’ve got to do this, I have to do this’?
John Gaeta: Pretty much. A lot of it is thanks to Kim Libreri (now CTO of Epic Games, makers of Unreal Engine). Kim and I have been on this since Matrix – since Universal Capture – we’ve been thinking about virtual constructs and a world where you can componentize things dynamically and have them available, so yeah, we’ve been thinking about it a lot.
Kim and Tim Sweeney (CEO of Epic Games) are a great match. Tim’s world view and Kim’s skills go together very well. They are going to be a different company than other engine makers, but they all have their place in different ways. The thing maybe that no one really expected is that they were going to have that incredible success with an actual new property like Fortnite. It should be a lesson to people because there’s no economic model that works well for cinema anymore. If you’re hoping that innovation is properly funded as it might have been in the past, what’s the reason for it? If the graphics community still involved in cinema does not figure out how to start crossing over so that the media that they produce is apply-able across multi platform, then it’s just going to get harsher and harder.
The thing about it is that if you have a real-time base and a way for people to be social inside of it. Fortnite is basically Hunger Games, so they’ve basically offered Hunger Games for free. That could absolutely take place in the real world, by the way, with virtual constructs, but that’s not where Epic is being successful monetarily. They are providing really simple personal expression for people. For example, I can change my skin. Now that’s incredibly cost effective to offer that, and there it is. P.S., it can happen out in the real world with mixed reality too.
b&a: There’s an obvious challenge though, for Magic Leap, isn’t there, to create an industry – a world – of users and good experiences that you can ultimately get income from?
John Gaeta: Here’s the way I would put it. So Magic Leap, first. Spatial computing is our application in a space that has context, with people that’s more context, and if the application is aware of what people are doing when they’re engaging. So, that doesn’t require glasses to do. That’s to geosynchronously pause in an app and have the feedback loop that gives you all the info that you need after you’ve meshed your space et cetera. A person with a mobile device can come in and interact with that app. A person with any kind of mixed reality glasses, even a person with VR glasses that are being tracked into the room. Even a person who might not actually be in the room but can reach the room like an online game, or going into VR chat or going into Fortnite. So I can still go in the room. If I’m on my PC I could be on the other side of the world and enter the room and be present in the room, co-present in the room.
So there becomes this sort of universal unit, the app, the spatial computing application inside that destination. That destination can exist in XR. It can be real world, it can be in VR and PC. It can be all those things concurrently. So, that’s what it’s about. And if developers make applications in the right way and/or, let’s say the map of the Magicverse comes together and that in and of itself is a spatial computing destination with all its layers and the systems that need to be in place, the feedback loop that has to be in place, that is a massive user base because all view ports can plug into that.
So Magic Leap will have one of the most powerful ones and that does have high value. I would also venture to say that the entertainment business is not the most compelling business for spatial computing. There’s so many other businesses. Developers and people who work in graphics and real-time have opportunities in all sorts of other sectors and there is going to be a lot of money in these sectors to create these applications.
So, yes, Magic Leap is a platform company but it’s also a device company. A lot of people mistake Magic Leap as a device company only. It’s not a device company. It’s a platform company. It’s interested in what it takes to enable spatial computing destinations. So think about it. All the things you would need to register in space, house that application, do those feedbacks, create those feedback loops, serve that stuff. If you understand it you can participate in it. So Magic Leap is about that.
And no doubt that this stuff is going to happen at city scale. Cities are going to be lit up by 5G in about two to five years. Think about the cities lit up by 5G in two to five years. Those will become the first really serious spatial computing destinations. They will be able to broadcast immersive simulcast high fidelity graphics, so people who do gaming and film, cinema and visual effects, will actually be able to produce content for that.
b&a: I wanted to finish by asking you a little about The Matrix and Bullet Time. Now, it’s probably one of the best ever DVD featurettes ever made, I think, is the Bullet Time featurette for that film, but when I re-watched it recently, you mention things on it like ‘virtual constructs’ and ‘immersion’ – it all feels like a prophecy, in a way, to what you’re doing now.
John Gaeta: Well, it was serendipity. It’s a movie about virtual reality, the nature of it, and the directors were young people. We were all a certain age. And they were completely open minded and they didn’t have any – how would I put it -they didn’t have any baggage that would prevent them from letting us try risky new techniques. And the techniques we tried were ones that – we’d thought about this really hard, what would you need to do? In the future, when you tried to do this stuff, you’d have to capture the world and you’d have to capture people and you’d have to do certain things to make virtual constructs and they said, ‘Try it!’
But they always had a backup. They always photographed a backup plan for many of these things, but they let Kim and I experiment and Kim is a super genius and Kim cracked incredible things in immersive photography, like spherical HDR. That wasn’t just used for 360 movies, we used for almost 90% of the backdrops of Speed Racer, but he also used them to create truly volumetric three-dimensional environments that looked cinematic. It was amazing. It seems like we’re at the beginning of the techniques actually being used.
John Gaeta will be discussing the Magicverse at FMX in Stuttgart on Wednesday May 1st at 5.45pm at the Hospitalhof.
