I made a video about photorealism. Really, I’m somewhat surprised how strong the response to this as been – I felt the stance I took in the episode was rather milquetoast/middle of the road, not really outright condemning photorealism nor singing its praises. *shrug*

—

Update 6/21/2016: Putting the script in here!

The general consensus when looking back at the history of graphical fidelity in videogames is “We’ve come a long way.” Every generation of hardware has opened up entirely new aesthetic possibilities. In eleven years we went from the crudity of Adventure to the colorful cartoon universe of Super Mario World, and in another eleven years we went from Doom to Half-Life 2.

Now, these days most developers and a sizeable subset of players see this progress as a broadening of our potential as a visual medium. Adventure looked like this because it had to look like this, but these days we have a huge and largely untapped palette of potential visual aesthetics. And in that regard, improved hardware power and advanced rendering engines have really opened up what we can do with videogames (at least as far as the video part is concerned).

But as each step has opened up new possibilities, it’s also enabled us to move towards a higher level of fidelity with the real world. And as a result, some people have developed a mistaken idea that photorealistic graphics are the end goal of game engines, and even games themselves. It’s an idea so ingrained in the history of the industry that quote-unquote ‘realistic graphics’ is an unequivocal statement of praise. It’s an idea that has executives at major publishers declaring we won’t have emotion in games until we achieve photorealism. And it’s an idea that has major game developers laughably declaring that they’ve conquered the uncanny valley because of a single animation trick they’ve pinned their entire game on. And again.

But there’s a lot of problems with this fixation on realistic imagery in games – not the least of which is that photorealism is a pipe dream. First of all, the tech just isn’t there. We don’t have engines that can do ray tracing at a consistent 60 FPS at HD resolutions against complicated geometry on commercial grade game hardware. And yes, I’m aware of experiments like Quake Wars: Ray Traced, but Quake Wars was a game early this generation that was likely far less VRAM and post-processing intensive than current games, and even then the highest frame rates were in the mid-thirties on gaming rigs designed specifically for the renderer. Proving that a ray tracing engine is possible is not the same thing as proving that it’s viable or even desirable.

But nevermind rendering technology, as the costs of asset creation alone would be enough to sink photorealism. As graphical fidelity has increased budgets have exploded. Just to go from this (Halo) to this (Halo 3) we’ve needed to balloon game budgets to upwards of sixty million dollars or more. Games today require armies of artists to render the scenes we see, and they’re already so expensive that mass consolidation has only gotten more and more common. We’re at a point where a lot of developers are really really concerned about what the next generation of games is going to cost ot make, and if Epic Games is to be believed, they’ll only look about this good. Remember how EA said that Dead Space 3 has to sell five million copies or more to justify the franchise continuing? This is why. And that’s on this console generation.

The point is that making photorelaistic graphics is simply not financially viable. Improved tools and procedural content generation may help alleviate the cost, but they’ve yet to really find a market in an industry that’s more comfortable hiring 20 more art school grads than changing its entire art asset pipeline around. So even if you had the technology you couldn’t afford to make a game’s worth of content with it at photorealistic levels of fidelity.

But even that doesn’t matter. Once you’ve fixed the technology and cost concerns and have overcome the uncanny valley with true photorealistic imagery… you’ve only just started.

Next you have to tackle all of the problems with animating a photorealistic scene. Things can look absolutely photoreal in stills, but if you’re using canned animations that repeat with mechanical timing you’re still going to be tipping your hand that something is really off and robotic. Imagine if something looked photoreal but used the animation system of Call of Duty. It’d straddle the line somewhere between immersion breaking and creepy as hell. Things like Euphoria or procedurally animated meshes are taking steps to correct this, but they’re only covering the overall meshes. If we’re going to be pedantic about true photorealism, you need to have dynamic mesh deformation on individual strands of hair based on the stresses they receive! Clothes need to fold and bend and flap instead of simply having a texture squished by moving two vertices next to each other! Lip synching needs to be in perfect harmony with the words that are spoken.

And speaking of faces, hey, let’s talk about emoting! The best we’ve done in the photorealistic facial animation space is LA Noire, and while that’s a really neat technical accomplishment it has several weaknesses. First, you need to hire a real actor who can perform his lines in a giant facial recording system, which is neither cheap nor easy. Second, it’s not exactly procedural. Sure, you can capture a character giving a convincing speech, but what happens five minutes later when you’re in the middle of actual gameplay and you start throwing trash at him? Does he have the same canned annoyance response? Does he make no response? How do you handle a photorealistic looking character responding to player stupidity? Dynamically animating a photoreal face in real time with all of the subtlety delivered by an actual actor is absolutely beyond anything we can currently do. If we could do it they’d be replacing actual actors!

Then there’s the problem of fluid dynamics. At our fanciest right now, liquids are treated either as an animation trick like in Bioshock or as a glorified particle effect like in Borderlands 2 that makes the water look all blobby and thick – closer to warm Gogurt than H2O. Look, here’s some CGI footage of water streaming around. It’s not quite photoreal, but it’s pretty impressive! Do you know how long this took to render? 220 HOURS. We do not have anything close to being able to render photoreal, believable liquids in real time on top of all other graphical and gameplay considerations. Not even close.

But, hypothetically, let’s say you’ve solve even those problems. Real time fluid dynamics aren’t a problem, cloth ripples and bends as characters move just like it would in real life, every body part of every character is driven by a complex AI that results in unique animations including lip synching, and the footage from your game is functionally indistinguishable from HD camera footage. This is so far beyond reasonably possible let alone affordable that it’s laughable, but just, stick with me here.

What do you think that means for gameplay? Specifically, what do you think having photorealistic graphics implies about the system of a game? The video part of a video game isn’t just to look pretty, but to convey the state of the game’s system. Consequently how a game looks informs how we interpret that system. Minecraft has a blocky aesthetic, but that presentation gives you a clear picture of both the depth and complexity of its systems as well as for how the world works – that is, that it’s built on blocks. Its graphics inform its gameplay, and the state of its gameplay is reflected in the graphics.

Now picture a genuinely photorealistic Minecraft. You can’t really do it, right? The mechanics and systems are too abstract to support a photorealistic environment. Either it’s a world built out of blocks and therefore plays like Minecraft, or it’s a photorealistic world that plays nothing like Minecraft – you can’t have both. And really, when you think about it games that look more or less identical to live action couldn’t support almost any of the game mechanics we enjoy today! At least not without either looking completely ridiculous or being fundamentally changed to work in a realistic way. Torchlight’s loot system where you can carry a truck’s worth of spears and shields in a backpack, and then have your dog sell them? Call of Duty’s spawn anywhere multiplayer where a graze from a knife instantly kills you? Assassin’s Creed’s generously ignorant townfolk? All would either need to be made realistic or would totally break with the otherwise realistic presentation in ways that would be utterly jarring. Modern games can get away with mechanics that might not make a lot of sense because both their systemization and graphical representation are highly metaphorical. Even games that take a realistic-ish bent to their graphics, allow you to do things that would at best really bizarre and at worst completely awkward if they looked photoreal (Crysis?).

Abstraction gives us freedom to talk about the parts of a system we care about and ignore the rest. If the placement of a camera is how films decide what’s worth observing and discussing, then deciding what to simulate if how games make the same decision. Photorealism isn’t just unobtainable, it would be an effort to remove that abstraction and demand higher fidelity mechanics to match the higher fidelity visuals. And at that point you don’t really have videogames as a form of expression through intentional system design and artful visuals, but instead you’ve achieved world simulations like The Matrix. And that’s cool, I mean, who wouldn’t want to have a Matrix to call their own, but it’s hardly a videogame as we understand it today.

The reality is that as much as some people don’t like to admit it, games aren’t just a visual medium, they’re an animated visual medium. They’re not playable movies, they’re playable cartoons or interactive paintings. We don’t really acknowledge this a lot – games reference live action far more than they reference animation. And game designers who are would-be-film-directors want to be the James Cameron, or Martin Scoresece or Quinten Tarantino of games, not the John Lasseter, Tex Avery, or Miayzaki of games. This sort of mentality got especially bad in the late 90’s and early 2000’s as 3D rendering engines were coming into their own. After the death of silliwood with with developers still desperate to be taken as seriously as film, the mainstream American games industry settled into this awkward compromise, aesthetically speaking. They started making games that looked realistic enough to be taken seriously given the techological limits of the time but stylized just enough to have some artistic cohesion. The result was basically a Poser Porn aesthetic – realistic enough to gte the job done, but still plastic-y and fake. You could dress the models up like Star Wars figures or Space Marines or Elves or even just normal people, but that’s mostly just the setting more than the visual aesthetic. For about a decade there we had almost exclusively plastic, pasty lookin’ people. There were exceptions, but they were almost exclusively from Japan – like your Jet Set Radios, your Vib Ribbons, your Parappa The Rappers, your Wind Wakers, your Viewtiful Joes, your Okamis…. Oh, and Rayman. Even though he’s from France.

But as 2012 draws to a close we’re starting to see western game developers and American developers in particular start to toy with the rendering styles. Games like Borderlands 2, The Walking Dead, Dishonored, Torchlight 2, Darksiders 2, the upcoming Sim City, and even older titles like Team Fortress 2 and that Prince of Persia reboot all show publishers and therefore audiences are increasingly comfortable with the idea of games being as an animated medium as much as an interactive one. They’re not as experimental as the Japanese in the early 2000’s, but this is a region where Call of Duty is still the number one selling game every year, so… baby steps.

My point isn’t that realistic graphics are bad – lots of my favorite games have realistic-ish graphics. My point is that we have this awesome palette of the modern computer screen to work with, complete with HD resolutions, half a gig to two gigs of video ram to play with, and full 32 bit color…. yet this obsession with photorealism has led us to taking advantage of very little of that power in an artistic sense if not an engineering sense. I mean, indie games have experimented with art styles for years, if only out necessity. And the results, I think, span a much more thorough and enjoyable cross section of what a computer is capable of displaying. I mean, if computers can do this…. and this…. and this… and this… why do so many games look like this?

So I guess my plea is as follows: If you haven’t yet, give up on photorealism. It’s a pipe dream, it’ll never happen, and even if it does it won’t result in anything that looks or plays like a videogame as you know it. Instead, encourage developers to embrace all aesthetics, from the realistic to the completely abstract, from Salvador Dali to the Fleisher brothers, from cohesive and lush to minimalist programmer art. The medium will be richer for it, and it’s a far less tragic thought than an entire industry chasing after a