A Huge Leap Forward: Graphics on the iPhone 3GS

Two weeks ago, the question in everybody’s mind was, what were the new features on the new iPhone Apple would surely be announcing at WWDC? Speculation was rampant, and guesses were all over the place. One of the ideas going around was that the iPhone would have an improved graphics chip, but I admit I completely dismissed that possibility. After all, the iPhone was already way ahead of the competition in terms of raw graphics capabilities and performance.

Low and behold, I was totally surprised when Apple disclosed the details of the iPhone 3Gs. It did have a new graphics chip, and it wasn’t just a speed improvement. It was a PowerVR SGX, the equivalent of a new generation of hardware in the console world! The surprise was quickly followed by extremely excitement thinking about all the possibilities that the new hardware opens up.

Shaders. Finally!

Coming from console game development, I have come to take vertex and fragment (pixel) shaders for granted. Just like most developers don’t think twice about having floating-point support in their hardware, I had grown totally accustomed to having shaders in my toolbox. Doing any kind of graphics programming for the original iPhone felt awkward and constrained without shaders. That’s not the case anymore, because the iPhone 3Gs offers a fully-programmable graphics pipeline with shader support.

Now we can finally implement different lighting models, easily add reflections, refractions, render surfaces with bump mapping, choose non-photorealistic rendering styles, and lots of other graphic features common in modern games today. Ultimately, this means richer, more detailed graphics in the final game.

In addition to making polygons prettier, vertex shaders can also be used to perform per-vertex calculations that would have been too expensive to compute on the CPU in previous incarnations of the iPhone. The idea of squishing, expanding, or twisting meshes becomes much more feasible.

With fragment shaders we’ll finally be able to do some postprocessing to the scene every frame to add depth of field, bloom, or color correction, but I wonder what the performance cost will be. I hope it performs well in most situations and doesn’t slow down to a crawl as soon as a slightly complex shader is used. We’ll see in the next few weeks as developers finally get their hands on it.

Beyond Shaders

There’s a lot more to the iPhone 3Gs than just shader support. The new hardware supports a lot of new features that aren’t as “sexy” and are not getting as much press, but will be a big boon for graphics programmers everywhere:

1. Stencil buffer. Stencil buffers are a very flexible tool that every graphics programmer wants to have around. They can be used to mark some areas that are not drawn to (just like a real-world stencil), but they also open the door to some interesting techniques (like stencil shadows).
2. True vertex buffer objects (VBOs). If you’ve tried using vertex buffer objects on the iPhone, you were probably disappointed with the results. The API was there, but the performance was severely lacking. It turns out, the way the OpenGL driver was implemented, vertex data was being updated and copied every frame, defeating the purpose of vertex buffer objects. The good news is that the iPhone 3Gs offers true VBOs, so you can upload your vertex data once, and then it will be used efficiently by the hardware every time its rendered.
3. Unified memory. Apple likes to keep hardware details as vague as possible. Perhaps that’s so they can update the hardware later on causing a minimum of problems, but at the same time, it discourages people from taking best advantage of the hardware. One of the little-known facts about the iPhone 3G is that it had 24MB of video memory, and that’s all you could use for mesh and texture data. The iPhone 3Gs has a unified memory system, which means that graphics resources can live anywhere in memory, without any limit. That, combined with the upgrade to 256MB of RAM is going to make for some very noticeable improvement in upcoming games.
4. Texture units. The iPhone 3G had two texture units, so you were limited to fairly basic effects: diffuse texture plus a detail or a lightmap. Anything else required multiple passes, rendering the same geometry twice and slowing things down significantly. The new and improved hardware offers 8 texture units, so the sky is the limit as far as the kind of effects we can dream up. This comes with the standard warning that performance will drop the more textures we use at the same time though.

Developer’s View

There’s no doubt that the iPhone 3Gs is a whole new beast: it’s faster, has more memory, and a whole slew of new features. We need to be honest and realize it’s a whole new platform. Before we could (sort of) ignore the differences between the iPhone 3G and the iPod Touch 2nd gen, but this is not the case anymore. It’s the first step towards a fragmented iPhone market.

We’re not talking about a new code path that can be dealt with by using a few #ifdefs either. This is a fundamentally different platform that is going to require graphic assets to be authored in a different way, a new pipeline to deal with shaders correctly, and a lot more programming and optimizations. In other words, if you want to make use of the new hardware, costs are going to go up. No doubt about it.

At the same time, with over 40 million previous-gen iPhones and iPod Touches out there, developers can’t ignore that market, so we’ll be forced to support both for quite a long time to come. I suspect that a lot of developers will continue targeting only the iPhone 3G hardware and not take advantage of any new graphics features. Let’s hope it stays at that, and Apple doesn’t introduce a totally new platform every year, which would give developers flashbacks of the nightmare that is PC development.

User’s View

How will users benefit from the new hardware? The iPhone 3G can be described as being extremely similar to the Sega Dreamcast game console. If that’s the case, the iPhone 3Gs is something like a really souped-up Microsoft Xbox, sitting in between the original Xbox and the Xbox 360.

The difference in games that support the new features should be readily apparent to all users, although the small form factor and resolution of the iPhone screen will decrease the impact of the difference (now if Apple officially implements a video out so that iPhones can be plugged in to TV sets, that would make it much more noticeable).

Conclusion

Games have the potential to look much better, but only as long as developers put the extra time and resources to support what it’s effectively a whole new platform. That’s not going to be possible as long as games keep being sold for less than the cost of a latte, so I can see some changes looming on the horizon already. Will the new iPhone help separate the wheat from the chaff and make for a healthier marketplace, or will it sink all developers trying to support a fragmented platform? Only time will tell.