Understanding MetalFX, Apple's graphics performance booster

In 2014 Apple introduced its new graphics and games rendering technology called Metal.

Metal is a rendering API that provides fast performance optimized for Apple hardware. With the advent of Apple Silicon GPUs, Metal gets an even bigger performance boost.

Metal - Game rendering

One problem in computer games graphics rendering is how to draw large numbers of polygon models fast with high-resolution surface shaders and effects quickly. As frame rates and game scene complexity have increased, this problem has created bottlenecks in graphics rendering pipelines.

The Metal API is an Apple C++ API which has its own shading language called, as you would expect, Metal Shading Language (MSL). The Metal API has been compared to DirectX 12 on Windows, or the third-party graphics rendering library Vulkan.

There are also Swift and Objective-C versions of the Metal API.

However, since Metal is from Apple and designed for Apple hardware, it provides the best graphics rendering performance of any available rendering technology on those devices.

You can see a sample of Metal rendering in action every time you start an Apple Arcade game on an iOS or tvOS device - the short Apple Arcade logo bounce animation that runs before every Apple Arcade game is rendered using Metal.

Many games use GPU hardware acceleration and precomputed shaders to speed up graphics rendering. In Metal, graphics commands are encoded in advance and then submitted to GPUs for rendering.

Precomputed render states tell the GPU in advance how to configure the rendering pipeline for maximum performance.

Metal - Rendering shaders

Rendering shaders, or precomputed programs that control light, darkness, and color, are used during 3D scene rending to provide realism and special effects. The MSL describes how Metal shaders work.

Shaders were first introduced in commercial software in 1988 with Pixar's RenderMan software, which has a long history on the Mac going back decades.

In most modern GPU rendering pipelines, both 3D data and shading language programs are sent to the GPU for processing, which then manages a multi-step process or pipeline by which each frame in a scene is created.

The rendering steps roughly involve geometry transformations, breaking models down into triangles and quads, applying fragment shaders, depth testing and combining, and the exclusion of occluded surfaces and objects. The last step involves combining it all into the final frame buffer for viewing.

All of this happens dozens of times per second on modern GPU hardware.

Metal also includes performance shaders - programs that speed up rendering while reducing GPU load. Performance shaders include image filtering, neural network, advanced math, and ray tracing.

Ray tracing vastly improves 3D model object surfaces, textures, and reflections from light sources.

What is MetalFX?

In 2020 Apple announced it had ported Metal to Apple Silicon CPUs and GPUs to provide much faster performance. At WWDC 2022, Apple introduced a new feature in Metal called MetalFX.

MetalFX allows for faster and more accurate scene rendering by recomputing and anti-aliasing some frames to recalculate and smooth rendered object edges.

MetalFX's secret weapon is a technology called upscaling.

The idea is to render an object or scene at a lower resolution, which makes overall frame buffer rendering faster. Then, it adds an additional step before it is displayed by enlarging the scene.

By doing most of the scene rendering at smaller dimensions, scenes can be rendered faster and with more detail and complexity. In most cases, it's usually faster to render scene objects at a smaller scale then improve the overall resolution of the final frame than it is to render the entire scene at a larger final dimension.

By using upscaling, MetalFX can provide both faster scene rendering and more object detail and sharpness.

Upscaling types in MetalFX

MetalFX uses two distinct upscaling types: spatial upscaling and temporal anti-aliasing upscaling.

In spatial upscaling, two Metal classes - MTLFXSpatialScalerDescriptor and MTLFXSpatialScaler - are created. The descriptor object describes details of the spatial scaler object, and the MTLFXSpatialScaler object does the actual encoding (via its encode method).

When encode is called, the MTLFXSpatialScaler object creates and sends render commands to a MTLCommandBuffer object for rendering. This process is repeated for every frame in a scene.

MetalFX spatial upscaling can provide roughly up to double the scene detail with no material loss of performance.

Creating MTLFXSpatialScaler objects is computationally expensive and should only be done when an application starts or a display changes resolutions.

In temporal anti-aliasing upscaling, data from the previous frame is used as input for upscaling the current frame.

Supersampling and temporal sampling are techniques that use individual pixel data from frames to achieve upscaling. Data both from each pixel and pixels from across multiple frames are sampled.

Temporal anti-aliasing is slightly more compute-intensive and requires additional color, depth, and motion data in order to work.

Here's another example of anti-aliasing upscaling from Apple's WWDC presentation:

Most MetalFX upscaling processing should occur after rendering but before any final post-processing is done on frames.

One use of MetalFX anti-aliasing upscaling is to increase the resolution of 1080p frames so that they approach 4K resolution.

Just as with spatial upscaling, MetalFX provides two classes for anti-aliasing upscaling: MTLFXTemporalScalerDescriptor and MTLFXTemporalScaler.

Complete MetalFX documentation is available on Apple's developer website. Apple also held a MetalFX session video at WWDC '22 titled "Boost performance with MetalFX Upscaling" which describes the use of MetalFX in detail, including sample code.

Several AAA game titles including No Man's Sky already use MetalFX on Apple devices.

MetalFX promises modern graphics compute techniques to increase both scene resolution and game performance on modern GPU hardware. It's likely the new Metal technology will revolutionize gaming on Apple devices.