Microsoft Announces MLVC: A Neural Video Codec

Suggested Pre-read: Apple Introduces PICO: AI Image Codec

ML Video Codec (MLVC)

Microsoft has announced a neural-network-based video codec, which has been open-sourced to the community under the MIT License.

MLVC aims to significantly reduce bandwidth consumption compared to traditional video codecs. Neural codecs have been making massive headlines lately, in addition to this announcement, Apple recently introduced PICO, an AI image codec (linked above).

The snapshot below is captured from a video on Microsoft’s website. It demonstrates the superior quality MLVC provides at a low 200 Kbps bitrate compared to H.265 (a traditional video codec).

Source: Microsoft

Why MLVC

MLVC is an iteration that improves upon DCVC (Deep Contextual Video Compression) family of NVC (Neural Video Codec), which Microsoft Research has been open-sourcing since 2021.

Core benefits over DCVC include:

• Improved compression efficiency
• Real-time performance on commodity Neural Processing Units (NPUs)
• Cross-platform support

Microsoft is releasing MLVC’s source code to the open-source community to encourage broader video community contributions, which is excellent news.

BORROWING a part of my PICO article and posting it here for relevance:

AI/Learned Image Codecs vs Traditional Image Codecs.

Traditional image codecs have been around for decades, and with each release, we see new features and better compression techniques. So what exactly are these AI image codecs trying to solve?.

Traditional image codecs rely on a static ruleset—a fixed set of mathematical formulas. In contrast, learned codecs rely on machine learning-based prediction techniques backed by the massive volumes of data they are trained on. Learned image codecs are designed to dynamically determine which data the human eye will or won’t notice, allowing them to efficiently discard unnecessary information while generating highly realistic textures during the decoding process.

Traditional video codecs require substantial engineering effort with each generation to achieve incremental gains. Furthermore, they rely on dedicated hardware support, which takes years to become widely available. As a result, older standards like H.264 remain active and dominant.

In contrast, MLVC replaces conventional primitives — such as motion estimation, transforms and entropy modeling — with end-to-end learned neural compression, trained directly against rate-distortion objectives, and run on general-purpose NPU devices.

Bandwidth Savings

This table, posted by Microsoft, compares MLVC to popular video codecs, highlighting its lower bitrate and the resulting savings in bandwidth and storage.

For example, for 360p video at 30 fps, where H.264 requires 1 Mbps, MLVC requires roughly 122 kbps for equivalent quality — about one-eight the bitrate under real-time conditions. The inference compute was kept approximately equal for the 360p and 540p resolutions.

Beyond video compression efficiency, MLVC also offers:

• NPU-first design Built to run on the AI accelerators already shipping in modern devices from Apple, Intel, and Qualcomm.
• Real-time execution Microsoft has demonstrated this at 540p(30 fps) on the above chips.
• Content adaptive behavior Does not require hand-tuned Rate Distortion Optimization heuristics.
• Scaling trajectory MLVC’s efficiency improves with additional training and increased model capacity.

Live in Production

Microsoft is currently rolling it out in Microsoft Teams to validate its improvements during real peer-to-peer video calls. To ensure reliability and gather insights, they are actively conducting A/B testing.

Source Code

The MLVC repository is available at https://github.com/microsoft/mlvc and shared under the MIT License.