For some project I had the opportunity to look at H264 decoding and the hardware decoding using VAAPI. An ideal excuse to write a post on that subject...
What is VAAPI ?
Taken from Wikipedia:
The main motivation for VA API is to enable hardware-accelerated video decode at various entry-points (VLD, IDCT, motion compensation, deblocking) for the prevailing coding standards today (MPEG-2, MPEG-4 ASP/H.263, MPEG-4 AVC/H.264, H.265/HEVC, and VC-1/WMV3). Extending XvMC was considered, but due to its original design for MPEG-2 MotionComp only, it made more sense to design an interface from scratch that can fully expose the video decode capabilities in today's GPUs.
This API is also usable directly with a DRM device, a DRI render node for example: very neat to offload the GPU decoding without a X server. You can also use it from Wayland of course.
The idea is to feed the GPU with a video stream (H264, VP9 ou MPEG) and the GPU will do the decoding and the rendering in a surface.