Surveillance Codec Guide

Published Jan 11, 2022 22:09 PM

Codecs are core to surveillance, with names like H.264, H.265, and MJPEG commonly cited. How do they work? Why should you use them? What issues may you face?

IPVM Image

In this tutorial, we examine this in-depth covering:

  • Uncompressed vs. Compressed Video
  • Inter vs. Intra Frame Compression
  • I vs P Frames
  • Other Frame Types
  • H.264 vs H.265 vs MJPEG
  • H.265 Emergence
  • Smart Codec Growth
  • Proprietary Codecs
  • Scalable Codecs - JPEG2000, SVC
  • Quality of Codecs
  • Codec Support in Surveillance
  • Future Codecs
  • What Codecs to Choose?

Uncompressed **. ********** *****

*********** *** ************ ***** ** **********, as ******* *** ******* ******* ***** easily ** **** ******* *** ************.

**** ***** ** *********, ** ** initially ************. ***** *** * **** factors ** *** **** ** ************ video:

  • *** ***** ** ****** ********* *** each *****
  • *** ***** ****** ** ****** *** frame
  • *** ***** ****** ** ****** *** second

** **** *** ***** ****, *** multiply **** ** *****. ***'* **** through **** ** *****.

Pixel ******

**** ***** ** ***** * ***** represented ** * ****** ****** * range. *** ***** ********** *** ********* the ***** *** ** ******* *** also ******* ******* *** *********/****.

  • **** *********, ***** ********** *** * range ** *** ****** (* ****) with * ************ *****, *** ************ white *** *** ******* ** ******* representing ****** ** ****.
  • ** ******, ****** *** ************ ***** today ******** ***** ** *** ***** of ****** ****** ** ********* *** the ****** ** *** *******. ** bits ** **,*** ****** ** ******.

Resolution/Framerate ******

*** ***** *** ******* *** *** easier ** ********** ** **** *** the********** ** *** *** ******, *********** ********** ***** ******** ***** counts:

IPVM Image

** **** ******** *** ******, ****** ****, *****, *****, ***. The **** ******** ** ************ ******* (>80% ** *** *******) *** ******** at ******* * *** ** *** (see******* ***** **** ***** ************ ****).

Adding ** **

**** ** ******** **** ** *********** how ******* ************ ***** *** ******. Take * ***** ***** ****** ** 30fps. ************ ** ** *** ******* of *** *********:

*********** ***** ***** ******* ******* ** ~1 **/* *** ************ *****/***** *****. In * ***, ** **** ****, you ***** **** ~***** ** ******* from * ****** ******. ***** **** drives ******** ** *** ******, * 16 ****** ************ ************ ****** ******* for ** ****, ***** **** ****** 6PBs ** ******* ****** * *** hundred **** ****** **** ***** **** hundreds ** ********* ** *******.

Codecs *** ***

******* *** **** **** ** ************ video, ********** ************ ******* ** ** cameras ********* *** ** ******** *** or **** ***** **********. *** **** this ******? ******.

***** ****** *** ***********/************* *** *** act ** *********** ** *** ******* element ** ******** *********/******* ***********.

****** ***** ***** ** ***** ***** its *** ****** ***** ** ********** wasteful ** **** ****** *** ****** with * ***** ****** ** ******* colors. ****** ******** ***** ** ******** the ****** ** ****** ******** ***** tracking ***** ****** **** *** **** or ******* ******, ******** ** ** transmit *** ***** ******* ** ****.

Inter **. ***** ***** ***********

***** *** *** *********** ********** ** compression: *****-***** *** *****-*****. ** ** critical ** ********** *** ************ ******* the *** ** **** ****** ********* consumption, ********** ***** ************, *** ******* risks.

  • *****-***** *********** ** ****** * ****** frame **** *** *** ****** ****** (example - *****)
  • *****-***** *********** ** ****** ******** ****** AND ****** ****** ****** (******** - H.264, *.***, ****-*)

*** ****** ******* *****-***** *********** *** some **** **** ******* **** ***** and ***** ***** ***********.

Intra-frame ***********

*****-***** *********** **** ***** ** *** frame ** * ****, ***** *** best ** ******** **** ** ** that *****. **** ****** ***** ** a ****** ** ******, *****-***** *********** sees **** * ***** ** * time, ******** *** "******." ** *****-***** codec ******'* ********** ****** **** **** the ****** ** * *****:

IPVM Image

*** ****** ** **** **** ** simple ** ** *************** *** **** significantly ******** *****. *** ********, * 1080p/30fps ***** ****** ***** ** *****-***** compression **** ***** *** **** *** bitrate ******* **** ~**** **/* ** ~40Mb/s. *******, *** ******** ** **** even **** ******* *** ********* ** you ******* ******* ******.

Inter-frame ***********

***** *****-***** ***********, *** **** ** the ***** ****** *** ***** *******, the ***** ******** ******** ****** ** further ******** *** *****. **** ** feasible ******* ***** **** ****** ******* from *** ***** ** *** ****. For *******, ***** *** **** ***** of * ****** ******, *****-***** *********** would **** **** *** *******'* ***.

IPVM Image

**** ** **** ** * ***** scene ********* ******, ******* **** ******* in *** ***** ***** *********** *********/*******. For ********, *** **** *****/***** ****** that ***** **** ****/* **** *****, an *****-***** **** *****, *** **** need ***/* **** *.***, * ***** that **** ****.

*******, *** **** ******** ** *****-***** compression ** **** ** ** *** more *************** *********, ***** *** ******** performance *** ******* ***** (******** *****).

I ** * ******

***** *** *** ****** ***** ***** in *****-***** *********** **** ** *.***/*.***, I *** *.

*-******

*** ***** ***** ** * ***** group ** ******** ** ****** ** I-frame, ***** *** *****-*****, *** ** essentially * **** ***** ** *****, as ******* ** **** ******* ***** in * ******. *** ******** ******* two *-****** ** ******** ** ** I-frame ********, *** (***** ** *****), or *** (***** ** ********).

** ** *******, **** ***** ***** the *-***** ** ** ******* ***** used ** *** ******* (***** *** full ****):

IPVM Image

*-******

*-****** ********* *** **** ***** ** the ******** *-***** ** **** **** the ******* ** *** *****. ******* may ** *****, **** ** ******* noise ** ***** ******* ********, ** large, **** ** * *** ****** moving **** ****** ** ******. *** P ** *-***** ****** *** "**********."

*** ******* **** ** * *-***** from *** **** ***** *** ** seen ** *** ***** ***** (***** for **** ****). **** *** ***** near *** ****, ***** **** ******, and ****** ******* ** *** *****, are **** ** *******.

IPVM Image

******* ***** ******** ** *** **** below, *** *-***** ******* *** ** seen "********" **** **** *****, ***** the **** *-***** ** **** (~** seconds).

Other ***** *****

**** **** ***** *** ***** ***** types ** ******** ** * *** P, **** ** *, **, *** SP, ****** **** *** ********* ****** in ************. **** ** ******* ******* B-frame *******, *** *** *** ***** are ******* ** ******** ******** ****, so **** *** ********* *** ****.

Standard ******

** ** ****, **** *********** *** H.265, ****** ** **** * ***** margin, **** **** ***** ***** *.*** (see*.*** ***** ********** ****). *.*** *** **** ****** **** decreased ** ******** ******** ******* *** improved, ****** *.*** ******* **** ******** intensive, ******* **** ** ****** ***** codec *.*** ** *.***.

***** ** ***** ****** **** ** some ***********, *** ****** ** ********** applications **** ** *** ** ***** analytics. *******, ************* **** ***** ** limit ******* *** *****, ******** ****** *** ********* ** ***** stream.

Proprietary ******

************ *** *** * ******* ** proprietary ******, *** **** *** ** these *** ** *** ** ****. In ***** ****** *******, ***** *** encoded, ****** *** ******* ** *** same ********* (*.*., * ***), ****** it ****** ** *** * *********** codec ** *** ************ ********** *** entire *** ** *** *******.

*******, ** ******* ****** ***** ** themselves *** **** **** ** ******** this ******* ***** ** * ********/*** for ******* *** ******** **. *********** codecs ******** *** ********** ** *******, managing, *** ********** ***** ** **** must **** ** ********** ** *** VMS. *** **** ***** ** ***** this *** ******** *********** ****** **** motivated **** ** ****** ************ ******.

Mobotix ***** *********

*******'********* ****, *** ***** **, *** most ****** ***** *********** ***** **** in ***** ************. ********** ** ****, MxPEG ******** *****-***** ***********, ********* **** MJPEG *** ******** ********* *********** ******** to ** (*** *** *****). *******, ***** ** ** ***********, few *** ***** ***** ******* **, though **** **** ** ******* ** Milestone **** *** ***** *******. ************, H.264's ********* *********** ** ********* **** lower **** *** ********** ***** ******, further ******** *** **************.

Smart ******

** *** **** *** *****, *.*** and *.*** ***** ****** **** ****** common, ****** ** ******* ****** ******** compared ** ******** *.***/*. ******* *** these ***** ****** ******** ******, *** generally **** *** *** **********, ******** below. ******* ****** *** ******** ***** ******** **** ******* ** ***** ************.

Smart ***********

******* ** ******** *** **** *********** level ** *** ****** *****, ***** codecs *********** ****** *********** *** ******** in *** ******'* ***. *** *******, looking ** *** ***** *****, *********** could ** *** ** "***" *** the ******** ****** ** **** ******* high, *** *** ***** ********** *** be *** ** "****", ***** ** do *** **** ******* ** *** white ****.

IPVM Image

Dynamic *-***** ********

******, ***** ****** ********* *********** ****** I-frame ******** ***** ** ******** ** the *****. ** ** * ***** has ****** ** ** ******, *** camera ***** *-****** ************ (*, **, 20 ******* ** ****), *** **** activity ** ******** ** *** *****, it *********** ***** ** *-***** *** switches **** ** ****** (********* * second) *-***** ********* ** **** ** activity *********.

**** ***** **** * ****** ******** shows *** ******* ** ******* *-***** interval:

IPVM Image

Smart ***** ********

** *** *****, ***** ****** **** reduced ******** *************, ~**% ** * minimum, *** ** ** **%+ ** some ******. ***** ****** *** **** effective ** ***** ******, ** *-***** intervals ****** ****** *** *********** ****** due ** *** **** ** ********. See ******** ***** ******** **** *******.

H.264 **. ***** *******

***** *.*** ** ******* *** **** widely **** ***** ** ************, *** many ***** * ****** ****** ******* about ******* **** ******** ** *****, and **** ***** ** ******* *.*** is ********, **** *****. ************ *.*** ** ***** ********* ****, **** ******** **********, *.*** delivers *** **** ******* ******* ** MJPEG.

*******, ***** *** **** ****** **** can ********* *******, *********:

  • ******* **** *********** ******:** ************** ********* *** *** ****, *** ***** will **** ******* ***********. *** ***** ****** ************ *********** ************* **** ******* *** *** ******** manufacturers ****** ***********.
  • **** ********** ******:**** ****** ***** *** ******* ** complex ******, ***** ***** ** **** of ****** - *************, ******, ***.
  • ******* *** ******** *** *****:******* **** *** *** *** **** is *** *** ***, *** ***** will **** ******* ***********.

*******, ** **** *****, **** **** using ******* ********, *.*** *** ********* quality ******* ** ***** ** *** tests.

Scalable ******

**** ****** *** **** ******* * single **********. *** ********, ******** **** a *** ****** ** * *** one ***** ***** ** *.***, ******** either ******** * ********** *** ****** or *********** (*.*. **-**********) *** *** stream ** **** ** ***. *******, in **** ***** *** **** ** change *** ****** ********** ******* ********* a *** ******, *** ********, ** you *** ******* ** * ****** over * ***** ********* ********** (*.*., mobile) ** **** ** ****** ******* size ** ***** *****.

* ******* ***** ** ******, ****** 'scalable', *** ** **** *************, ******* having ** ******* * *** ****** or **-******* ** ******** ******. * scalable ***** *** *********** '****' *** frames ** ********** ****** **** **** out ** * ******. **** ****** pruning ****** ** ********** **** **** as **** ** *********** ********* ********** / *** *** ****** / ****** clients.

***** *** *** **** ***** ******** codecs:

  • *** ** *.*** **** ***********. *** plus **** ** **** ** ******** scalability **** *** ********* ******** ** H.264. *************, **** *** ************* ******* this. **** ********** **** ******* *****-********* (sending ******** ******* ************** ** ********* resolution/frame *****).
  • ********, ***** ** *********** *****, **** the ******** ** ***********. **** ** the ***** **** ******** ************* **** but *** ****** *** ** *** cameras. *** **** ******** ** ********, like *****, ** ******* *********/******* ******** compared ** *.***, **** ********* ****** by *******.

*** *********** *** **** **** ****, with ****** ****** ******** ******* ************ in **** ******* (** ***** ** paper) *** ******* *****. **** *** bitrate ********** ** *.*** ***** ****** (which ******* ****** ** ** *** VMS ***********) *** *.*** ****** **********, SVC ** ******** ** ****** * major ****** ** ************ ******.

Future ******

***** **** *********** ****** *** ********* or *******, ** ** ****** ******** that *** *** ************ **** **** gain **** ******** ** ************.

*** ********, **** ********** ***** ******* bandwidth ******* **** ***** *********** ******, such ********* ******** *** / *******. *******, **** ***** ******* **** camera ************* *** *** ********** ** implement ***** ****** ** ***** ********. Additionally, ***** *** *********** ******* ********** of ***** ******, ***** *** ********** with *** ******** ** ******* *.*** recorders/VMS, *** ********** ** *********** ** add ***** *********** ****** ** **** more ********.

**** **** ********* ******'**** *** ********* ** ************* ** ************, ********** as * *******-**** *********** ** *.*** (which *** ************ *** ******* ****). *******, ***** ****** *** ****** to ** ******** **** *************, **** no ************ ******** ****** ** ***** them, *** **** **** ****** ** traction.

What ****** ** ******?

**** ** ****, *** **** ***** combination *** **** *** ***** ** H.264 **** ***** ***** ******* ** smart ****** ************* ******** ********** ** H.264 **** ******* *********.

*.*** ** *************, *** ** **** cases **** ** ***** *** *** to *** ******* ***********. *******, ** is ******* ******** *** **** ***** used ** ***-**-*** ******/*** ******* **** a ****** ************ (*.*., *****, *********, etc.) ***** ************* ****** *** *********.

***** ******* * ***** *** ********** applications *** ***** *** **** (******* unreasonably) **** **** *****-***** ***********.

[****: **** *** ********** ********* ** 2013, *** ************* **-******* ** ******* technology ******** ** *.*** *** ***** codecs.]

Comments (23)
UM
Undisclosed Manufacturer #1
Nov 30, 2016

Very good guide!

10 points!

(6)
(2)
U
Undisclosed #2
Nov 30, 2016
IPVMU Certified

Great Update!

One question about:

While the change from MJPEG to H.264 resulted in drastic bitrate reductions, often 50-75%, our tests of H.265 show that 15-30% savings over H.264 is more likely.

Was the change usually from MJPEG to h.264 directly, or was there often a baby step of MPEG-4 (Part 2) involved? If so, how big was the change from MPEG-4 to h.264?

JH
John Honovich
Nov 30, 2016
IPVM

Was the change usually from MJPEG to h.264 directly, or was there often a baby step of MPEG-4 (Part 2) involved?

For MP, the move was straight from MJPEG to H.264. See 2008 article - H.264 makes Megapixel go Mainstream.

Avatar
Sergey Bystrov
Jan 04, 2019
NetworkOptix

how big was the change from MPEG-4 to h.264?

How big the change depends on actual codec implementations and scenes. So it's always a challenging question. In the entertainment industry, assuming very good implementation of codecs in both cases ~40%. Practically it always depends...

 

However, H.264 has 2 very sufficient differences vs MPEG4(part2):


1) In H.264 DCT( discrete cosine transform) is done in integer values. 
In MPEG4(part2)(as well as in MPEG1,MPEG2,H.261,H.263) the DCT was done in float values, and standard did not dictate how accurate should it be. So while encoders used one accuracy(due to the hardware of software specifics), decoders doing inverse DCT used another formula (accuracy), which led to the error accumulation with every next P frame. So, by the end of the GOP(with new I frame) it was sort of re-synchronization. Visually it led to image quality jump with every new I frame(typically every second). You might remember this visual effect of MPEG4, that image changes significantly every second. Not the case with H.264. 

This is the reason why GOP size was very limited in MPEG4(image degradation accumulation). It's not the case in H.264. Practically H.264 GOP size could be very huge( practically unlimited). The only limitation comes from editing software( in order to decode frame N inside the GOP, decoder have to decode all previous frames).

So, H.264 made a huge step forward eliminating GOP size issue and image degradation towards the end of the GOP. 

 

2) deblocking filter became part of the standard. So they have to be universally implemented across all H.264 codecs. Wich also made H.264 stream look the same on any device. 

 

So, H.264 is not only bandwidth saver but solves some fundamental issues of previous standards.  

 

(1)
(4)
UI
Undisclosed Integrator #3
Nov 30, 2016

Excellent article, the inclusion of all the smart codec data is very useful.

(2)
(1)
SM
Sujit Mahapatro
Nov 30, 2016

Excellent article. It will add high value add for users. Grate Job.

(1)
(1)
(1)
Jp
Jeffrey pope
Feb 27, 2018
IPVMU Certified

Sounds like there is always room for improvement on H. 264, but it has proven to be the foundation in which to build on. The H. 265 sounds good at the drawing board, but when put into practice, it has its "ghosts" to be worked out.

UE
Undisclosed End User #4
May 20, 2018

I have a better understanding of compression

Avatar
John Dornik
May 21, 2018
Integrated Access Security • IPVMU Certified

Thanks for taking the mystery out of compression!

Avatar
Sergey Bystrov
Jan 03, 2019
NetworkOptix

Good article. 

Few points:

P-frames reference the full image of the previous I-frame to send only the changes in the scene. 

Just in case: P frame reference the full image of any I or P previous frames.  Advanced codecs can refer to multiple frames back minimizing the difference(part of the image could be taken from one reference frame, another part form another). Practically P frame most of the time refers previous P frame.  


 

MISC: as a paradox, typically in the surveillance industry(due to static scenes)  H.264 takes less CPU to decode than MJPEG. (since most of the time most of the image parts difference with prev frame is about zero). 

 

H.265 is designed in such a way that besides potential bandwidth savings it's better parallelized for multiple CPU cores to encode/decode. 

(4)
Avatar
Joel Kriener
Jan 03, 2019
IPVMU Certified

Thank you for the excellent summary and outline to explain CODECS. Well done!

AS
Alfredo Santiago
Feb 07, 2019
ALFREDO'S TELEPHONE SERVICE

With "Inter Framing" if I set my HD to record on motion is there further compression or are they separate?

JH
John Honovich
Feb 07, 2019
IPVM

Motion based recording does not 'further compress' video, it simply determines when to record video or not at all.

Avatar
Luca Fogliati
Feb 27, 2019

I would like to know more about SVC.I am completely ignorant on the topic.Can it be used also to obtain rtsp streams or does it work only with a server/vms that supports the function?

JH
John Honovich
Feb 27, 2019
IPVM

Luca, we know no video surveillance manufacturer that offers a scalable video codec today. It's effectively not used.

Today, it's almost always H.264, H.265 combined with smart codecs, no svc.

Avatar
Luca Fogliati
Feb 27, 2019

Actually UNV allows to enable SVC in the video page.Checking around IPVM i've seen also tests you did with Vivotek

 

JH
John Honovich
Feb 27, 2019
IPVM

My mistake to say none are 'offered'. 

My point is none are used. For example, we have never heard Uniview nor Vivotek recommend using SVC nor any VMSes do so for those cameras.

We are not against SVC, there is just no sign that anyone on the manufacturing side is promoting or recommending using it.

TC
Trisha (Chris' wife) Dearing
Feb 27, 2019
IPVMU Certified

My point is none are used.

Hikvision, Dahua and Vivotek all have a SVC-T setting exposed, that works with at least their own recorders. Of course, this is actually enabled because of the underlying SOC supporting it, as you once mentioned: 

Does Hikvision SVC Really Work?  

On the other hand, neither you nor I consider this a full implementation of SVC, because it is only frame-rate reduction based, temporal -T.  The h.26x SVC we were all waiting for, one with spatial reduction -S, (ala JPEG2000) never arrived: 

SVC - A Better H.264 Coming For Video Surveillance

 

Avatar
Quincy White
Mar 01, 2020
H. Stephen Jones and Associates

Excellent article. It was good to see and understand how they test to determine what makes one better then the other. Great Job!!

AS
Amitkumar Sirjusingh
Jan 02, 2021

H.264 remains the standard in video surveillance codecs notwithstanding H.265 and other codec introductions such as Google VP8 and VP9. Codecs play a vital function in achieving efficiency when it comes to bandwidth utilization.

UM
Undisclosed Manufacturer #5
Jan 13, 2022

With the advent of smart codecs does using event-based recording instead of continuous buy much these days?

JH
John Honovich
Jan 14, 2022
IPVM

#5, I ran a poll, see the results:

IPVM Image

This is, of course, informal, though minimally it is safe to say that many are still using event based recording. That said, as you mention smart codecs (plus declining hard drive costs) makes event-based far less needed than years ago.

(1)
JO
Jesse Osterhout
Jun 21, 2023

Very informative read!