Subscriber Discussion

Does Disk Fragmentation Exist In Nvrs?

UD
Undisclosed Distributor #1
Feb 13, 2017

NVR's are designed to record in blocks leading to disc fragmentation play back slowing and eventually performing badly. Is this true?

(1)
Avatar
Josh Hendricks
Feb 13, 2017
Milestone Systems

The answer is very much "it depends".

I cannot speak for other VMS's, but Milestone products pre-allocate files or "blocks" to record to. This seems to prevent file-fragmentation related performance loss. To be honest I haven't personally delved too deep into this topic, but I can tell you in over 10 years I have never said "You need to defrag your drive" in response to a performance problem.

If an application is not designed to pre-allocate files and instead appends to them until some pre-defined time/size, then unless they're doing something to ensure additional file allocations are contiguous, this will definitely result in fragmentation and potential disk performance issues.

This is low-hanging fruit for VMS developers so I would be surprised if any popular VMS software didn't make design decisions to minimize fragmentation.

(1)
(1)
Avatar
Josh Hendricks
Feb 13, 2017
Milestone Systems

I just realized you specifically mentioned "NVR's", but my response is the same - NVR's are simply servers/appliances with VMS software on them.

(1)
U
Undisclosed #2
Feb 13, 2017
IPVMU Certified

... my response is the same - NVR's are simply servers/appliances with VMS software on them.

Disagree.  

Milestone NVRs do not benefit from defraging, but that is only because MSSQL handles the allocation of raw database chunks from the OS.

IMHO, many, if not most NVRs, use the OS filesystem to store video, which could theorectically benefit from defrag.  

(2)
(1)
Avatar
Josh Hendricks
Feb 13, 2017
Milestone Systems

Agree to disagree :)

Milestone software, and NVR's do not, and have never used Microsoft SQL or any other SQL-style database for storage of video/audio. We use our own file-based database and pre-allocate block files for storing media data.

We use MSSQL for storage of configuration, audit logs, events/alarms, maps, etc. But SQL was never used for storing video because, at least at the time (late 90's), it was decided SQL was not fast enough.

As I said, I can't speak for other VMS/NVR vendors, but this strategy isn't revolutionary IMO, so I would not expect any other major players overlooked it.

(1)
(2)
U
Undisclosed #2
Feb 13, 2017
IPVMU Certified

My bad.

Thanks for clarifying.

U
Undisclosed #2
Feb 14, 2017
IPVMU Certified

A couple of questions, if you don't mind on

We use our own file-based database and pre-allocate block files for storing media data.

Are these files allocated by the OS from an NTFS partition?

Do you create them only on database initialization, or are new files created and deleted as the video is recorded?

Avatar
Josh Hendricks
Feb 14, 2017
Milestone Systems

The filesystem would be NTFS, and I don't believe we do any super low-level / raw storage manipulation so yes allocations would be done through the standard methods available in STDIO most likely.

Each block file is allocated as needed. For example I think XProtect Professional allocates in 2.5MB chunks. If video prebuffering is enabled in this version, then video is cached/deleted from this allocated space.

In XProtect Corporate, we now support prebuffering to RAM so there are no IOs except to store data.

U
Undisclosed #2
Feb 14, 2017
IPVMU Certified

...allocations would be done through the standard methods available in STDIO most likely.

In that case, since the OS is allocating the space to a standard file system as needed, one would typically expect fragmentation of the files over time, especially when the storage is full or nearly so.

Have you ever brought the defrag analyzer on a drive?

Whether the fragmentation would actually affect performance much is another question altogether as there are many variables to consider...

 

(1)
Avatar
Josh Hendricks
Feb 14, 2017
Milestone Systems

I'm not a storage/filesystem expert, nor do I have code-level knowledge of our media databases. It's certainly possible that files could get fragmented. But generally unless a disk is already fragmented, new file allocations are contiguous. 

It's common for a file to get fragmented as it grows. For example, on one of my test systems, one of the Windows log files has nearly 10k fragments.

In contrast, the live database drive currently has ~2500 files and only 9 of them are fragmented with the most fragmented file in 5 pieces. So it is possible for fragmentation to occur, but by pre-allocating files on a healthy filesystem, the fragmentation is greatly minimized.

(1)
U
Undisclosed #2
Feb 14, 2017
IPVMU Certified

but by pre-allocating files on a healthy filesystem, fragmentation is greatly minimized.

True, but you also said

Each block file is allocated as needed. For example I think XProtect Professional allocates in 2.5MB chunks.

Avatar
Josh Hendricks
Feb 14, 2017
Milestone Systems

Yes, we allocate a 2.5 MB file, and when it's full we allocate a new 2.5 MB file. Assuming there is not already significant fragmentation, those two files should be relatively contiguous.

They are unlikely to line up on the disk one after the other, unless maybe you are only recording from one camera. But as data is laid down on disk, on a healthy system most data from a certain time period should be in roughly the same area. The files themselves would typically be unfragmented or lightly fragmented at most.

U
Undisclosed #2
Feb 15, 2017
IPVMU Certified

When are the files deleted?

Avatar
Josh Hendricks
Feb 15, 2017
Milestone Systems

When the data inside them has reached the configured retention time. It's more complex than that since most systems perform archiving, so data might sit on the live drive for as short as an hour or as long as the retention time depending on the product version and configuration.

When data is archived, there's a good chance the final files are unfragmented if they were fragmented to begin with.

In the event a low disk space threshold is met, data may be deleted prior to the configured retention time. Data is typically grouped in "tables" with at least an hour of data each. There must be at least enough space available for the data to be archived, so in most cases there should not be attempts to "squeeze" data into the gaps of a fragmented file system.

UM
Undisclosed Manufacturer #3
Feb 14, 2017

I think "Disk fragmentation" is a disk (File system) issue, right ?

JH
John Honovich
Feb 13, 2017
IPVM

I cannot speak for other VMS's, but Milestone products pre-allocate files or "blocks" to record to.

Joshua, does that apply for software only or for just Milestone's NVRs like Husky?

Avatar
Josh Hendricks
Feb 13, 2017
Milestone Systems

This applies to all Milestone products - Advanced VMS code base, Professional VMS code base, and all Husky NVR's since they are running the same code bases.

Even the Husky M10 running linux will behave this way as the media database is based on the Advanced VMS products.

There are significant differences in the structure and operation of the media databases between XProtect Professional and XProtect Corporate, but both are strategically pre-allocating disk space for recordings.

(2)
Avatar
Murat Altu
Feb 28, 2017
AxxonSoft

What Joshua saying not relates to fragmentation. Milestone uses stdio, and it means that OS is managing real place of each block. One file never means one solid block of storage. To manage blocks you need to unmount drive from OS and than have code inside VMS which manages sectors of hard drive directly without using OS!

And this way is implemented only in AxxonNext. I don't know any other VMS who implemented their own file system to completely prevent fragmentation. In AxxonSoft we record sector by sector like reel-2-reel tape and this is the only 100% guarantee of no fragmentation.

(1)
(1)
U
Undisclosed #2
Feb 28, 2017
IPVMU Certified

One file never means one solid block of storage.

What about when the files are small, e.g. less than the block size, or big, e.g. as big as the whole device?

Avatar
Murat Altu
Feb 28, 2017
AxxonSoft

When files are small, each file will be in different phisical place, so fragmentation is there. When file is the whole drive, than you need to develop kind of file system to manage data inside this file and this is the same as managing whole drive, but less efficient.

UM
Undisclosed Manufacturer #3
Mar 01, 2017

No need to develop a new file system. Use XFS instead of windows file system...
Even your VMS's database somehow never delete or create files.
But Windows itself will do this. " temporary files" always doing this.
Administrator need to do a lot of things to prevent files fragmentation (Easy to reduce but not prevent)

(1)
Avatar
Murat Altu
Mar 01, 2017
AxxonSoft

XFS can reduce, but not prevent. And nothing can be comparable in performance with permanent sector by sector recording.

UM
Undisclosed Manufacturer #3
Mar 01, 2017

It will reduce a lot fragmentation by belayed allocation.
And I think XFS can do the "sector by sector" magic too.
XFS can find contiguous block to write a new file, right ?

Avatar
Murat Altu
Mar 01, 2017
AxxonSoft

Yes, but no guarantee that all files in a sequence follows each other. And when you start deleting files (cycle recording) defragmentation occurs. The main trick in our FS is that we never delete files and do true cycle recording sector by sector in whole drive scale, not only in one file.

(1)
UM
Undisclosed Manufacturer #3
Mar 01, 2017

So you mean Axxon can prevent directory fragmentation? It's always a weak spot for IT industry.
But it should be easy to keep it clear if VMS won't delete files.
And Axxon must using RAW disk for recording? If not. How can this prevent some situation like using C Drive for recording?
Any more information about this technology?

On the other hand. Why fragmentation affect to CCTV storage ?
In IT industry. It's always has over thousand users access to the storage and do different things. (Read, Write, Delete, Move)
That's why fragmentation can impact on performance.

What do you think of it ? Marketing scam or real deal ?

Avatar
Murat Altu
Mar 01, 2017
AxxonSoft

It is a real deal, not marketing scam. From marketing point, this feature is really difficult to promote, because not many people understands the problem.

Fragmentation always impact on performance. Typically you want to utilize all performance of hard drive for recording. But when fragmented, recording become much slower. Typically, after some time you can find that your recordings are not 25/30fps anymore. And lifetime of your hard discs smaller when fragmented.

AxxonNext SolidStore guarantees that recording performance is always at max hardware level. For this you need to unmount drive from Windows (you'll not see it as a letter D or so) and then Axxon will offer to use this partition for recording.

UM
Undisclosed Manufacturer #3
Mar 02, 2017

Not so difficult to promote if your listener is IT admin.
Good to know that SolidStore is really using RAW disk to do it.
Can it mount to a iSCSI disk ? and what's the maximum capacity ?

It's nice to have it with free version.
Thanks for your information.

Avatar
Murat Altu
Mar 15, 2017
AxxonSoft

Yes! When iSCSC disc is mounted as raw partition, Axxon offers to use SolidStore on it!

I don't think that we have any limit in size of partition. I'll check if any and let you know.

UM
Undisclosed Manufacturer #3
Mar 15, 2017

It's really as good as you told.
Nice to see some awesome software in the market.
Using a single file to avoid fragment even recording in Drive C. Great! (But I'm a little concern about the file corruption.)

The best part is cameras can recording to multi partitions/drives with different FPS, stream and retention period.

I know some of other VMS can do this but they always need additional license.

Can I ask you one little question? How can I display a single camera in "Really" full screen?
Thanks!

Avatar
Murat Altu
Mar 15, 2017
AxxonSoft

File corruption not an issue, because there is no file. This is another benefit of SolidStore. It is sector by sector recording without any file tables or other things. Only exact sector of data will be corrupted and in this case you loose only data from that sector.

 

In AxxonNext you also can create any number of archives and configure recording to each of them with different parameters and conditions. Like you can have 1fps 24hours low stream and 25fps detector based high stream. And no need for additional license.

 

Regarding your question: Yeah...we had this request. Next month v4.1 is coming and there you can zoom "really" full screen!

Any other wish ? We'll be happy to fulfill all expectations :)

UM
Undisclosed Manufacturer #3
Mar 16, 2017

Just a little suggestions.
1. Incremental search in "Layout" tab should be included layouts and maps.
2. Add public/private access right option in maps and layouts.
3. Exported EXE player should be add some features. At least pause the video, fast forward/backward, re-export the video.

My personal top VMS for now (listed in no particular order) :
Security Center, Milestone, Axxon, ACC5

But I need to say something to your UI programmer :
Why are you hate Paul Fitt so much ??

Avatar
Murat Altu
Mar 16, 2017
AxxonSoft

1. Good idea. We'll implement it.

2. Currently rights are connected to exact users and doesn't matter they are private or public. Let's discuss this in details by email.

3. Done. In v4.1 embedded EXE player is very powerful with great usability and features.

Regarding usability: we are very proud with usability of monitoring and control, but still have some to improve in configuring. v4.1 is good step forward with automatic configuring of embedded detectors and motion recording during adding cameras, applying single parameter for group of cameras and so on. Hope you'll like it :)

U
Undisclosed #2
Mar 16, 2017
IPVMU Certified

Why are you hate Paul Fitt so much ??

I love big buttons and I cannot lie...

(1)
Avatar
Murat Altu
Mar 16, 2017
AxxonSoft

Not for VMS case...we want to see video or buttons :)

U
Undisclosed #4
Mar 17, 2017

Murat, I don't get it

If you don't have file tables or whatever

How do you index video for searching etc...

Can you please explain

Thanks

Avatar
Murat Altu
Mar 17, 2017
AxxonSoft

During recording, archive distributes index blocks throughout the whole drive in a way to quickly and effectively read them at start. In case of single index block corruption, archive quickly recovers it by reading very small part of drive.

U
Undisclosed #4
Mar 18, 2017

Thanks for info

so, the way I understand you do have your own file system

Avatar
Murat Altu
Mar 18, 2017
AxxonSoft

Yes, because this is the only way to eliminate defragmentation.

UM
Undisclosed Manufacturer #3
Feb 14, 2017

Tested some of those embedded Linux NVR are using Ext2 or Ext3.
So the answer is no.

AM
Alastair McLeod
Feb 28, 2017

Veracity's COLDSTORE surveillance storage system features a speficially designed Sequential Filing System or SFS which bypasses the OS and records all video in a sequential manner.  It is done this way for two reasons: to eliminate vibration (and therefore wear) on the drive and also to optimise write and read speeds.  

Murat is correct that unless you bypass the OS, the OS will decide how the files are written to the disk. File-block allocation helps a little, but by no means does it eliminate fragmentation.  However, the key issue is not fragmentation, but the rapid motion of the read/write head, which causes vibration and wear and can affect other drives.  All conventional disk filing systems use directory areas and data areas and so the disk read/write head is always skipping between them.  Naturally this decreases total throughput and performance.

The SFS does not have a directory as all the data is stored sequentially.  You can only do this with sequential data.  However, security data (video, audio, alarms, ACS events etc.) are all generated in real time and so are effectively sequential.

This is the key reason why surveillance storage systems should be designed specially for video and other real-time data.

 

(2)
U
Undisclosed #2
Feb 28, 2017
IPVMU Certified

The SFS does not have a directory as all the data is stored sequentially.

Is each stream stored sequentially and contiguously or multiple streams interleaved?

AM
Alastair McLeod
Feb 28, 2017

Multiple streams are interleaved.  It is actually quite a sophisticated system and the development took considerable effort and ingenuity on the part of our engineers.  However, it has been around for a number of years now and is very well proven.

(1)
Avatar
Murat Altu
Feb 28, 2017
AxxonSoft

Great to know it! So, not only Axxon...my respect to Veracity!

BTW, we call our file system SolidStore...quite similar to ColdStore :)

And yes: it increase hard disc life, recording speed and prevent possible losses of big amount of data, because we don't need partition table - point of terrible failure.

(1)
UM
Undisclosed Manufacturer #3
Mar 01, 2017

REALLY ? SFS can eliminate vibration ? Don't tell me you test it by SSD.
Did you mean with the same disk enclosure, same cooling system, same HDD, same VMS. SFS array can do less vibration then RAID5/6? (Let you slide on "eliminate")
What range of your vibration measurement ? (5Hz ? 200KHz ?)
What the vibration period with the same data writing ?

SFS allow how many disks failure?

AM
Alastair McLeod
Mar 01, 2017

Yes, really.  We eliminate vibration caused by the read/write head jumping back and forward between the directory and data areas.  The SFS makes the read/write head work much like the needle on an old-fashioned vinyl record player.  It starts at the outside of the disk and very slowly moves inwards as the disk fills up.  So the read/write head is virtually motionless for most of the time, except when jumping back to read data already written (for playback). 

Other sources of vibrations will still be present (fans, spindle motor) but are much less of an issue, especially with our system, which is specifically designed for purpose (very low power dissipation, very low speed fans).  Further, SFS is only the disk filing system part.  The whole sequential storage concept includes a special method of writing across the array.  This is called LAID or Linear Array of Idle Disks.  The system writes in a mirrored overlapping-pair pattern, providing mirrored writing security during the all-important writing process.  All other drives are switched off unless they are the writing pair or are being accessed for playback.

I would be happy to describe the entire system concept in full to you outside of this forum, as we are in danger of drifting off-topic, which was to do with disk fragmentation.  If you wish to discuss, I'm sure the moderator will put you in touch with me.  Thanks.

U
Undisclosed #2
Mar 01, 2017
IPVMU Certified

This is called LAID or Linear Array of Idle Disks.

After some hesitation, I finally got LAID and couldn't be happier!

(3)
AM
Alastair McLeod
Mar 02, 2017

Glad to be of service.  (I think)  :-)

U
Undisclosed #5
Mar 16, 2017

tough crowd.

UM
Undisclosed Manufacturer #3
Mar 02, 2017
SFS maybe can make the HDD head move less then other file system. But how about the spindle? This is the main reason of HDD vibration. Let say HDD head won't move but spindle will. (Unless the disk is not be reading or writing) Spindle and platters are much more heavier than HDD head. It's make a lot of vibration too. SSD can eliminate vibration . Even SD card, USB thumb drive can. But not HDDs. If anybody really can eliminate HDDs vibration. I think HDD can develop to a whole new level. Maybe 20 platters in a disk ? LAID has mirror to the new data only. So it assume disks can't be failure after switched off ?
AM
Alastair McLeod
Mar 02, 2017

Modern disk drives are incredibly well balanced and so the drive motors and spindle do not actually produce much vibration.  In any case, with the LAID writing pattern, all the drives (except the pair being written to) are switched off, so the vibration from those is in fact zero.

Yes, the LAID system has mirrored data for the disks that are being written to.  we exploit the fact that disks very, very rarely fail when being read from, as this is a purely passive process.  I'm sure we have all experienced a disk failure in a desktop or laptop PC at start-up time, but in fact loading a modern OS (such as Windows) involves a huge amount of read-write processing and so is not a purely passive process.  The LAID system within COLDSTORE has been deployed for many years - in fact the oldest installations are reaching 7 years old now, still with 99% of their original drives, still working fine and proving the principle above.  There is an option to store full mirrored pairs too, which of course uses twice the number of drives, but as the drives do not need to be RAID or Enterprise-level drives, they are considerably cheaper to buy. 

Further, as soon as SSD drives become more price-competitive with HDDs at high capacities, we will switch to them.  The SFS will be less important (although writing large blocks strictly sequentially will still increase the lifetime of the SSDs), but the writing pattern will still be totally valid, as SSDs fail just like any storage technology.  The mechanism and nature of SSD failure is different from that of HDD (and well documented elsewhere), but the point is that our patented SFS and LAID techniques will still be advantageous for SSD and will still help to lower the total cost of ownership by increased reliability, lower maintenance and much lower power consumption.

(1)
Avatar
Mike Dotson
Feb 28, 2017
Formerly of Seneca • IPVMU Certified

When I am testing systems in the lab and I suspect the target is 'fragged'.... I use a tool called Auslogics Disk Defrag to get a view.

It also lets you actually run a defrag operation.

(1)
U
Undisclosed #4
Feb 28, 2017

Mike have a look

 DiskExplorer for NTFS V4.32

This unique disk editor gives you low-level access to your hard drive. Switch between several views, such as hex, text, directory, MFT entries, partition table, boot record.

 

Avatar
Mike Dotson
Feb 28, 2017
Formerly of Seneca • IPVMU Certified

I see this is a try-before-buy tool.  How long does it let you use it in practice?

FYI...the one I mentioned is free.

U
Undisclosed #4
Feb 28, 2017

I have much older version

no idea about new one :(

UD
Undisclosed Distributor #6
Feb 28, 2017

Yes NVR's get fragmented, however, Uniview has UBS Block Storage which increases the amount of space you can store and eliminates fragmentation.

 

U-Code and UBS have been some of the few great software innovations to come out of Chinese software/firmware that are unique to one company and not immediately available to every single seller out there.

U
Undisclosed #5
Mar 16, 2017

Wow, this is how it was done in the 20 century. Nowadays we R/W all data at the electron level. No need to scale as storage is infinite.  - Buzz Lightyear 2101

New discussion

Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.

Newest discussions