Advice Needed On A Challenging 10,000 Camera Project

Live video only ? Adobe media server should work.

What's the WAN topology/capacity?

I have put in a city-wide system using Exacqvision Enterprise. Not 10,000 cameras, but probably 1000. Segregated VLAN on the fiber (and microwave) to bring camera traffic back to the servers. I run a separate server running Exacqvision Web Service, that basically connects like a client to all the servers. Secondary NIC on this server runs to a designated outbound/public connection (SSL enabled). Users (probably 50) are grouped in Exacq based on rights and camera access. They can access via browser or mobile app. Dedicated client workstations (dispatch, police) run unrestricted on the camera/security vlan for Gigabit live viewing and control (PTZ and search).

This has been running since 2008 over the fiber (First Exacq server actually installed early 2007 running on a city mesh network-SkyPilot 4.9 I installed), with few outages and minimal problems. (Growth, including new sites, remote wireless sites, and clients probably 250 cameras in 2008 to 1000 2016). This is about 30 sites (schools, parking garages, city halls, libraries, fire stations, police stations, ema, etc) plus maybe 25 remote wireless PTZ cameras.

Exacq has worked out well for this and I believe the Web Service may address what you are trying to do.

Hi Gavin, here's two new, promising alternatives that a number of cities are considering for large scale deployments. Keep in mind that if you literally need to deploy a private network within a relatively short period (several months), the following is most likely not suitable and you'll need to architect a COTS solution with infrastructure providers for example like Avaya (managed switches) and Siklu (wireless).

If the clients are largely mobile, one solution that has moved from "emerging" to "go to market" is Mobile Edge Computing. MEC servers are deployed on generic computing platforms within a Radio Access Network (RAN) like 4G and the more promising 5G. In short the design offers an improved user experience based on low latency, high bandwidth content, real time network status, location-based services and the need for private, non-shared content.

The other consideration is based on whether the city has a local PBS station that can take advantage of unused radio spectrum. This has been in successful pilots in cities like Houston and includes a managed, encrypted private network often used for first responders.

If any of this sounds interesting, I have plenty of documentation on all three.

Best/Steve Surfaro/Axis Communications

The First thing you should consider is License or No license? You could easily save 1 Million dollars for un-needed fees.

What kind of cameras are going in (PTZ/Fixed, all one manufacturer/multiple, ONVIF/non-ONVIF, etc)?

If you are going to be going over a public network than multicast may not be supported by the ISP in the area so finding that out will be important.

With you not being able to dictate that you clients use the same VMS as you, will they be adding these "public" cameras to their systems or viewing them independently from their own cameras?

This sounds especially challenging as a result of the "users" need to use their own VMS with the same cameras. This almost necessitates that you have some kind of system in the middle which presents either a simple RTSP stream or maybe full ONVIF "simulation" of the devices on the other side. But you would have to be able to restrict access to a specific stream since multiple users requesting different resolutions or frame rates will quickly either overwelm the camera or result in user A's stream request resulting in a change to user B's existing stream.

I work for Milestone and we have a product called ONVIF Bridge (tested by IPVM: https://ipvm.com/reports/milestone-bridge-test)

The ONVIF Bridge server is a free add-on which works in coordination with our VMS. It is not fully ONVIF compliant as it presents only the bare minimum services to retrieve a stream or previously recorded sequences and does not allow changing settings of the camera via ONVIF. It did receive a recent update though, and if the users can't use the ONVIF interface, they may be able to still use the raw RTSP streams as most VMS's allow this.

This would allow you to have the ONVIF Bridge server on the public side, and the VMS/cameras on the private side of the network which would be a good layer of abstraction to prevent multiple users using their own VMS from interfering with each others streams.

I'll follow this thread with curiosity, and best of luck to you on this project regardless of the product stack you wind up with.

This may be a case for a PSIM which can support disparate VMS solutions on a common operating platform.

This is an interesting question.

You write: 'I can't expose the camera stream directly to the client (security, bandwidth, etc.)'. What exactly do you mean by 'expose directly'?

The first idea that comes to my (simple) mind is to multicast a limited number of streams of different quality from each camera and let the clients subscribe directly to the camera streams they need. Do you consider this 'direct exposure' and if so, what are the bandwidth / security issues? In one of your answers you write that you install, support, maintain and connect the cameras, so I would expect that you have full control over bandwidth and security.

Sorry, for clarity. I intend to make a single stream form each camera available (let's call it a 3MP / 12 fps / H.264 stream).

If I give my client the IP address of the camera, then each client will consume a unicast stream - not viable. I'd also prefer to keep my "possibly insecure" client networks off my "rather more secure" camera network.

I agree, if I use multicast then each client can tap into that stream and I don't multiply my b/w consumption.

Now grab a calculator and assume I need 10,000 multicast streams. Even in sparse-mode where I extend only the streams my client needs out into their network/control room, my network still has to manage some 56Gbps of multicast content - those numbers make Cisco and Juniper squirm.

Now I get into a big network segmentation exercise, and complex IGMP and routing challenges.

Also, where to I generate the multicast content from? The camera? What cameras provide a multicast source at a reasonable price point?

What quality will I sacrifice using a UDP source form the camera all the way to my client's control room? Everything you read suggests sticking with TCP for good quality, reliable footage.

Thus my question, is this a VMS solution or a networking one. (BOTH is the obvious answer).

Some thoughts

A high capacity, reliable, general purpose proxy, like Squid for example, could bridge from your camera network to your client network. It can proxy media streams, so you could proxy from unicast on your camera network to multi-cast on your client network.

You won't have much in the way of licence fees if you create an open source solution, and you can engage people with lots of experience in high performance computing platforms. Open source though, is not going to vendor sell itself to you, even if it is the best solution. You have to choose it and chase it down.

100Gbit line cards are becoming common on mid range cisco, so that scale of bandwidth is doable.

You're going to need something like puppet or ansible to push out config updates to your camera fleet.

Your dhcp, dns, snmp & ntp servers will need to be carefully configured, you'll need real tools, not toys. If you get them all to work together nicely you'll have better uptime statistics.

You'll need traffic probes at strategic points in your network, so you can see what's going on with dodgy camera configs, misconfigured network elements, unintended consequences.

Clients will need to limit the camera feeds they choose to their available incoming bandwith. Even if you go multicast, the last mile to the client carries the same amount of traffic as unicast.

You might end up going UDP, simply because the ack & retrans request traffic will kill your network the moment you have congestion. It will make a bad situation a lot worse, rather than just gracefully dropping a few frames as you approach network limits.

If your clients love your project, they will buy dark fibre from their data centre to yours and light it up with 10Gbit or 100Gbit gear, since they are likely in the same city. Then you'll need more intermediation servers, as they consume a greater percentage of your available feeds.

Half your clients will be public entities of some sort. If your regional and national government is smart about how it spends your tax dollars, it will direct them all to collaborate, then they'll ask you to install the VMS, and they will then just consume finished and ready to use video walls & history searches.

I've been deploying NX Witness at micro scale. It seems very efficient, and their people innovative. Because it runs well on Linux, it could probably scale to a substantially larger cluster than any Windoz platform VMS. It also virtualises well on Proxmox or OpenStack, so you could scale out with automation. Could be a project NX would adapt their software for. It would certainly need some more meta tools.

Good luck, exciting project.

I can share what I know about Moscow Safe City project. The project had more than 50,000 cameras. Separate buildings, like Schools, were connected by Fiber optic channels to monitoring nodes. Nodes were connected to monitoring centers. Pensioners were hired to do monitoring in the monitoring centers. As for VMS, as far as I know were used VMS of different manufacturers. One of them was AxxonSoft.

Pensioners were hired to do monitoring in the monitoring centers. As for VMS, as far as I know were used VMS of different manufacturers. One of them was AxxonSoft.

To do camera layout I recommend using a video surveillance design software with 3D modeling capabilities. I don't like to toot my own horn, but I recommend our IP Video System Design Tool that helped to design some parts of Moscow Safe City project.

The project started almost 8 years ago and the typical camera resolutions were quite low. Also there were discussions that it would be better to use a frame based MJPEG video compression in compare with stream based codecs, like MPEG4.

so the situation is take one feed from camera (RTSP) and distribute it using a mid layer software to multiple VMSes that your clients use through RTSP.

Might be worth investigating WOWZA. Not used it myself, but this looks relevant:

How to re-stream video from an IP camera (RTSP/RTP re-streaming)

So that would output an RTSP stream, and all VMSs will be able to connect to it.

I'd also look at setting up two streams to each camera, and re-stream both with a different RTSP URL, allowing client VMSs to make use of dual streaming (I am assuming VMSs allow a different RTSP URL for each stream - not sure how many do).

Should you re-stream as ONVIF or just RTSP? If you are not using PTZ and do not care about events from the camera (motion events etc.), then there may not be any real advantage in using ONVIF over RTSP. ONVIF is RTSP but with additional XML packets to configure the stream (resolution, frame rate, GOP etc). I am guessing the streams you are exposing would have a pre-configured resolution and frame rate, (since they just forwarding from the camera). By using just RTSP, you may have more (and cheaper) options than using an ONVIF bridge.

So, basically, we're setting up for Watchdogs. Fun.

I think this boils down to cold hard math and distributing support for these massive, astounding numbers.
Logistically, on the minutia of camera level now (which really is just petty details at this point, but every bit you can shave off the numbers, the better) the more you can have h.265+ and multimager/fisheye panoramic cameras.

This is about minimizing the necessary connections, bandwidth, storage, etc.

You still need scalability for future upgrades and to compensate for the fact that a landscape of a city hardly ever consistent in terms of surveillance needs.

However, this all comes secondary to the networking. In fact, we should barely even think about the surveillance or access control aspects before we finish tackling the node distribution and who, where, and how you'll be housing everything. The sheer number of hard drives you'd need for this alone would make your head spin and you can bet your ass you'll want WD Gold 10TBs (FYI, try to get a price of less than $520 from Ingram Micro for those if you decide to swing that way) at a minimum. The server room design alone for that will be a feat of engineering.

You're basically trying to build a backplane for a technologically advanced private municipal network. Everything has to work and stay working. The cameras and their systems are literally the least of your concern as you can swap them out for any number of options.

I definitely do not recommend a purely centralized solution. There will need to be some edge options and nodal options to relieve strain and maintain stability.

1) what resolution, fps and bit rate are the source streams
2) how many users will access a single stream
3) where and what kind of infrastructure is available for the redistribution software
4) will the source and destination (client VMS or users accessing) streams be RTSP

Actually, as a follow-up, this should be a very strict order of operations in terms of prioritization:

1.a) Network Stability & Scalability (and all logistical prerequisites to accomplish this)

1.b) Cyber Security, Cyber Security, CYBER SECURITY!!!

2) Physical Security/Surveillance

I have created some Interfaces that maybe helpful in cases like this. I have asked for feedback about these Interfaces here:

https://ipvm.com/forums/video-surveillance/topics/looking-for-feedback-on-my-unique-interfaces

The Interfaces are totally secure and can use HTTP or HTTPS even when the IP Cameras are using only HTTP. No information is exposed about any IP Camera and the Interfaces work with any Internet  browser capable devices that are using any browser. From Computers to Tablets to Phones and TV's. Without the use of any Plug-In's or media players.

You can display as many IP Cameras using different brands and models as you wish on the same page and you can use IP Camera controls or simply display the IP Camera on a per IP Camera basis. You can force a login or not require any login. Your choice. The Interfaces use snapshots so you can also can control the FPS ("Frames Per Second") on a per IP Camera basis.

The link above also has live demonstrations so that you can get an idea of how the Interfaces work in real-life. Please leave any feedback or post questions about the Interfaces at the link above.

Don

1. The network is reliable.
2. Latency is zero.
3. Bandwidth is infinite.
4. The network is secure.
5. Topology doesn't change.
6. There is one administrator.
7. Transport cost is zero.
8. The network is homogeneous.

If you'd like to learn more, please reach out to me via LinkedIn

Michael von Hauff

CTO

Osprey Informatics

What about CNL or Vidsys?

CNL did a large scale citywide implantation in Atlanta that sounds very similar to this. 

Hi: Gavin 

 (1) Have you considered the network multicast/network solution? NOT the VMS,

       This is a distribution platform, at the head-end site say MUX all 10,000 streams as

       few Transport Streams (TS) then through CDN distributes TS to multiple clients site.

(2) At each client site, extracts the TS (De-MUX)/decode the streams either for

     recording or view......

     We have the H265/264 live trans-coder/encoder/decoder equipment, which can

     reduce the total network bandwidth by 50% if you are  interested, I can send you 

     more info.

     by the way, at the head-end there is NO need to store just monitors all 10,000

     cameras signal is coming or NOT, once it discontinues then you have the alarm and

     know what's wrong of the camera.     

Okay, I see generally we're covering our bases here but missing the mark. You need something that will offer up certain combinations of these cameras to 20 separate VMSes because the clients are entirely separate entities. Got it.

Will all cameras be shared with all clients? How is the overall split going to work? Is there any reserved groupings?

Can you perhaps provide the list of VMS software they plan to use?

Actually, the kernel for such project is the Trans-coder/MUX and De-MUX/decoder.

For example, hook-up the H264 IP cameras (3MP/12fps/4Mbps) through multicast

network address -> transcode to H265 (3MP/12fps/2Mbps), MUX combine it as a TS

stream > transmit by CDN to the 20 separate clients at the same time.

In the solution, each client receives 1. same TS H265 streams or 2. different TS H265

streams TS1 (client 1), TS2 (client 2).......TS20 (client 20), MUX at the headend site.

Choose 1 or 2 is up to your project decision. At case 1, each client De-MUX/decodes

Pre-defined H265 streams by different metadata ID. (in any case, client 1 can't see

client 2, client 2 can't see client 1 for the security reason)

As you can see the VMS suppliers would need to support H265 CODEC/TS wrapper for

the recording/playout purpose. Also CDN (edge server) takes care all 20 distribution

different sites.        

My Votes on Wowza. If you need it streamed they can do it to just about anything in any way possible. If Wowza cant do it, it cant be done!

We have several clients who pull camera feeds from server A on a secured isolated LAN and then push the video to an unsecured public Lan. No reach in requests allowed on secured side.

Then it can be embedded on a website, pushed to youtube, Facebook, Multicasted, re-recorded, played on iphone, android, etc, etc, etc, RTSP, RTMP, anything needed today.

Concerned about bandwidth deploy a local "receiver" type server at the stake holder site where you think you may have a considerable load for single stream PtP bandwidth reductions.

BUT you get no control of PTZ this is strictly rebroadcasted video, not ONVIF.

This user doesn't have 10k streams, but they do hit into the 1,000's of connected streams.

If you need the service up and running fast, this is COTS ready to go. Simple to commission, with no real programming experience needed. The website embedding does requires some additional skill obviously, and the API's are there to build onto down the road. For instance, stream stoppage on the public side versus the secured security side.

And just about any professional VMS can accept an RTSP stream so they can do their own local recording at the distributed stake holders.

From our Wowza experience, it's base on the per channel (camera) structure through the

public network internet/Cloud transmission.  In this scale project, if you calculate the

total number of the 20 sites distributions PLUS the bandwidth, make it impractical in the

real implement,Neither the CDN can do.

We can have the CDMA(Coarse Wavelength Division Multiplexer)solution through the

Fiber, so between the DC and 20 sites all are treated as the LAN/private network

environments. The Re-MUX H264 streams would be feed into CDMA sender equipment

with each different wavelength, at each client site would have the CDMA receiver to

handle/De-MUX the streams.  

Gavin, we can offer the above solution.  Kindly answer the below questions

            (1) What's the longest distance between DC and client-site?

            (2) IP Camera (3MP/12fps/4Mbps) provides the H264 TS (Transport Stream) or

                  NOT? multicast? or what's the brand of the camera?