Multi-Camera Tracking Works?

By: Carlton Purvis, Published on Jun 27, 2013

Being able to track people across cameras and buildings simply by using cameras and analytics is something of a 'holy grail' for the surveillance industry.

Two researchers from Carnegie Mellon University have developed just such a system that allows them use surveillance cameras to track a person's movements throughout a building, even when they are out of sight of the camera. The set-up combines three different datasets that allow them to tag and identify people and place their movements on a map. In this note, we interview one of the researchers and breakdown how the system works.

Alex Hauptmann and Shoou-I Yu undertook this research because past datasets used to test multi-camera tracking usually include at least one camera with a global view of the whole environment, which is not the case in most surveillance deployments. They were especially interested in successfully tracking people in “complex indoor environments.”

Its first live test took place in a nursing home in 2005, with narrow corridors (see the camera layout above), many walls and a moderate amount of foot traffic during the day. Hauptmann and Yu found that they could successfully track people, even without seeing them by combining data from clothing, trajectory and facial recognition. The research was just published this year.

See the demo below:

Equipment Used:

  • 15 Panasonic PVDV953 3MP Cameras with MPEG2 encoders. The FoV among the cameras ranged from 30 to 60 degrees.
  • One server for every four cameras -- “We encoded [footage] into MPEG2 in 15 min chunks, and sent it to a server. Nowadays you would just have the encoding in the cameras but we didn't back then," Hauptmann said.
  • PittPatt Facial recognition software (before it was bought by Google)
  • A tracking algorithm created to combine clothing color, trajectory, a pre-loaded 3D map of the facility, and facial recognition.

Hauptmann says, “We didn’t actually pick this [equipment] to support facial recognition or person detection so when we recorded, we recorded in as high a resolution as possible (720X480), and this is what we ended up with. For this application, it worked reasonably well.” They were able to track a subject within one meter 88 percent of the time.  

Multiple Data Sources

Get Notified of Video Surveillance Breaking News
Get Notified of Video Surveillance Breaking News

Clothing Color

“Trackers not using color information will have difficulty avoiding identity switches when multiple people come very close together and split up. Exploiting color information will help in disambiguating different people,” according to their final report. Therefore, color was their primary data collected to track a person throughout the facility.

Lighting changes caused colors to appear lighter or darker to different cameras so the system was set to allows cameras to make their own color determination. If it stayed within a certain range, the camera would still recognize it. Another frequent problem with the nursing home residents was that “they don’t dress in fashionably different colors,” Hauptmann said. Often many of the residents were wearing similar colors, giving the camera a challenge in distinguishing between them.  

In both cases the algorithm would use trajectory to “verify” it was the same person as before. “Because you know in 3D where you think the person is, the new camera says ‘OK I’ve got the person in view and I’m taking this color determination for this person now.’" If two people appeared on camera wearing similar colors, the trajectory component would kick in and be weighted higher for identification. They later added a mutual exclusion limitation -- a person cannot be in two places at one time.

Trajectory

Trajectory estimation shows an expectation of where a person should be, based on their direction of travel, a 3D construction of the facility that the algorithm “sees” and other cameras. Even when a camera is not contributing information on a person, on its monitor, you can still see if a person is on the opposite side of a wall, indicated by a broken outline. Trajectories shorter than three seconds long were ignored. One limitation noted in the study would be corridors only covered by one camera. 

These images show the location same person in two camera views. One camera actually sees him, the other is estimating based on other camera views and his trajectory.

Facial Recognition

Facial detection was primarily used to supplement tracking when a person could not be identified by clothing color or predicted trajectory. It was also the most time intensive (taking about six seconds to recognize a face). Additionally some camera angles were less suitable for clear face shots, and because “many of the subjects were older people so as they were walking they were often looking down.” When cameras did get a clear shot of a person’s face, that data would verify that it was the same person detected by trajectory and clothing color.

“The rate of [images using facial recognition] was not very high but there was enough information coming in to let us know it was this person again,” Hauptmann said. Only 10 percent of all of the video frames collected produced usable facial information, but “it provided a component of error correction that you didn’t get otherwise. Without facial recognition the system could track a person within one meter 58 percent of the time. The algorithm using facial recognition was able to track a person within one meter 88 percent of the time. 

If you just go by color direction and trajectory, if a person stops of goes backward, you have almost no chance to keep track of them. The facial recognition gives us sort of a checkpoint that says this has to be this person here right now.”

The researchers note however that the algorithm is “not effective in very crowded sequences where each person wears the same color clothes,” presumably because of facial recognition difficulty, no reliable trajectory (people moving too slow/too erratically to estimate a path) and too much of the same color.

They present their results this week at the Computer Vision and Pattern Recognition Conference.

Future Plans

"We have already improved the algorithm substantially further – this project is ongoing in order to refine the technology. Currently there are no specific plans to repeat the study. This would need a significant amount of preparation and funding," Hauptmann said. 

There are no plans to commercialize the software (according to university policy it is owned by the creators and the university), but it could be made available for licensing, Hauptmann says. 

IPVM Analysis

Hauptmann said nursing homes were the application they had in mind when they started their research. The tracking system would be a way to keep tabs on patients who could wander off or to know where people are in case of a medical emergency. This technology could be beneficial in environments like hospital wards where people need to be accounted for health and safety reasons, but also in high security areas. The first application of this that came to mind was secure government facilities. The Mark Center for example has spent millions on layers of access control to know who is going to what areas of the building. Hauptmann's technology could give them additional ability to know exactly where they are in a building, not just what door's they've gone through. Another researcher is already working on similar technology that would be usable in crowded and more large scale environments to track a person of interest. 

On the negative side, the sheer density of camera coverage - 15 across a half dozen rooms, suggests significant cost and logistical constraints.

However, Carnegie Mellon was using technology from 2003 for study conducted in 2005, so it would be interesting to see how this study would turn out if replicated in 2013 with current technology and faster processing power.

Comments (3) : PRO Members only. Login. or Join.

Related Reports on Facial Recognition

Hikvision Global News Reports Directory on Nov 11, 2019
Hikvision has received the most global news reporting of any video surveillance company, ever, ranging from the WSJ, the Financial Times, Reuters,...
Critiquing Carnegie's AI Surveillance Paper on Sep 25, 2019
The Carnegie Endowment has issued an ambitious paper on the Global Expansion of AI Surveillance. While its aim is applaudable, the paper has...
IFSEC 2019 Show Report on Jun 19, 2019
The UK's largest trade show, IFSEC, is underway and IPVM has been examining what is new and happening at the show. Inside, we cover: Huawei...
Hikvision Chairman Warns Of AI New Entrants Victimizing Customers on Mar 12, 2019
Hikvision chairman Chen Zongnian has warned that customers will become “victims” of AI hype from new industry entrants, even saying such newcomers...
Massive Leak Of Chinese VMS Provider Exposes Xinjiang Surveillance on Feb 20, 2019
A subsidiary of China’s claimed largest VMS provider is tracking the precise location and ethnicity of millions in China’s Xinjiang region,...
Casino Surveillance Pro Interview: James Lathrop on Feb 15, 2019
James Lathrop [link no longer available] has been working in casinos for almost 25 years. During that time, he says he has held "just about every...
Intersec 2019 Show Report on Jan 23, 2019
The 2019 Intersec show, held annually in Dubai, is now complete. IPVM attended for 3 days, interviewing numerous Chinese and Western video...
NYPD Refutes False SCMP Hikvision Story on Jan 18, 2019
The NYPD has refuted the SCMP Hikvision story, the Voice of America has reported. On January 11, 2018, the SCMP alleged that the NYPD was using...
The False SCMP Story on Hikvision NYC AI on Jan 14, 2019
In the past week, one of Asia's largest publications, the South China Morning Post (SCMP), posted an article about "Chinese [facial recognition]...
CES 2019 Show Report on Jan 10, 2019
IPVM was at CES 2019 for the second year (see our CES 2018 Show Final Report) and is reporting on announcements and interesting news from the show...

Most Recent Industry Reports

ADT Stock Surges - "Leading The Commercial Space" on Nov 15, 2019
Don't call it comeback... but maybe call it a commercial provider. ADT, whose stock dropped by as much as 2/3rds since IPOing in 2018, has now...
Gatekeeper Security Company Profile - Detecting Faces Inside Vehicles on Nov 14, 2019
Border security is a common discussion in mainstream US news and politics, as is the use of banned Chinese equipment by US Government agencies....
Hikvision CEO And Vice-Chair Under PRC Government Investigation on Nov 14, 2019
In a surprising and globally covered move, Hikvision CEO Hu Yangzhong and Vice-Chairman Gong Hongjia are being investigated by China's securities...
Camera Field of View (FoV) Guide on Nov 13, 2019
Field of View (FoV) and Angle of View (AoV), are deceptively complex. At their most basic, they simply describe what the camera can "see" and seem...
UK Big Brother Watch: Hikvision Is 'Morally Bankrupt' on Nov 13, 2019
UK civil liberties advocate Big Brother Watch has condemned Hikvision as being 'morally bankrupt' following IPVM exposing Hikvision marketing...
Color Low Light Mega Camera Shootout - Avigilon, Axis, Bosch, Dahua, Hanwha, Hikvision, Panasonic, Speco, Sony, Vivotek on Nov 12, 2019
This is the biggest color low light shootout ever, testing 20+ super low light models from 10 manufacturers: Increasingly, each manufacturer...
Wireless / WiFi Access Lock Guide on Nov 12, 2019
For some access openings, running wires can add thousands in cost, and wireless alternatives that avoid it becomes appealing. But using wireless...
Hikvision Global News Reports Directory on Nov 11, 2019
Hikvision has received the most global news reporting of any video surveillance company, ever, ranging from the WSJ, the Financial Times, Reuters,...
Hikvision Markets Uyghur Ethnicity Analytics, Now Covers Up on Nov 11, 2019
Hikvision has marketed an AI camera that automatically identifies Uyghurs, on its China website, only covering it up days ago after IPVM questioned...
Open vs End-to-End Systems: Integrator Statistics 2019 on Nov 11, 2019
Preference for open systems is on the decline, according to new IPVM statistics. We asked integrators: For video surveillance systems, do you...