Casino Security Consultant Carl Lindgren Interview

Very interesting interview, thank you. Do you have any upcoming interviews for the retail side of the business? A large grocery or department store? Convenience or restaurant/fast food chain with analytics?

Reactions:

(2)

Dan Gelinas

In reply to Undisclosed Integrator #1

IPVM

Hello UI #1. At this time, there are none on the slate. However, if you have contacts in that space who might be able to fill that void, please feel free to email me at dan@ipvm.com and we can discuss.

Reactions:

(1)

Denis Halabudenco

In reply to Undisclosed Integrator #1

I can help if you want.

Reactions:

Ross Vander Klok

IPVMU Certified

Interesting read! I would like to hear some of his war stories about horrible installs, tapes being eaten, the issues with going from analog to IP when no one else had done it so no one had the experience, etc.

Reactions:

(1)

Carl Lindgren

In reply to Ross Vander Klok

Hey Ross,

Videotapes and VCRs were a nightmare. It took two people 4 hours to do tape prep, change, rewind and refile per shift so 24 man-hours were wasted per day. Then additional man-hours were wasted every time a tape review was required. The Investigator (agent/observer/???) had to prepare temporary substitute tape(s) to insert in the VCR while the tape(s) with the incident were being played back on their console VCR. Often, multiple cameras' video was required for the review so multiple tapes had to be swapped.

Then, if long-term evidence storage was required, the incident video from one or multiple tapes was dubbed to one "Master" and copies made for distribution to various enforcement entities. The end result was often pretty poor quality since the original recordings were in "EP" (SLP) and each generation of copies deteriorated from there.

The original switch from VCR to digital took around 4 months to install. One of the main problems we had both times was a lack of sufficient Server Room space. It's tough to fit 1,000 channels of encoders, servers and storage, etc. into a 14-1/2 ft. by 14-1/2 ft. room. The Alphapoint /Honeywell system also used Pelco CM9760-MDA Master Distribution Amplifiers to distribute video from our analog cameras to the encoders and our Pelco Matrix. The CM9760-MDA's were a holdover from the VCR system - we originally bought them for signal equalization on long runs and to insert time and date and camera # onto the video before it was recorded.

They had originally allowed us to purchase consumer-grade VCRs at ~$140 each instead of the Sanyo Casino VCRs at >$600 each. Over time, the cost of the MDAs was more than made up for in VCR cost savings.

Anyway, the lack of space and requirement to keep 14+ days retention of video with minimal downtime per camera dictated that equipment replacements be "staged". Essentially, we and our Integrators temporarily installed some servers in parallel with the legacy equipment, recorded simultaneously on both systems until the new system met our retention time goals, then replaced the old with the new in the old space. Rinse and repeat. Since every camera remained analog in 2003 and ~700 remained analog in 2013 the MDAs were in use until 2013 when we finally abandoned our matrix switcher.

The 2003 system definitely had some teething problems. One of the most egregious was with the Parallel SCSI interface. Connectors were not gold plated and we kept experiencing problems with bandwidth throttling due to bad connections on SCSI cables and terminators. U160 would throttle back in steps to 80mbps, then to 40mbps. We averaged around 50-60mbps for the 32 cameras-per-server so the system would drop I-frames. That resulted in some weird issues with the resulting video. One particularly odd defect in the video showed a walkway between table games and a wall. Video playback showed an empty walkway, then someone appeared walking away from the camera around midway down the aisle. They walked a few steps, then disappeared. We called it the Sycuan Ghost.

In 2006, we replaced the servers and storage and after consulting with multiple storage manufacturers, chose fiber channel; as the data transport method between the two. Fiber channel was far more expensive than i-SCSI but i-SCSI was in its infancy at the time (and the "SCSI" name scared us). Anyway, fiber was infinitely more reliable and we had very few problems after that.

Surprisingly (or not), the 2013 switch to IndigoVision was far smoother than in 2003.

Reactions:

(2)

(10)

Undisclosed Manufacturer #3

I once had a prison client that saved a significant amount of money on fungicide when they switched from tapes to digital! The only place they had to store the tapes was in a damp room somewhere in the bowels of this old prison. They kept video for a long time and tapes would apparently get moldy without the fungicide.

Reactions:

(3)

Carl Lindgren

In reply to Undisclosed Manufacturer #3

Not only damp storage but there were a huge number of problems with tape and VCRs.

Rewinding - many casinos would rewind their tapes in the VCRs. While that saved some time, it meant the VCRs weren't recording for the minute(s) it took to rewind. Also, rewinding in the VCR could chew up the tape - T160 tapes were so thin, they would stretch or fold if you even looked at them funny. In either case, the mutilated tape(s) would then clog up the heads and recordings suffered.

Crap Tapes - as manufacturers moved production offshore or abandoned making tapes altogether, the quality deteriorated. Cases also got cheaper and less immune from daily handling. The end result was more lost or unusable video and more repairs required to the VCRs.

Cheaply made VCRs - as "Pro" machines with beefy drive mechanisms were discontinued, replacements were often just "consumer grade" VCRs with the tuners removed and RCA (phono) video connectors replaced by BNCs. Otherwise, the so-called "Pro" machines were no better, and often far more expensive, than their consumer counterparts. I saw no need to pay almost $500 to purchase a "consumer" machine with no tuner and BNC video connectors.

Lack of Parts - often VCR manufacturers stopped selling parts soon after discontinuing a model. So toward the end, we were replacing VCRs annually. That cost around $84,000/year. It was one of the arguments we used to put our case to go digital in front of the Tribal Council. That, along with efficiency improvements gained by eliminating tape handling and VCR replacements/repairs.

Reactions:

(2)

(8)

Ross Vander Klok

IPVMU Certified

Thanks for the response Carl! I had a hard time maintaining five VCR's and just thinking back to those days gives me nightmares.

Reactions:

(1)

Carl Lindgren

In reply to Ross Vander Klok

Ross,

Maintaining 700 VCRs was a trip! I had a helper whose primary job was to clean the tape path of VCRs. She would do an average of maybe 20-25 a day when we didn't need her help installing cameras or pulling cables. If she found one that was chewing tapes or otherwise still wouldn't work properly after cleaning, I would take over.

Eventually, we decided it was cheaper and easier to replace VCRs that had major problems rather than repairing them. Worn out or damaged video heads became the death knell for all but the few true Pro machines we kept for their capabilities. We went through a succession of Mitsubishi models starting with the HS-U445's we originally bought to replace the expensive Sanyos. HS-U445 was followed by HS-U446, which became HS-U447, which was followed by the 448 and finally, the HS-U449.

But that was nothing! When I was between jobs prior to being hired by Sycuan, I actually interviewed for a bench tech job at a Technicolor tape duplication facility in Camarillo, California. Just north of LA. Technicolor Videocassette Inc. produced video recordings of many major movies. The facility had something like 10,000 modified VCRs that recorded at some multiple of normal speed. They had something like 15 or 20 technicians constantly servicing the machines.

When I went there for an interview, each "bench" had literally 50 or more VCRs stacked up next to it and I was told the PM technicians (cleaning) were expected to process at least 50 a day. Bench techs were expected to repair at least 30 a day.

Honestly, the place could have been the poster child for an article about sweat shops. None of the techs seemed to enjoy their work - they all had frowns. When I was waiting to talk to the manager, she was verbally berating one tech for not meeting production goals - right in front of everyone. When my interview finally took place, I begged off telling her I was no longer interested in the position.

Whew!!!!

Reactions:

(1)

(3)

Lee Brown

Very good interview.

As a consultant for casino projects what kind of integration methods do you champion for slot data, point of sale, and access control integration?

In our operation we see ourselves, "Surveillance", as a data consumer for these event sources. We let IT / Casino security manage the access control and POS services but connect through our own managed firewall.

The stereotypical VMS access control integration is backwards from our perspective, in that we would rather present the AC/POS data in the VMS and not push our video out to an AC client/API.

Can you recall any integrators accommodating this model or providing integrator led software development services ?

Reactions:

(1)

(2)

Carl Lindgren

In reply to Lee Brown

Hi Lee,

That varies from casino to casino depending on who is deemed responsible for what. In our case, IT and the Slot Department handled all slot data, and they were regulated by another department of the Gaming Commission. At one point, Surveillance had access to slot data but it was a distraction for us since the system reported EVERY jackpot, EVERY belly door jostle by a customer's legs and a whole bunch of unidentified alarms no one could explain. In the end, we just turned the programs off.

All of those "peripheral functions" not directly determined to fit in the Surveillance Department's primary goal "To Protect the Assets of the Casino and the Tribe" were often delegated to other departments with Gaming Commission departments, including Surveillance, providing oversight.

I understand other entities may prefer to have Surveillance perform some, or all, of those peripheral functions so I have familiarized myself with systems and methods to provide them. I'm most familiar with eConnect's POS interface and have worked with access control on side jobs and for other departments that have their own systems.

I've worked with Reliable Security and eConnect on the POS integration side but the system never caught any problems. Food and Bev did an outstanding job of policing their own so our job ultimately would up just collecting and storing video evidence.

Reactions:

(1)

(2)

Stephen Green

IONODES

Thanks for sharing Carl. Great report. I've been told that casinos record the activity of surveillance managers in the VMS application. The rationale for doing this is for investigations into insiders partnering with scammers. I guess there would also be a deterrent value. Is this accurate and how common is it?

*Full disclosure* I work with a company that makes an encoder that could be used in the scenario I'm asking about.

Reactions:

Lee Brown

Stephen,

Can you share more details about your encoder as we have this exact purpose in mind. We are using some OEM type HDMI->Onvif / rtsp but we are not real impressed with the performance , passthru and connection options.

Reactions:

Stephen Green

IONODES

In reply to Lee Brown

Hi Lee. Yes that is the scenario: HDMI out, to encoder, and generate an ONVIF video stream of the operator's desktop. I don't want to hijack this thread with a product sales pitch. If you Google ION-E100HD you should find the details for our encoder. Or e-mail me at steve.green@ionodes.com and I'll send you all the details

Maybe you can answer my original question. Is this standard practice in gaming?

Reactions:

Lee Brown

Thanks Stephen,

I have seen it done in a few places. Usually the audit log feature in the VMS is sufficient. Having the the desktop recording helps us a great deal with training or examining workflow issues.

Ill ping you through your site for additional product questions.

Reactions:

(1)

Carl Lindgren

Stephen,

You mean the activities of (Agents/Observers/Investigators/...) - the camera watchers?

If so, yes. Most Casino Surveillance departments record activities in the Control Room. Many also record audio too. In part that may be to document collusion or other "questionable" activities by Surveillance personnel but far more often it is to used provide backup to the "He said, she said" scenario where someone claims they notified Surveillance of something when they didn't or claims Surveillance said something that they didn't say.

Reactions:

(3)

Stephen Green

IONODES

Hi Carl. Yes that is exactly the scenario I was asking about. Thanks for the response. The audio part is interesting, I hadn't thought of that. Is the ambient audio in the Control room recorded or is it recorded directly from the phone line? Is the audio recording integrated/synchronized with the VMS?

Reactions:

Carl Lindgren

Stephen,

Typically, casinos record the ambient sound though I have heard of them recording phone lines. I would bet LouRoe sells a large number of microphones for that purpose.

Reactions:

(1)

Cameron Javdani

•Apr 01, 2019

Louroe Electronics

I'm a little late to the party here, but we see two very common applications for our mics in casinos. One is in casheir/count rooms to verify marker amounts and repayments. The other is in the security investigations/interview room. I've also seen installations in control rooms and at high limit tables, though in my experience these aren't as common.

Our products don't record over the phone line, they are designed to pick up the environmental audio. To sync with the VMS we can either connect through an IP camera of have one of our IP devices stream to the VMS independently.

Reactions:

(1)

Undisclosed #2

IPVMU Certified

I did the VCR repairs on my own...

@Carl, did you own a tubebox?

back when TV repair was DIY :)

Reactions:

(2)

(3)

Carl Lindgren

Undisclosed #2,

No, but that was because I only did TV Field Service very briefly. I did, however, make use of this a number of times during my early career:

Reactions:

(3)

Cary Menage

IPVMU Certified

I'm really enjoying these interviews. This one in particular for nostalgic sake. I once worked for a small "mom and pop" that specialized in VCR and monitor repairs, and tube replacement for the Vidicons and Ultracons for RCA (Burle) cameras. In High School I took Radio and TV repair where we would work on teachers and staffs TVs and Stereos, and all they had to pay was for the SAMS schematics and parts which helped us build up a rather extensive collection.

Thanks Carl and Dan!

Reactions:

Undisclosed #2

IPVMU Certified

In reply to Cary Menage

I once worked for a small "mom and pop" that specialized in VCR and monitor repairs, and tube replacement for the Vidicons and Ultracons for RCA (Burle) cameras.

then you may be one of the few people on this site that know where the arcane and confusing sensor size format comes from, e.g. 1/2” 2/3”, as it was simply the outer diameter of the glass vidicon tube, not some fudged up diagonal measurement like it is with solid state sensors.

though a 2/3” vid is humongous (overall) compared to a cmos/ccd sensor:

Reactions:

(2)

Carl Lindgren

When I first started out, I often dealt with these:

Panasonic WV-1400 series B&W Vidicon CCTV camera.

Reactions:

(1)

Undisclosed #2

IPVMU Certified

The NIGC's [National Indian Gaming Commission] MICS, that's Minimum Internal Control Standards, stated that for certain games you needed to have “real time” or at least 20 frames per second for all critical cameras—table games, bingo, cash operations—they required what they called real-time. And they specified what you needed to see and you couldn’t do it with 320X240 and 15 FPS.

was the MICS requirement for 20 FPS interlaced or progressive?

Reactions:

Carl Lindgren

In reply to Undisclosed #2

Undisclosed #2,

The actual text is "Sufficient clarity means use of monitoring and recording at a minimum of twenty (20) frames per second. Multiplexer tape recordings are insufficient to satisfy the requirement of sufficient clarity." The MICS goes on to apply the term to a number of applications including Gaming Tables, Bingo, Progressive slot machines, Keno, Pari-mutuel betting, fills and credits and currency and coin counting and storage areas.

The MICS is, in my opinion, poorly worded since a reading of the term "sufficient clarity" when applied to the applications listed above, appears to refer to image resolution while the definition appears to apply to frame rate. Once, when we were visited by the NIGC bigwigs, including the Chairman, I questioned him about that ambiguity. He didn't answer but my boss was not pleased. ;^O

Here's a link. https://www.ecfr.gov/cgi-bin/retrieveECFR?gp=&SID=f313da06102dfd89fadbad83ae68de3b&r=PART&n=25y2.0.2.4.26

In any case, that apparent lack of sufficient clarity (pun intended) as to what was actually required made us err on the side of caution. We figured if we recorded ALL cameras at D1/4CIF @30fps, no one could ever say we didn't meet MICS requirements. And that was true. The NIGC inspected us a number of times over the years and we never failed.

Back to your question: The MICS specifies frames per second but the 20 number actually derives from so-called Real-time VCRs that ran the tape at 1/3 of SLP speed and only recorded odd (or even) fields to yield 24 hours at 20+fps on a single tape. VCRs have 2 heads located 180 degrees apart on a drum spinning at 1,800 RPM (which equals 30 revolutions/images per second). Each head records and plays back a field. Because the tape is running so slow, one head would overwrite the recording of the other so they only used one of the two heads and the other was a dummy. That means Real-time VCRs only recorded every other field.

Reactions:

(1)

(4)

Undisclosed #2

IPVMU Certified

Each head records and plays back a field. Because the tape is running so slow, one head would overwrite the recording of the other so they only used one of the two heads and the other was a dummy. That means Real-time VCRs only recorded every other field.

but on playback, did it use two heads or one?

if it used just one, did it compensate for the missing field by repeating the previous field, or did it squish and letterbox the image?

if two, did the second head “read the bleed”, similar to repeating the field with one head?

either way, the effective vertical resolution is halved on playback, right?

Reactions:

Carl Lindgren

In reply to Undisclosed #2

IIRC, most repeated the field on playback. It's been a while but I seem to recall the difference between standard VCR head drums and Real-time VCR head drums is that on the standard VCR head drums, the two head' "gaps" were offset in angle to minimize crosstalk between adjacent tracks. +/- 7 degrees from "vertical" comes to mind. I believe Real-time VCRs had two heads with the offset being in the same direction, effectively allowing the two heads to play the same track twice.

Yes, I believe Real-time VCRs had an effective vertical resolution of 240 lines.

Reactions:

Undisclosed Integrator #4

I was both surprised and not surprised about your comments on Honeywell.

I work as an inhouse technician and installer for a casino and they've been Honeywell for like close to twenty years now. Seems like a ton of Casino's are.

I'm not a huge fan of the product myself, I don't have a ton of experience with other VMS', but I feel like there must be better. Seems like a lot of Casino's are just too scared to do a total system switch over.

The ones I work for seem to threaten it every once in awhile when Honeywell tries to charge for something they used to do for free, or really jacks up the price of Licensing or SSA or something. But I'm not sure if they'd ever follow through with it.

Have you found that to be true? Casino's kind of locking themselves in with one vendor and getting stuck?

Granted everything I hear about other casino's comes through my manager, who pretty much just deals with other Honeywell Casino's.

Reactions:

(1)

(2)

Carl Lindgren

In reply to Undisclosed Integrator #4

Undisclosed #4,

Yes, casinos are notoriously resistant to change. In fact, the man who hired me was transferred to another position and stayed with the casino when the Gaming Commission replaced him. My new "boss" was a different person and something rare in the industry: he believed in empowering his employees. He also promoted me to Supervisor from Lead Technician.

After we had had the system some time, my old boss asked me if we still had the system. I said Yes, why wouldn't we? He said he would never have approved it and he though it would break down constantly or, at the very least, not pay for itself.

He originally came from Las Vegas and that was, and probably still is, the mindset in Corporate Casinos. If you can't demonstrate an ROI, you don't get any funding. That's why Corporate Casinos tend to be the last to adopt new Surveillance technology, if at all.

Reactions:

(2)

(1)

Carl Lindgren

IMO, Honeywell is one of the worst companies in the industry. They forced Alphapoint out of their product niche by offering to buy up something like a year's worth of encoder production from the OEM: FAST Video Security, AG in Switzerland. Both Honeywell and Alphapoint sold their own "flavors" of the Alpha system. The primary difference was in the technical competence of the two companies. Honeywell used cheap consumer-grade networking while Alphapoint installed a Cisco 6700 core with VLANs.

Early Honeywell customers had major networking headaches while our network performed flawlessly. Did we both have issues? You bet. I believe both company's versions used parallel SCSI interfaces between the servers and storage. Different equipment manufacturers but the same technology. I heard some casinos were not thrilled with Honeywell but were unwilling to make waves by demanding the system work as advertised. We were not so timid ;-)

Reactions:

(2)

Undisclosed #2

In reply to Undisclosed Integrator #4

IPVMU Certified

Seems like a lot of Casino's are just too scared to do a total system switch over.

oddly, casinos don’t like to gamble :)

Reactions:

(1)

Carl Lindgren

In reply to Undisclosed #2

LOL, it's not odd at all. It's common knowledge in the industry that the only sure winner is the house.

Reactions:

Undisclosed #2

IPVMU Certified

...the only sure winner is the house.

true. for instance, IDIS offers 36 to 1 :)

Reactions:

Lee Brown