Been installing edge IP-POE readers for a long time. Concerns about security and how they are addressed:
1. IP readers have an optical tamper sensor on the back - if they are removed from the mounting surface, the devices goes into tamper alarm and does not allow admits while in tamper mode. Some newer IP readers will have a gyroscopic tamper sensor which should be fairly sensitive to the initial disruption of the reader if it is removed fro the mounting surface.
2. The server software records the tamper alarm almost immediately, which can cause an email to go out to someone and/or another reader's TTL output to go active via an easy script which can cause a loud horn/siren to sound.
3. The compromised reader's own TTL (spare) outputs can be wired NC to an audible/visual local alarm annunciation device via a small inexpensive TTL relay which will cause a horn/siren to sound if the reader is removed from the wall (as long as the power source for the horn is on the secure side)
4. Brian has already mentioned the existence of a serial lock control relay device (like Isonas uses - the EDK), which places the relay that controls the lock on the secure side of the door.
5. The network port that the IP reader is connected to should be locked down by MAC address, requiring a reset of the arp cache on the managed switch (hopefully a managed switch would be used on a secure installation) in order for any other IP device to communicate.
Additionally, managed POE switches would allow the change of state of what the switch port sees to generate their own type of alarm - I'm not quite as familiar with the setup of SNMP alarms and such on managed Ethernet switches, but they have various alarm capabilities that can be set up.
Additionally, the IP readers should be set up on their own VLAN, no different than IP cameras, thus limiting the exposure on the overall network that the external ports would allow.
6. Removing the IP reader from the network in a rogue fashion will also cause a controller failure alarm to be generated in the management software (because the IP reader no longer is sendig its status to the server software), which can also trigger an email or some other action at another reader via a simple script.
7. The IP readers offer 256 bit encryption to be turned on between the readers and the management software. If someone tries to connect to the reader with their own instance of the manufacturer's software tools to try and harvest stored badge numbers or events and does not have the 64-character encryption key, they should not be able to communicate to the reader's firmware. Entering 64 character strings is a pain in the butt and most customers do not use this additional level of security, but it is there and mostly used for applications where the reader is being remotely managed via a cloud application.
These days, nothing is hack or vandal proof .. the bad guys are always a step ahead. From a practical standpoint, there are easier ways to get into a controlled door than by removing the IP reader and doing something from there, but I suppose if someone is trying to compromise a door undetected (not a brute force entry or drilling into the door frame to get to the actual lock), the IP reader could be seen as a logical place to start.
Finally, taking the security in depth two steps further, why wouldn't there be an IDS system involved in a critical/secure application? Doesn't cost much and would be the most dependable way to remotely monitor/alarm the entry point. Then, video camera(s) and integration of video and access would allow the reader tampering to be fully recorded, even without the video integration (that would link stored/indexed video records to both the reader tamper event and the reader failure events) you would have a way to review what happened. The integration could be used to allow the VMS's alarm capabilities to be used to address the reader alarms as well.
I have never seen an IP reader compromised by a rogue event ... most of the time it ends up being a mechanical/physical door issue with either the door itself not closing properly due to some external factor (intermittent air pressure changes, improperly adjusted door closers, mis-aligned doors and frames, ill-fitting door hardware, failure of a power supply) that is corrected once the issue is observed and repeated.
The sheer labor savings from IP/POE readers is real and turn's a 2-man job into a 1-man job at a door. There are certainly times when an IP/POE controller mounted on the secure side with only a regular prox or smart card reader mounted outside is a bit easier because it eliminates the need to get a Cat5/6 all the way down to them reader location, but it sure is nice and clean to not have to mount a panel at every door.