This is a difficult question to answer in a general way as there can be a lot of causes for this symptom. But I suppose the very simplest answer I can give you is the VMS probably has a "bug".
Restarts should never be necessary, regardless of the component (camera, software, switch, router, etc). But since software is written by imperfect people, it usually ends up being imperfect as well.
I put quotes around the word "bug" above because it might not necessarily be something wrong with the software. Sometimes cameras do unexpected things like send malformed video, or respond to HTTP GET/POST only partially, or in otherwise unexpected ways. It might not be that the VMS did anything wrong to cause the issue, but there's usually something you can do to recover gracefully. And sometimes it's easier said than done depending on the type of failure and the software design.
Milestone has the ability to detect and recover from errors. For example, you can almost always sever the network connection and restore it later, and the camera will come back online. I've even used Clumsy to make the video stream so corrupt that it introduces weird blocky colors and eventually kills the video stream and when I turn off Clumsy, it returns to normal. If a broad driver-wide issue occurs somehow which can affect multiple cameras, we'll even reload that entire driver. Still, we might not be prepared to handle every possible error condition. So if restarting the server solves the problem, we want to know about it so we can make the software even more robust.