If the current resolution in Windows matches the native resolution of the display, then I would assume everything else in Windows looks okay?
If the issue is only the quality of the video within Smart Client, then I'd be interested to know what the source resolution is from the camera(s). If the cameras are outputting 1080p but the display is UHD/4K, then the video may be upscaled to fit the resolution of the screen, in which case you should expect to see some amount of pixelation.
Does the video look okay if the cameras are in a 2x2 (4 camera) view layout?
There's also an option while in setup mode in Smart Client to change the image quality. If it is not "Full", make sure to change it to full, as well as "unlimited" frame rate and try again. Changing these options results in conversion of the original video to MJPEG as the specified quality/framerate.