Based on a LOT of experience with this...
1. You do NOT need active baluns for a mere 120m. Even cheap passives should be good to 200-300m. Either way, beyond 100m, you're pushing the limits for ethernet, so if you want to upgrade those cameras to IP later, you *may* need some sort of extenders or twisted-pair adapters (or maybe not, I have heard of ethernet working at that range, although it's beyond spec).
2. If you lost signal after 50m, there's something else wrong. FIRST AND FOREMOST, you want to use ONE PAIR for video, period. Mixing-and-matching pairs will result in a loss of each pair's noise-rejection properties, and you'll end up being much more susceptible to interference. It sounds like you may have used some sort of cheap UTP-to-BNC adapters that are flooding eBay lately, masquerading as baluns, although they aren't. Big names like MuxLab and NVT are fine, but stupid expensive; a balun is a pretty simple device and I've had just as much success with low-cost units... just make sure they ARE actually baluns.
As far as power goes, yes you can use the remaining pairs, but how far you can go will depend on the voltage used (12VDC vs. 24VAC), and the current draw of the camera. On one site, I have two CNB domes running 200' on a single Cat5e - one pair for video, one pair for power for each camera. They draw only 2.2W, or 180mA at 12VDC, and will operate any anything between about 10-30V, so the minimal voltage drop still leaves sufficient power for the camera. Critical to this is the fact that they don't have heaters or IR - IR will typically add 500mA or more to a camera's current requirements.
With Cat5e, three 24ga. pairs together is about equivalent to one pair of 18ga., as far as current capacity and voltage loss... with Cat6 it should be a little closer to 16ga. If the camera will run on 24VAC, you should have no problem powering it at 120m.
I've always found this calculator particularly helpful.
3. It shouldn't matter which pair is used for video. I've typically used the blue pair, chosen mainly as the center pair of an RJ45 connector, so I could even use a basic two-wire phone cord to extend it if need be. However, we've recently been using these power/balun units with much success, and they put the video on pair 1 (pins 1 and 2 - orange if you wire T-568B). At the camera end, we'll either use these all-in-one baluns, or if something small is needed (such as to fit inside a dome housing), we'll use something like this on the video pair, and just wire the power pair in directly.
Remember that there's nothing special or magical about a power-and-video balun - internally, two pins of the RJ45 jack go into the balun, and the others wire straight through to the power connectors. Same applies to the VPS units above.
Some nice things about the VPS boxes, BTW:
- they help reduce cable clutter
- the 12V power is regulated
- there's self-resetting overcurrent protection on every channel... so if you have a short in a power pair or make a wiring mistake, that channel will shut off until the short is removed
- In addition to the balun function, each group of four video pairs goes to another RJ45 jack, so you can take those four cameras and run them to another location over a single UTP cable, then split them out there with another VPS, or with a simple multi-channel balun unit.
In the case of that last item, I did exactly that on a recent job: one 16-channel VPS in a remote wiring closet; four Cat5e linking patchbays there and in the main IT closet; another set of patchbays linking the wall rack with the rolling rack containing the DVRs; and a the 16-channel passive balun in the back of the rolling rack. So each camera has a Cat5e run that patches into a channel of the VPS, each group of four is then fed from the VPS, through the patches, down to the passive balun, and the "tails" from the DVR connectors then hook straight into the BNC outputs on the balun using these, and a couple of these where I need to split off an analog signal. It's working GREAT, and a benefit of terminating everything in a patchbay means that switching an analog camera to IP means simply unplugging the camera, moving the patch from the VPS to a PoE switch, and plugging in the new camera (which I've already done once on this site).
Re: 24VAC power: there is no problem using it with baluns. The voltage and current are low enough so as not to generate significant interference in the first place, and the unique twisted pairs used in Cat5/6 means each pair has its own noise-rejection properties anyway. If distance and power requirements call for it, have no fear in using it.
One final note (for now): I have no relation to the above linked vendors, EasternCCTV and ConnectorsPlus, other than that of a very satisfied customer. Neither are by any means the only suppliers of these devices, but both pre- and post-sale support from both has been outstanding and I'm happy to recommend (and link) them. That probably doesn't help Heng, but may be useful to others reading this thread :)