Thanks! Great questions too!
The answer is HIGHLY dependent on the processor used, but the short answer is: Both.
Given the fact that most 'better' video processors support "3D" noise reduction, the only easy way to do this is using a frame buffer. It has to look back in time to get a sense for how frequently that pixel is changing in time, so it has to have at least one frame behind it.
But that frame buffer could be used on RGB or YUV data, which negates the original author’s request of having all those delicious unmolested bits to play with. The noise reduction is easier on less information, so doing it only on the Y data makes sense. Noise is usually changes in intensity, especially in low light, so again, Y only fits the bill for a frame buffer.
If they’re doing that then the incoming RGGB data is fed into line buffers, processed into intermediary stages and then dumped to a frame buffer.
That said, memory is also getting insanely cheap and blazingly fast now, so having big DRAM buffers to support multiple FULL bayer frames isn't too bad (at least less bad than it was 3 years ago).
Even if it was RGGB data, all the voodoo that the processing engines use to get frame rates and resolutions ultra-high, might mean however that it might be in non-contiguous hunks, etc, so putting it all back together is at best 'tricky'.
These are very closely held secrets and I don't have any inside knowledge aside from having worked in this space for a very long time and know the limitations pretty well, which give insights to how it is architected.
The major limitation is the CPU can’t get in there fast enough to write back out a full buffer worth of data before the other frame comes in and clobbers the one you were trying to read out. Like before, it might not all be continuous chunks and it’s possible that there’s never a whole frame that could be used.
All that to say, “I don’t know” but I’m guessing both :)
We have really wanted to get RAW / RGGB frames as well, and when I’ve plied the vendor’s hardware engineers with lots (and lots) of beer, they’ve allowed us to have “chunks” of RGGB data, but compiled from a bunch of frames that could then be combined to make up a complete image.
For time lapses of stuff that doesn’t move very quickly that technique would work, but it’s very hairy and those semi ‘back doors’ are prone to being lost in the next revision of the firmware. So even though this would likely satify the OP’s dreams, we gave up on having this as a feature in our cameras since it was too hard to maintain and it’s rarely used.
Great question on Rolling Shutter versus Global Shutters… For the rest of the group reading along, there are two basic types of electronic shutters available with modern CMOS sensors.
A rolling shutter starts scanning at the top and then drops line by line, ‘rolling’ its way down the imager. Once it gets to the bottom, it starts at the top and keeps going, ad infinitum… This can lead to some very weird artifacts with objects that move through the scene while the capture is being taken.
For example, if a truck drives through the scene, the top of the truck will be ‘seen’ first. As the truck swipes through the image, it is ‘seen’ at later times, but at that time it’s driven a few inches since then, so for the next row, it ‘looks’ like the truck shifted over a bit. By the time it gets to the bottom, the entire truck will be leaned ‘back’ and the wheels will be significantly further forward from the top. Weird.
Global shutters capture the entire scene at one time, so the top of the truck and the bottom of the truck are captured exactly at the same moment, so there is no distortion.
Rolling shutters are a lot easier and less complicated than global shutters, so they have ruled supreme in the video and cellphone markets.
The read out speeds are typically VERY fast as well, meaning that the time it takes from the top to the bottom has decreased dramatically as sensors have gotten faster. The faster you go from top to bottom, the less something moves in between, making it almost look like a global shutter was used. That’s why in my example above for the data rate, we put the sensor in 60 FPS (Hz) mode when at all possible, even if we’re only supplying 1080P30 to the H.264 engine. The faster frame rate makes things look better if they are moving.
Ok… all that aside, to finally answer your question, how does that change things buffer wise… the answer is not much.
The video frame is read in so quickly, even with a rolling shutter, that the processor has to gobble it in as fast as it can.
On the other side, the sensors with global shutters still have to feed that data out row by row as well, so they’re pretty much the same when it’s all said and done.