You could do this using REAPER w/ joystick-MIDI support and a custom video processor... would take a bit of scripting though, yeah. Maybe at some point I could do it... would you be looking for interactive realtime use, or for automated use to render later, or both? The way video is handled, realtime preview while recording automation is perhaps a touch tricky.