On the OSS side of things you could take a look at Blender: http://www.blender.org/

While Blender is known primarily for the 3D modelling/animation side it also has a non-linear editor and compositing capability.

The most recent versions have focused on the motion-tracking and compositing needed for their most recent live action+CGI film which is due for release shortly: http://www.tearsofsteel.org/