Reading trough the big amount of video playback related topics it seems there will never a good and easy solution for high res playback with audio on Windows.
For most of our projects we use a DXT5 solution which is very stable, easy and works well in threads (for playback of multiple videos at once) but it lacks audio.
So with Cinder having a good Linux version I was wondering how video works.
Is it possible to play multiple videos at once with some level of HW decoding?
Multiple videos at once should be possible with the Linux video player depending of course on resolutions and hardware. With decent machines I have had very good experience with few HD videos and also 4k playback. On Desktop there is no support for HW level decoding currently since this is an area where development is still fluid in GStreamer upstream. There are some elements like gstreamer-vaapi but there are currently under heavy dev and work for some combinations of hardware and not for others. I think we will have to wait a bit more for this…
That said, Cinder’s video player implementation uses the gst-gl plugin which has brought a lot of performance improvements ( i.e color space conversion is happening on the GPU which traditionally has been one of the biggest bottlenecks, pbo’s are used for uploading data to the GPU etc. )
Also, GStreamer is by de facto heavily threaded internally so there is no need to have multiple threads running on your side in order to have multiple players running in parallel.
One thing to note is that the current implementation by default will load synchronously the video files which means that directly after creating/loading your player you can be sure that the first frame of the video has already arrived and is available together with metadata like width, height, fps etc. The upside of this is that you don’t have to check if the video is loaded after creation and you can be sure that its ready to use – the downside is that this will block the main thread until the video its fully loaded.
If you don’t care if you have to check if the video is loaded before using it then you can enable async loading with just a boolean flag and this will keep your main application thread smooth and non-blocking while loading even multiple videos in parallel ( it actually might worth considering to make async loading the default option since this is how it works, I think, with the AVFoundation player on macOS currently ).
I would say give it a spin and lets us know how it works for you!
Isn’t the problem with having async by default that some operations would crash it if you don’t check that the video is loaded first?
As an aside, it’s a shame you can’t just read the header in some faster manner, to fill out things like resolution for example, but then let everything else work async.
Also really cool to hear that the gstreamer community is actively working on linux hardware acceleration, this seems tricky and really no other video solutions have done a great job at it yet, in any easy way for us to use.
Trying to access the video texture pointer directly after the load function has return, in the async scenario, would lead to a crash since it will be empty until isLoaded() returns true. All other operations would not lead to a crash but they would be reporting incorrect ( the defaults basically ) values until isLoaded() returns true.
The sync loading scenario can come handy in cases where you don’t have / need dynamic video loading / unloading ( i.e just load on app startup ) so you don’t want to have more complicated logic than it is necessary in order to control the video player or other elements that depend somehow on the player. It might be for example that an element’s creation depends somehow on the resolution of the video, so you could sync load your video and then directly after pass it to your element constructor and be sure that it will get all the correct values without having to check in an update loop if its actually loaded.
Did some tests today.
Its really amazing how fast it is to setup a Ubuntu with Cinder, big thanks for making this so smooth.
Also videoplayback is nice and easy.
Do you know the size of what the texture will be before the load function has returned? If so, would creating a black texture of the right size be a decent middle ground?
I do believe other implementations assume there is always a gl::TextureRef available (example), but I’m not certain if that is necessary.
Sweet that you got into it so fast! Regarding the memory issue – The code is valgrind clean from the GstPlayer side so if there is a memory leak it shouldn’t come from the implementation of the player. Actually, I believe ( hope ) that what you are experiencing is just the aggressive nature of caching on Linux and not an actual leak – You can read more about it but the very end result is that the system is keeping a lot of resources in memory for faster accessing times and its on the operating system’s discretion to share the memory as it it sees fit to the rest of the programs. You could test this by keeping your test running and check if you system really does get out-of-memory and unresponsive. I would also suggest to check memory consumption by running watch free -h on a terminal.
If caching is not the case and there is actually a leak somewhere then it could be that your graphics driver is leaking. Bare in mind I have tested the code only with NVIDIA hardware and always with the latest drivers manually installed from the NVIDIA website ( this is a hint for the really old and outdated NVIDIA drivers that ship by default with *Ubuntu and that one should not use ) .
Another note is that I have been personally using the latest available GStreamer sources since there have been lots of fixes ( including memory issues ) on the gst-gl plugin since the version that ships with 16.04. For this reason I have created this script for making your life easy when it comes to installing the latest versions of GStreamer. You can run it with sh create_gst_uninstalled_env.sh -b 1.12.0 for example and will download, compile and configure everything for you in order to be able to use the latest, as of now, GStreamer version. You can read more about it in the Github repo.
Lastly, the GstPlayer besides support for the experimental gst-gl API ( you really need still to specifically activate experimental support ) it also has support for the traditional API which might be a bit slower but it can still be perfect for a lot of scenarios. This should be a CMake config flag in the future but for now you can force it by setting this flag to false.
As a closing bonus here is a recording of loading bbb at 4k and some web video every 3sec together with my memory consumption. For the shake of testing I 've used the version of GStreamer ( v1.8.2 ) that ships with *Ubuntu 16.04. You will notice probably an initial increase but then the memory is reaching a plateau.
The Linux video player behaves the same as the other implementations on that front. You basically have to check if the texture pointer is valid before trying to use it as in the example you linked. On macOS, also I believe currently if you omit this check the application will most probably crash.
I m not aware of a way through which you can access video metadata before the pipeline has pre-rolled ( in GStreamer terms ) and I m fairly sure that there is none actually so I suppose for cases that you really need to have this data directly after calling load you would have to use the sync loading option.
All these would be nice to discuss in a more general context also. Currently everything is mapped in the qt namespace and I think it would be great if we could start a discussion around a more modular Cinder Video API. Currently there is a lot of functionality in the GstPlayer that is not really accessible because of the current design of the qt player and in addition this design forces some practices that are not ideal in the case of GStreamer i.e reloading a video currently requires to destroy the whole player which means that GStreamer has to flush and reconstruct the pipeline in its entirety when this would not be necessary if there was a load function for example where we could just reset the necessary parts and not the pipeline as whole.
Just want to +1 this. Video on mac is great, but QuickTime on windows is basically unusable these days, so might be time to formalise a new api with more backends. I currently use MediaFoundation with a shared texture via NV_GL_DX_Interop2, for example.
Indeed it seems Linux caching the memory went stable after a while.
We currently started this project on OSX but will look at again on the next project.