Options for windows-based hardware accelerated video in Cinder 0.9.2-dev

Hey Embers,

I know this topic rolls around every few years, but since I hadn’t seen one in recent times I thought I’d stick my head out and ask what, if any, experience other developers have with hardware accelerated video playback in Windows, in Cinder?

About two years ago I used the often cited WMFVideo block. However, with my current codebase this block just seems to throw a fit, specifically it seems to cause an access read violation when I snag enough memory.

hr = OnSessionEvent( pEvent, meType ); <--- causes a read access violation.

I’m currently not convinced the error is due to my large codebase as turning different portions of it on/off appears to cause (or not cause) the read access violation. I.e. I’ve so far not been able to correlate WMFVideo failing consistently with a particular part of this largish code base. That’s - of course - not a guarantee it’s not my code, just my current view.

Anyway, given this, I’d like to explore alternatives. I don’t even mind if performance is a bit dodgy. I.e. I’d rather sacrifice some video resolution for something that won’t haphazardly crash. Anyone got anything they’d recommend personally and have used themselves?

Cheers,
Lasse

Lately i’ve been using HPV, with my implementation based on an openFrameworks one. I like it because it’s a lot more deterministic, which was important for my last project which required video to be synchronised across multiple machines.

The downside is that it’s a custom “codec”, so your video must be processed ahead of time, and the file sizes are comparatively huge. This was less of a problem for me because i’d written a full asset pipeline so transcoding the videos just became a part of that, but may be a dealbreaker if you don’t have control of the videos ahead of time.

2 Likes

Having spent a little time researching options now, I thought I’d share my findings in case anyone else has to do some decision making in this area. Take what I’ve written with a grain of salt, as I’ve not exhaustively investigated all options, more tried to get a quick overview of things.

Options for decoding video frames on to an OpenGL texture in Windows in Cinder:

CinderWMF

  • Pro: High-performance, hardware-accelerated
  • Pro: Wide format support, given that it leverages the Windows Media Foundation
  • Pro: Cinder native implementation exists
  • Con: Appears to be unstable, possibly due to having to leverage DirectX to access video frames

Given how straight-forward it is to test if your code can work with this cinder-block, this should likely be your thing to try.

HPV


  • Con: High-performance, although not hardware-accelerated
  • Con: Video files must be in HPV format
  • Con: No cinder native code exists
  • Pro: A good starting point for code exists, in the OpenFrameworks code (similar to Cinder).

I’d wager this option, although requiring some elbow-grease in the form of coding, is most likely to cause the least on-going head-aches once complete. Not having to deal with multiple rendering contexts, or playing in DirectX can be worth some serious concessions.

ffmpeg-cpp

  • Con: No cinder native code, but I believe this library to be a provide more access to the raw video data, i.e. it’s likely possible, although not straight-forward to leverage ffmpeg to get individual video frames on to an OpenGL texture.
  • Con: Documentation for ffmpeg is allegedly almost non-existant.
  • Pro: High-performance, sometimes hardware-accelerated
  • Pro: Wide format support

If you absolutely must have wide-format support and hardware-acceleration and stability, then this might be your best option. My cursory research found a few sparse hints/examples of using the library to decode video into a texture, but nothing resembling a full-blown tutorial. Expect to spend a lot of time doing trial-and-error.



Honorary mentions:

LibVLC
https://www.videolan.org/vlc/libvlc.html

  • Con: No cinder native code, and my cursory examination indicates that the library is more about giving you the option to integrate a built-in video player, rather than ‘easily’ manipulate/access individual frames on a texture.
  • Pro: High-performance, sometimes hardware-accelerated
  • Pro: Wide format support

I’m not sure this library allows easy access to individual video frames. Therefore I’ve put it under honorable mentions. I’m sure you can brute force your way in somehow, but likely at the cost of stability and head-aches.

HAP-Cinder

  • Con: As far as I know, this doesn’t actually give you a frame in an OpenGL texture and is therefore technically not a viable option, unless you can figure out how to access the DirectShow output.
  • Pro: Cinder native implementation exists
  • Pro: High-performance, hardware-accelerated
  • Con: Video files must be in HAP format

I think this library does give more ‘easy’ access to frames, but not in a native OpenGL context, thus it ends up an honorable mention.

This may be a silly question @lithium, but I’m assuming that your own work with the HPV framework isn’t available somewhere for public consumption…?

Cheers,
Gazoo

2 Likes

Great writeup, thanks for going to the effort of finishing up a thread, not enough people do that.

Unfortunately I can’t share the production code because of (bullshit) reasons, but I have dug up the test code from when i was evaluating video options earlier on. It’s very similar to the end result, minus a few conveniences (automatic init, update, shutdown etc) and a lot of tidy up, but should be enough for you to test with.

2 Likes

Much appreciated @lithium. I do worry that I don’t contribute enough to the community, so I especially appreciate the pat on the back when I feel I’ve done a tiny thing to try and help others.

I have full understanding for not being able to share the production code, and am deeply grateful for the demo code you have shared. That’s seriously awesome @lithium. Thank you so much. I’ll have a look immediately.

If you’re still using stimulant/Cinder-WMFVideo, you should be aware that years of fixes have since been made in the potion fork and everyone should be using that, which has been suggested many times on this forum. If I had to guess, I’d say at least a dozen production installations are living out there that use this player (of which I’ve worked on three) and although it has it’s shortcomings, crashing is not one of them (to my knowledge). At least not except in rare cases that haven’t been reproducible.

If you are seeing crashes, I’d suggest reporting it to that repro with ways to reproduce - the devs at Potion are still actively using it.

Other than this, the obvious answers are ffmpeg or gstreamer, both of which present new issues, so choose your battles. Some adventurous souls even decide to write their own video players. :stuck_out_tongue:

I’d say at least a dozen production installations are living out there that use this player

I wrote a digital jukebox that is using it in hundreds of locations, so you can add those to your list :wink:

1 Like

I took your sage advice @rich.e and @lithium and swapped out my current Cinder-WMFVideo source with the latest potion fork, and sure enough the exceptions/crashes vanished. Either my source was just out of date, or I’d meddled too much in mine. Either way, exceedingly grateful for the suggestions.

To anyone else keen on working with the latest potion fork in the latest Visual studio and C++17, you’ll likely have to change this:

const Node *pNode
to this
typename const List<T*>::Node *pNode

… in a few places.

If I get some spare time I’ll make a pull request on the fork!

2 Likes

@Gazoo could you elaborate on the changes you had to make? I’m trying to get WMFVideo to run on the latest builds of Windows, and I’m getting a number of errors in TinyMap.h and linklist.h that all seem to be related to what you’ve alluded to here.

I’m also finding that "glload/wgl_all.h" can’t be found – not sure if you saw this issue as well…

@morphogencc - Given that it’s 9 months ago, my memory is a tad hazy, but I like to think I detail the all the most non-obvious changes necessary. In other words, I’m not sure there’s anything more I can elaborate on as the change I recommend is pretty much as explicit as it can get, e.g. exchange:

const Node *pNode
to
typename const List<T*>::Node *pNode

I can’t current speak to the header being missing, but I do know there’s an impending ‘large scale’ windows update incoming for me. It may well update change headers. Is your windows version later than this:

10.0.19041 Build 19041

Cheers,
Gazoo

I was finally able to get it working – looks like there were a few places where an explicit typename was necessary and it looks like Cinder now uses glad/glad_wgl.h to load the interops; I went ahead and made all of the changes and pushed it here: https://github.com/sitara-systems/Cinder-WMFVideo