Extracting Frame from Cinder-WMFVideo

CinderBlocks
1

I’m trying to extract individual frames from Cinder-WMFVideo so I can run some CV algorithms on each frame. I’m a bit shaky on how exactly the WMF Decoder works, but it looks like generally the decoding is happening on a separate thread (hence the need for the Shared Texture Lock) and then using an interop to share the texture between DirectX and OpenGL. Which is great for drawing textures, but not so helpful for pixel operations on the CPU.

My first thought was just to add a function to the player to extract the source:

ci::ImageSourceRef ciWMFVideoPlayer::getImageSource() {
	mPlayer->mEVRPresenter->lockSharedTexture();
	ci::ImageSourceRef source = mTex->createSource();
	mPlayer->mEVRPresenter->unlockSharedTexture();
	return source;
}

but this just returns an ImageSource with no data – the correct rows and columns, but the data pointer is empty.

I suspected that I was having issues with the texture being shared, so I tried copying the texture first, and then creating a source from:

ci::ImageSourceRef ciWMFVideoPlayer::getImageSource() {
	mPlayer->mEVRPresenter->lockSharedTexture();

	int w = mTex->getWidth();
	int h = mTex->getHeight();
	gl::Texture2d::Format format;
	format.setInternalFormat(mTex->getInternalFormat());

	ci::gl::TextureRef copyTex = gl::Texture::create(w, h, format);

	glCopyImageSubData(mTex->getId(), mTex->getTarget(), 0, 0, 0, 0,
		copyTex->getId(), copyTex->getTarget(), 0, 0, 0, 0,
		w, h, 1);

	mPlayer->mEVRPresenter->unlockSharedTexture();
	return copyTex->createSource();
}

And this works, but sure seems to be sloooow – it’s the second-slowest part of my code (after cinder::toOcv()).

Is there a better way to do this?

2

The main benefit of the hoops you have to jump through with media foundation custom presenters is performance by preventing any CPU readback. As soon as that no longer applies (i.e you need the data CPU side) I’d be ditching it in a heartbeat.

It’s pretty straight forward to get video samples using the Source Reader at the expense of having to do the COM dance for a while.

If you’re gung-ho about using the existing player, one thing that’s a potential issue I see with your original code is that I have a feeling ImageSource is lazily evaluated, meaning until another object consumes it, it doesn’t actually do anything, similar to how loadUrl on its own won’t do anything until it’s wrapped in a loadString or equivalent. If you make sure the copy is performed while the lock is still held you might have more luck. I suspect it’ll still be slow, but maybe not as slow :wink:

ci::Surface8uRef ciWMFVideoPlayer::getSurface ( ) const 
{
    mPlayer->mEVRPresenter->lockSharedTexture();
    ci::Surface8uRef result = ci::Surface8u::create ( mTex->createSource() );
	mPlayer->mEVRPresenter->unlockSharedTexture();
	return result;
}

Written inline and not tested, but should roughly resemble what I mean.

1 Like
3

I was worried that the answer was going to be that WMF is the wrong tool; I guess I was just hoping I could be lazy since it’d the best-working and best-documented player right now :sweat_smile:

As a note on your suggestion – I gave it a shot, but just like mt first example, it returns an empty Surface with no mData member. I’m not quite sure why copying the texture first and then extracting the surface works, but extracting the surface during the lock doesn’t – do you have any idea why that might be?

4

I would guess it’s a timing issue. There’s all kinds of nonsense going on behind the scenes with sharing the texture between directx and opengl and the whole player is asynchronous too. I bet if you put a fence (i.e gl::Sync) in the right place you’d be able to grab the data but inserting a copy into the command stream is potentially kicking the driver into waiting for the texture to be ready. My guess is if you ran this on an integrated GPU it’d probably work because the texture data isn’t being thrashed around as much.

5

Just had a little fiddle as a lunch time project and was able to get something working. Here’s a modified PresentEngine.cpp from the current fork of Cinder-WMFVideo. You’ll also need to declare IDirect3DSurface9 *d3d_system_surface; in PresentEngine.h, I just whacked it underneath the existing IDirect3DSurface9.

This just adds an extra copy of the render target to a system side memory surface, which is in turn loaded into a Surface8uRef. I did some really hacky shit to push that surface to the main app / thread just to verify it works, but I’ve left the actual plumbing up to you. There’s some potentially tricky threading / lifetime issues that you’ll need to be aware of, but the gist of what you need is there. I was able to run at 60fps using a 1080x1920 video that i ran a ci::ip::edgeDetectSobel on just to verify i was able to monkey with a CPU side version of the image.

Hopefully gets you somewhere closer to where you need to be.

A

3 Likes
6

Holy cow! Thanks so much mate; this is a great head start and I think I can handle it from here :slight_smile: I’m just not so terribly familiar with the DirectX guts of the plugin to know where to start… thanks again, this is great!

7

You’re welcome. Just FYI I finally said enough is enough last night and sat down to write a whole new video player based on IMFMediaEngine that i’ll be turning into a cinderblock and pushing to github over the weekend.

IMFMediaEngine has a fast path with DXGI that I haven’t implemented yet but the “slow” path (which still seems quite quick) works with CPU side buffers so will probably be up your alley. I’ll post a new thread when it’s ready.

1 Like