# Extracting Frame from Cinder-WMFVideo
**URL:** https://discourse.libcinder.org/t/extracting-frame-from-cinder-wmfvideo/1848
**Category:** CinderBlocks
**Created:** [August 16, 2021, 2:28am UTC](https://discourse.libcinder.org/t/extracting-frame-from-cinder-wmfvideo/1848 "2021-08-16T02:28:07Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@morphogencc](https://discourse.libcinder.org/u/morphogencc)
#### Post date: [August 16, 2021, 2:28am UTC](https://discourse.libcinder.org/t/extracting-frame-from-cinder-wmfvideo/1848/1 "2021-08-16T02:28:07Z")
I’m trying to extract individual frames from Cinder-WMFVideo so I can run some CV algorithms on each frame. I’m a bit shaky on how exactly the WMF Decoder works, but it looks like generally the decoding is happening on a separate thread (hence the need for the Shared Texture Lock) and then using an interop to share the texture between DirectX and OpenGL. Which is great for drawing textures, but not so helpful for pixel operations on the CPU.
My first thought was just to add a function to the player to extract the source:
```auto
ci::ImageSourceRef ciWMFVideoPlayer::getImageSource() {
mPlayer->mEVRPresenter->lockSharedTexture();
ci::ImageSourceRef source = mTex->createSource();
mPlayer->mEVRPresenter->unlockSharedTexture();
return source;
}
```
but this just returns an ImageSource with no data – the correct rows and columns, but the data pointer is empty.
I suspected that I was having issues with the texture being shared, so I tried copying the texture first, and then creating a source from:
```auto
ci::ImageSourceRef ciWMFVideoPlayer::getImageSource() {
mPlayer->mEVRPresenter->lockSharedTexture();
int w = mTex->getWidth();
int h = mTex->getHeight();
gl::Texture2d::Format format;
format.setInternalFormat(mTex->getInternalFormat());
ci::gl::TextureRef copyTex = gl::Texture::create(w, h, format);
glCopyImageSubData(mTex->getId(), mTex->getTarget(), 0, 0, 0, 0,
copyTex->getId(), copyTex->getTarget(), 0, 0, 0, 0,
w, h, 1);
mPlayer->mEVRPresenter->unlockSharedTexture();
return copyTex->createSource();
}
```
And this works, but sure seems to be sloooow – it’s the second-slowest part of my code (after cinder::toOcv()).
Is there a better way to do this?
---
### Author:  [@lithium](https://discourse.libcinder.org/u/lithium)
#### Post date: [August 17, 2021, 12:18am UTC](https://discourse.libcinder.org/t/extracting-frame-from-cinder-wmfvideo/1848/2 "2021-08-17T00:18:27Z")
The main benefit of the hoops you have to jump through with media foundation custom presenters is performance by preventing any CPU readback. As soon as that no longer applies (i.e you need the data CPU side) I’d be ditching it in a heartbeat.
It’s pretty straight forward to get video samples using the [Source Reader](https://docs.microsoft.com/en-us/windows/win32/medfound/source-reader?redirectedfrom=MSDN) at the expense of having to do the COM dance for a while.
If you’re gung-ho about using the existing player, one thing that’s a potential issue I see with your original code is that I have a feeling `ImageSource` is lazily evaluated, meaning until another object consumes it, it doesn’t actually do anything, similar to how `loadUrl` on its own won’t do anything until it’s wrapped in a `loadString` or equivalent. If you make sure the copy is performed while the lock is still held you might have more luck. I suspect it’ll still be slow, but maybe not _as_ slow 😉
```cpp
ci::Surface8uRef ciWMFVideoPlayer::getSurface ( ) const
{
mPlayer->mEVRPresenter->lockSharedTexture();
ci::Surface8uRef result = ci::Surface8u::create ( mTex->createSource() );
mPlayer->mEVRPresenter->unlockSharedTexture();
return result;
}
```
Written inline and not tested, but should roughly resemble what I mean.
---
### Author:  [@morphogencc](https://discourse.libcinder.org/u/morphogencc)
#### Post date: [August 17, 2021, 1:23am UTC](https://discourse.libcinder.org/t/extracting-frame-from-cinder-wmfvideo/1848/3 "2021-08-17T01:23:12Z")
I was worried that the answer was going to be that WMF is the wrong tool; I guess I was just hoping I could be lazy since it’d the best-working and best-documented player right now 😅
As a note on your suggestion – I gave it a shot, but just like mt first example, it returns an empty Surface with no mData member. I’m not quite sure why copying the texture first and then extracting the surface works, but extracting the surface during the lock doesn’t – do you have any idea why that might be?
---
### Author:  [@lithium](https://discourse.libcinder.org/u/lithium)
#### Post date: [August 17, 2021, 3:15am UTC](https://discourse.libcinder.org/t/extracting-frame-from-cinder-wmfvideo/1848/4 "2021-08-17T03:15:42Z")
I would guess it’s a timing issue. There’s all kinds of nonsense going on behind the scenes with sharing the texture between directx and opengl and the whole player is asynchronous too. I bet if you put a fence (i.e `gl::Sync`) in the right place you’d be able to grab the data but inserting a copy into the command stream is potentially kicking the driver into waiting for the texture to be ready. My guess is if you ran this on an integrated GPU it’d probably work because the texture data isn’t being thrashed around as much.
---
### Author:  [@lithium](https://discourse.libcinder.org/u/lithium)
#### Post date: [August 18, 2021, 6:41am UTC](https://discourse.libcinder.org/t/extracting-frame-from-cinder-wmfvideo/1848/5 "2021-08-18T06:41:41Z")
Just had a little fiddle as a lunch time project and was able to get something working. Here’s a modified [PresentEngine.cpp](https://gist.github.com/axjxwright/7271a465e666335251e384519d8aefc3) from the current fork of `Cinder-WMFVideo`. You’ll also need to declare `IDirect3DSurface9 *d3d_system_surface;` in `PresentEngine.h`, I just whacked it underneath the existing `IDirect3DSurface9`.
This just adds an extra copy of the render target to a system side memory surface, which is in turn loaded into a `Surface8uRef`. I did some really hacky shit to push that surface to the main app / thread just to verify it works, but I’ve left the actual plumbing up to you. There’s some potentially tricky threading / lifetime issues that you’ll [need to be aware of](https://gist.github.com/axjxwright/7271a465e666335251e384519d8aefc3#file-presentengine-cpp-L732-L736), but the gist of what you need is there. I was able to run at 60fps using a 1080x1920 video that i ran a `ci::ip::edgeDetectSobel` on just to verify i was able to monkey with a CPU side version of the image.
Hopefully gets you somewhere closer to where you need to be.
A
---
### Author:  [@morphogencc](https://discourse.libcinder.org/u/morphogencc)
#### Post date: [August 19, 2021, 7:22pm UTC](https://discourse.libcinder.org/t/extracting-frame-from-cinder-wmfvideo/1848/6 "2021-08-19T19:22:03Z")
Holy cow! Thanks so much mate; this is a great head start and I think I can handle it from here 🙂 I’m just not so terribly familiar with the DirectX guts of the plugin to know where to start… thanks again, this is great!
---
### Author:  [@lithium](https://discourse.libcinder.org/u/lithium)
#### Post date: [August 19, 2021, 11:46pm UTC](https://discourse.libcinder.org/t/extracting-frame-from-cinder-wmfvideo/1848/7 "2021-08-19T23:46:52Z")
You’re welcome. Just FYI I finally said enough is enough last night and sat down to write a whole new video player based on `IMFMediaEngine` that i’ll be turning into a cinderblock and pushing to github over the weekend.
`IMFMediaEngine` has a fast path with DXGI that I haven’t implemented yet but the “slow” path (which still seems quite quick) works with CPU side buffers so will probably be up your alley. I’ll post a new thread when it’s ready.