Mixed 2d and 3d drawing with Cinder-WMFPlayer

CinderBlocks
1

Hey all –

I’m trying to use the popular WMFPlayer block to playback video and texture map objects with video. However, I’m finding that drawing the video on top of a 3d scene (to show what the source material looks like) is slightly problematic.

My mixed drawing looks like:

void cinderTestApp::draw() {
	gl::clear( Color( 0, 0, 0 ) );

	gl::enableDepthRead();
	gl::enableDepthWrite();

	mCamera.lookAt( ... );
	ci::gl::setMatrices(mCamera);

	{
            auto lambert = gl::ShaderDef().lambert();
	    auto shader = gl::getStockShader(lambert);
	    shader->bind();

	    for (auto box : mBoxes) {
		gl::color(255, 255, 255);
		box->draw();
             }
	}

	gl::disableDepthWrite();
	gl::disableDepthRead();

	gl::setMatricesWindow(ci::app::getWindowSize());
        mWMFVideoPlayer->draw(0, 0);
}

With this code, I just get a white rectangle the size of the video drawn on top of my 3d scene.

If I replace the last line with a gl::draw call with a reference texture, that draws fine:

	gl::draw(gl::Texture::create(*mReferenceTexture)); // loaded as a surface with ci::loadImage()

I even thought that maybe there was something from the 3d calls that wasn’t falling out of scope, so I tried drawing the video to a FBO and then drawing the FBO on top of 3d scene:

void visualizationTestApp::update() {
	if (mVideoPlayer) {
		mVideoPlayer->update();
		std::cout << "Video position : " << mVideoPlayer->getPosition() << std::endl;
	}

	{
		gl::ScopedFramebuffer fboScope(mFbo);
		gl::ScopedViewport scpVp(ivec2(0), mFbo->getSize());
		gl::clear(Color(1, 1, 1));

		gl::ScopedColor white(1, 1, 1);
		//mVideoPlayer->draw(0, 0, mFbo->getWidth(), mFbo->getHeight());
		gl::draw(gl::Texture::create(*mReferenceTexture));
	}
}

and then drawing the FBO in draw(); but even within the FBO, the gl::draw call draws the texture properly, but WMFPlayer doesn’t render the image properly!

Can anyone help me understand what’s going on here?

thanks!

2

Hi,

Can you draw the viideo texture on the screen without any 3d?

-Gabor

3

Yes, I can. It draws just fine if my draw loop only contains ciWMFPlayer::draw. In fact, it’s only when I get into the drawing of the 3d where the issue occurs…

…and, trying a few things, it looks like if I switch to using a scoped shader, then it fixes the problem.

The working draw loop is:

void visualizationTestApp::draw() {
	gl::clear( Color( 0, 0, 0 ) );

	gl::enableDepthRead();
	gl::enableDepthWrite();

	mCamera.lookAt( ... );
	ci::gl::setMatrices(mCamera);
	
	{
	    gl::ScopedGlslProg shader(gl::getStockShader(gl::ShaderDef().lambert()));
	    gl::ScopedColor(1,1,1);

            for (auto box : mBoxes) {
                gl::color(255, 255, 255);
	        box->draw();
            }
	}

	gl::disableDepthWrite();
	gl::disableDepthRead();

	gl::setMatricesWindow(ci::app::getWindowSize());
	gl::draw(mFbo->getColorTexture(), Rectf(0, 0, 320, 240)); // draw from Fbo
	//mVideoPlayer->draw(0, 0); // this also works
}
4

Good that you have found the solution. Now that you have edited the post with the 3d drawing part of the code it shows that you left the lambert shader bound. which won’t work with the texture because it does not have normals.

5

Something to be aware of, the WMF video player uses the WGL_NV_DX_interop2 extension to share a texture from DirectX with OpenGL. This means that the texture needs to be locked when drawn, which is handled by the ciWMFPlayer::draw function, but not when you’re drawing it directly with cinder. My solution has been to write a FrameLease class that holds the lock on the texture and releases it when it goes out of scope. Something like:

class FrameLease
{
public:
    FrameLease                  ( VideoPlayer& player ) : _player(player) { _player.Lock(); }
    ~FrameLease                 ( ) { _player.Unlock(); };
    
    operator bool               ( ) const { return _player._currentFrame != nullptr; }
    operator ci::gl::TextureRef ( ) const { return _player._currentFrame; };
    
protected:
    
    VideoPlayer&                _player;
};

I’ve written a small abstraction layer over the WMFPlayer and cinder’s native AVF based player on mac which provides a common interface because WMFPlayer inexplicably didn’t follow the API, but that’s where the Lock/Unlock calls come into it. The VideoPlayer has a GetCurrentFrame method that returns a FrameLease, which you can directly gl::draw(). Anyway this is one possible solution to the issue you’re seeing, and this way you don’t have to think to much about how you’re using the texture or where it came from.

A

6

@lithium interesting – doesn’t the latest version of WMFPlayer correct this? It looks like in the ScopedVideoTextureBind class, the constructor and destructor use:

ciWMFVideoPlayer::ScopedVideoTextureBind::ScopedVideoTextureBind( const ciWMFVideoPlayer& video, uint8_t textureUnit )
	: mCtx( gl::context() )
	, mTarget( video.mTex->getTarget() )
	, mTextureUnit( textureUnit )
	, mPlayer( video.mPlayer ) {
	mPlayer->mEVRPresenter->lockSharedTexture();
	mCtx->pushTextureBinding( mTarget, video.mTex->getId(), mTextureUnit );
}

ciWMFVideoPlayer::ScopedVideoTextureBind::~ScopedVideoTextureBind() {
	mCtx->popTextureBinding( mTarget, mTextureUnit );
	mPlayer->mEVRPresenter->unlockSharedTexture();
}

Which seems to lock and unlock the texture. Is there something additional that needs to be done to make sure the texture can be drawn outside of the player’s draw() call?