OpenCV flip - Cinder-KCB (SDK 1.8 Kinect Xbox360 )


#1

Hello I’ve been trying to use cv::flip to flip the color image but the result appears all black.

The exact same code works fine with static images and also with the Capture Basic sample.

Since the updating of the data in Cinder-KCB is happening on its own thread, I tried using a std::lock_guardstd::mutex but it didn’t make any difference at all.

I also checked the minimum and maximum values on the flipped cv::Mat and they change when moving the kinect and are in the range 0 - 255.

Also, before I draw the TextureRef I check if it evaluates to True and it is…

I’ve been stuck on this and cannot seem to figure out whats going on…

Does anyone have any suggestions??

Thanks!

Edit: Adding code

void kcbFlipImageApp::setup()
{
	mDevice = MsKinect::Device::create();
	mDevice->connectEventHandler([&](MsKinect::Frame frame)
	{
		std::lock_guard<std::mutex> lg(mMutex);
		if (frame.getColorSurface()) {
			Surface8uRef mColor = frame.getColorSurface();
			cv::Mat flipped;
			cv::flip(toOcv(*mColor), flipped, 1);
			if (mTextureColor) {

				mTextureColor->update(Surface8u(fromOcv(flipped)));
			}
			else {
				mTextureColor = gl::Texture::create(Surface8u(fromOcv(flipped)));
			}
		}
	});
	MsKinect::DeviceOptions options;
	mDevice->start(options);
}

void kcbFlipImageApp::draw()
{
	std::lock_guard<std::mutex> lg(mMutex);
	gl::viewport( getWindowSize() );
	gl::clear();
	gl::setMatricesWindow( getWindowSize() );
	
	gl::translate( 0.0f, getWindowCenter().y * 0.5f );
	Rectf bounds = Rectf( getWindowBounds() ) * 0.5f;
	if ( mTextureColor ) {

		gl::draw( mTextureColor, mTextureColor->getBounds(), bounds );
	}

}

#2

I’m on mac so i can’t run your code, but cinder provides its own flipping routines in cinder/ip/Flip.h. Perhaps you can try those just to rule out that it’s not some peccadillo of opencv’s?

*edit

Re-reading your post, you won’t be able to create opengl objects (specifically textures in this case) on a secondary thread without creating a shared context. There’s a few ways to go about syncing with the main thread for your opengl calls, so don’t take this method as gospel, but I’m pretty sure it’s a solution to the problem you’re seeing. Importantly this has no idea how the kinect library works internally so it’ll be up to you to make sure there’s no thread safety issues elsewhere, as well as things like writing directly into the frame’s color surface, etc.

mDevice->connectEventHandler([&](MsKinect::Frame frame)
{
    if ( frame.getColorSurface() )
    {
        Surface8uRef mColor = frame.getColorSurface();
        ip::flipVertical(mColor.get());
        
        app::App::get()->dispatchAsync([&mTextureColor, &mColor]
        {
            if (mTextureColor)
            {
                mTextureColor->update(*mColor.get());
            }
            else
            {
                mTextureColor = gl::Texture::create(*mColor.get());
            }
        });
    }
});

Just written without compiling so there might be a few problems, but it should be ok. You won’t need to lock the mutex in your draw call anymore as all touching of mTextureColor now happens on the main thread.

Hope it helps,

A.


#3

@lithium : would this not lead to a race condition?


#4

it probably will because looking at it again it’s probably a safe assumption the Surface returned by frame.getColorSurface(); is reused internally by the kinect library. I tried to weasel my way out of responsibility for that but can’t put a thing past you Paul. :wink:

@okinp once you’ve confirmed that the secondary thread is indeed what’s causing your textures to show up black, i suggest you have a look at some more robust multithreaded opengl code like the FlickrTestMultithreaded sample that comes with cinder for how to proceed from there.

A.


#5

Thanks a lot @lithium & @paul.houx !!

I tried using ip::horizontalFlip and it kind of works… but I found a bug in Cinder.

Surface->hasAlpha() returns false for SurfaceChannelOrder::BGRX (mCode = 5) which makes the ip::horizontalFlip use an increment of 3 bytes instead of 4.

So basically wherever in Cinder “hasAlpha” is used to decide what the byte increment should be, for specific SurfaceChannelOrders ( I suspect RGBX, BGRX, XRGB, XBGR) there will be a mistake.

Thanks again!

edit: I just posted the bug on a separate thread: