Debugging a seemingly random drop in framerate playing video

Using Cinder
1

Hi All,

I’ve run out of ideas on how to debug the problem I’m experiencing and was hoping someone in the forum could shed some light.

I’m essentially drawing two video sources alongside one another. For one of the sources i’m using ciWMFplayer and the other isn’t a video per se, but rather a vector of Surfaces captured from a camera at a rate of 50fps. I’ve disabled verticalSync and set my app’s frameRate at 50fps. Every frame ciWMFplayer is left to update itself, and I simply advance the index which I use to reference the surface from the vector. Each source has a custom GlslProg that lets me add some basic effects. An extract of the code is below:

            // inside the init() method...
            gl::QueryRef    mQuery;
            mQuery = gl::Query::create(GL_TIME_ELAPSED);

            // inside the render() method...
            mQuery->begin();

            ci::gl::ScopedViewport scpVwPt(ci::app::getWindowSize());

			// basically pause the video when it ends
			if (mFrameIndex > frames.size() - 1) mFrameIndex = frames.size() - 1;

			// draw the black and white video
			{
                ci::vec2 resBw = ci::vec2(mVideo.getWidth(), mVideo.getHeight());
				ci::gl::ScopedGlslProg bwScp(mBlackAndWhiteGlsl);
				mBlackAndWhiteGlsl->uniform("uMovieTexture", 0);
				mBlackAndWhiteGlsl->uniform("uResolution", resBw);
				ciWMFVideoPlayer::ScopedVideoTextureBind scpTex(mVideo, (uint8_t)0);
				ci::gl::drawSolidRect(rightVideoRectf);
			}

			// draw the standard video from the sequence of frames
			{
                ci::vec2 resStd = ci::vec2(*frames[mFrameIndex]->getWidth(), *frames[mFrameIndex]->getHeight());
				ci::gl::ScopedGlslProg glscp(mStandardGlsl);
				mSideBySideGlsl->uniform("uFrameTexture", 0);
				mSideBySideGlsl->uniform("uResolution", resStd);
				ci::gl::Texture::Format fmt;
				fmt.loadTopDown(false);
				auto mTex = ci::gl::Texture::create(*frames[mFrameIndex], fmt);
				ci::gl::ScopedTextureBind scpTex(mTex, (uint8_t)0);
				ci::gl::drawSolidRect(leftVideoRectf);
			}

            mQuery->end();

            console() << "Query value: " << (double)(mQuery->getValueInt64()) / 1000000 << "ms" << std::endl;

Most of the time this works without issues, but on odd occasions, maybe 15-20% of the time, the framerate will drop from a steady 50fps to ~33fps. What’s odd is that it seems to affect the two different playback styles differently. The playback using frames finishes as expectedly, but the ciWMFplayer seems to go in slow motion, at some point the framerate overcompensates and jumps higher to around 70fps, seemingly to get the video to finish using the correct duration of the file.

I can’t seem to manual reproduce the problem, and have delved into using ci::gl::Query to try and see if it’s an issue on the GPU side, but I can’t glean information that helps me identify the problem ( The gl::Query is a recent addition and the issue was present before this was added).

What I find strange is that sometimes it works exactly as expected, and others not. I’ve reworked several parts of the app to reduce the number of draw calls and see no reason why it should be suffering fps issues! Has anyone encountered similar issues in the past?

App is running on MSW10 with a Nvidia 1060.

Thanks!

2

Can you share your encoder settings for the videos? We’ve had success by ensuring that our videos are constant bitrate instead of variable and generally adjusting key frames and compression did a lot for smoother playback. Check out this thread with a possibly related conversation: Delay / lag / stall with WMF Video Player

3

Hey Ben,

Thanks for the swift response. All the videos are created using ffmpeg via nodejs server running on a separate machine. See the JS snipped below for ffmpeg args.

let imageSequence = filename + '_%05d.jpg';

const defaults = {
  cwd: mediaPath // set the working directory for the worker (ie. where the images live)
};

		let child = execFile('ffmpeg', ['-f', 'image2', '-framerate', '50', '-i', imageSequence, '-b:v', '1000000000', '-y', outputPath], 
			defaults, 
			(error, stdout, stderr) => {
		  if (error) {
		    console.error(error)
		    resolve(false)
		  }
})

I’ll dig into all the things mentioned in the other post - hopefully it’s just an encoding issue.

4
  • Do you also see the same behavior when running just one of the video’s (either the ciWMF one or the other)?
  • Have you tried queuing frames in advance? You’re uploading a Surface and then use it right away. This will stall the pipeline until the memory transfer is complete. You can avoid that by creating the texture a few frames in advance and upload it using a Pbo. For an explanation, see this.

-Paul

1 Like
5

Thanks Paul. Hopefully the encoding is causing the issue to have impact the fps so dramatically, but I’ll implement the Pbo for good measure!