I’ve been developing a simple Cinder app which mostly renders post-processing fragment shaders with multiple passes. Part of the workflow is hot reloading the shader every time I edit the file so I can iterate without restarting the app. I noticed that fairly intermittently the rendering would freeze when I edit a shader. After closer investigation, it seems that it’s a GL_INVALID_OPERATION error which happens after I call drawSolidRect(). With the multiple passes I am doing quite a lot of texture / FBO binding and unbinding, but not sure why this error is so intermittent.
Any idea what is happening? Is there a way to get better debug information? What I did is basically sprinkle getError() calls through my rendering code to see what call exactly was creating an issue and this is how I got to drawSolidRect().
If you’re on windows you can create a debug context that will call you back or break when you do something iffy. You set this up in your renderer options, like so:
You can try running with the OpenGL Profiler attached and click “break on errors”. However the debugger is a piece of shit and rarely works properly. In fact I have to connect to my own machine as a remote debugging session to make it attach at all, but that’s the state of GL on mac. In other words, good luck.
The only reliable way I’ve been able to trace back gl errors on Mac is to sprinkle CI_CHECK_GL() statements after every single gl call in your project, until you get back to the culprit. Sometimes this takes adding them directly to libcinder gl code too… yep it’s a pain and a shame apple didn’t ever bother implementing the debug break features before ceasing OpenGL development. Lack of critical OpenGL features is the main reason why many of us stopped developing on OS X.
I did what you suggested and it seems the culprit is ctx->drawArrays( GL_TRIANGLE_STRIP, 0, 4 ) in drawSolidRect() in draw.cpp. Any idea why this would raise a GL_INVALID_OPERATION but only on a very very intermittent basis? Maybe a race condition somewhere?
Yeah that’s definitely a possibility since the issue happens very intermittently but always right after a hot reload. But as far as I know I’m not creating any thread myself manually. Any pointer on how to debug threading issues?
I guess one thing that could be happening is that there is a bad race condition where in some cases the render thread attempts to draw things while the thread where the hot reloading happens in still recompiling the new shader or loading the new textures. Is there a way to pause the render thread until the hot reloading is done for sure? That would obviously create a moment of freeze but I’d be ok with that if it can ensure I avoid this race condition. One option would be to use a mutex and lock the render thread until the hot reload finishes, but it doesn’t sound like a great idea to block the render thread like that as that could cause other problems?
I see. You mean that dispatchAsync guarantees that the function will finish executing before the next update is called? Both update and draw render on the same thread, one after the other?
It seems update and draw run sequentially on the main thread, so I’m not sure what dispatchAsync really does. In my case, the logic for the hot reloading happens as part of update, so everything should be ready to go by the time draw is called next which should eliminate the risk of race conditions? I’m clearly missing something. Thanks for your help!
dispatchAsync lets you push some execution to the main thread. One of the things cinder’s update cycle does is poll an io_service to see if there’s any jobs to run. Since OpenGL is inherently single threaded (for all intents and purposes), this is an easy way to have a monitoring thread look for changes, and then push the actual loading back to the main thread to appease the GL gods (may they burn in hell).
e.g: (written inline, not tested)
void YourApp::yourMonitoringThreadsCallbackFunc ( )
{
while ( true )
{
assert ( !app::isMainThread() ); // We're on a different thread
if ( someVertexShaderFileChanged || someFragmentShaderFileChanged )
{
dispatchAsync ( [=]
{
assert ( app::isMainThread() ); // We're on the main thread, so GL won't bitch
yourShader = gl::GlslProg::create ( loadAsset ( someVertexShaderFile ), loadAsset ( someFragmentShaderFile ) );
});
}
}
}
It seems from your comments that you are not using a separate thread for doing the hot-reloading part so not sure what is going wrong here since the update and draw functions are both called on the main thread which should have a valid GL context at all points.
In any case, doing file I/O is an ideal operation candidate for moving it in a separate thread and the path that @lithium described should be the way to go but consider using the built-in FileWatcher instead which does exactly that for you
Thanks @lithium@petros and @gabor_papp! I understand what dispatchAsync can be useful for now. Not quite sure what’s going wrong for me as I’m not doing anything off the main thread afaik, but will definitely look into FileWatcher. I was aware of Watchdog by @Simon but didn’t realize there was a built-in solution now.
Could you share how you check for the shader changes? Or maybe if you can reproduce the crash in a minimal example and share it we can help you more find the the cause of the crash.
Then, if a shader change is detected, the GlslProg is recreated, as well as some FBOs and Textures (I’m using this multipass shader syntax where a shader change can potentially mean that new FBOs and Textures are needed, which is why it’s not enough to just recreate the GlslProg). I’m assuming that all these operations are synchronous so since all this chain of events is kicked off from the update method which runs on the main thread, I’m expecting that everything will always be ready to go by the next time draw is called. But it does seem there is a race condition somewhere that I’m missing and that is causing this intermittent GL_INVALID_OPERATION. Let me know if you see something suspicious by looking at the code. If it’s too messy to look at I’ll go ahead and create a more minimal example.
In the meantime I’ll go ahead and try using FileWatcher instead of my custom code. Always happy to use the awesome Cinder code instead of my own whenever possible
Hey folks, sorry for the delay I took a little break with the new year celebrations A couple updates:
@gabor_papp I can still repro the issue when updateShaders is called from drawScene
It seems using FileWatcher instead of my custom code fixes the issue. There must be some threading code in there that gets rid of the race condition. This is a double win because it also means less custom code and more Cinder code yay! Thanks for suggesting that @petros
Thanks for your help everyone! If anyone has an insight on why my original approach was causing this intermittent issue don’t hesitate.
