Crash Reporting

Does anyone have a method they like for cross platform crash reporting? I’ve got an app deployed in the wild and I’d love to automate crash feedback.

C

Been using ampm along with in-app log statements (file logging + timestamps enabled) for this purpose myself, although there are some services that will let you send off a crash dump and (presumably) debug it with your local toolset.

One key for me has been some cpp trickery to make asserts log to file, so if something like a vector or map blows up, I can still get a stack trace.

We have been using PM2. it has some nice memory statistics, but you need to be careful with process priority configuration. An app was running slower when lunched from pm2, and we had to dig around a bit to find the right config.

@rich.e, can you elaborate a bit on the c++ trickery? that sounds super useful!

1 Like

PM2 looks really nice! I think that with a little wrapper that could do the heartbeat / google analytics things that AMPM does will be something I look into in the future.

About the C++ asserts to log trick: it’s basically a combination of a few pieces:

  1. Override the CinderMswUserDefines.h file as described here and add the lines:
//! This allows us to define our own assertion handlers (in psCommon.cpp)
#define CI_ENABLE_ASSERTS
#define CI_ENABLE_ASSERT_HANDLER
  1. Add some custom assert functions that only log at the FATAL level but don’t abort:
// Our own handlers for assertions which print fatal messages instead.
// CI_LOG_F() will cause the breakpoint to be hit when in non-production mode.
namespace cinder {
//! Called when CI_ASSERT() fails
void assertionFailed( char const *expr, char const *function, char const *file, long line )
{
	CI_LOG_F( "*** [nw] Assertion Failed *** | expression: ( " << expr << " ), location: " << file << "[" << line << "], " << function );
}

//! Called when CI_ASSERT_MSG() fails
void assertionFailedMessage( char const *expr, char const *msg, char const *function, char const *file, long line )
{
	CI_LOG_F( "*** [nw] Assertion Failed *** | expression: ( " << expr << " ), location: " << file << "[" << line << "], " << function << "\n\tmessage: " );
}

} // namespace cinder
  1. Enable file logging and also the LoggerBreakpoint with at least a level of FATAL:
log::makeOrGetLogger<log::LoggerFileRotating>( "logs", "YourApp.%Y.%m.%d_%H.%M.%S.log" );
log::makeOrGetLogger<log::LoggerBreakpoint>()->setTriggerLevel( log::Level::LEVEL_FATAL );

On Posix this is a tad easier, you can skip the CinderMswDefines.h step and just build everything on the command line with those same additional #defines.

After this, if you get an assertion failure in either Debug or Release, you’ll first get a message logged to file, and then a break into the debugger. When you are running outside of a debugger, you’ll just get the message logged to file after which the routine will carry on as if there was no assertion failure.

This covers you for lots of things like a std::vector out of bounds and asserting, which would other wise end in the app log saying “we aborted”, or maybe nothing at all. The other thing I’ve been trying to catch and (safely) log are things like SIGFPE or SIGABRT, but haven’t yet figured that out…

cheers,
Rich

Just stumbled upon this as we’re also looking to switch from AM/PM to PM2. Could you elaborate or share some of your launch parameters from PM2?

Hey there Ben,

We mainly run pm2 using the javascript API, but in some cases also from the command line. For instance this is the line that starts the server via pm2 (which in turn starts a bunch of other things like cinder apps, webapps etc):

set LOG_FILE="logs/pm2/system_control_monitor.log"
pm2 start src\Server.js --name "System Control Monitor" --output %LOG_FILE% --error %LOG_FILE% --log-date-format "YYYY-MM-DD HH:mm:ss.SSS Z"

The %LOG_FILE% lines are so that error and regular lines are concatenated, by default they end up in separate files which I find to be annoying and difficult for tracking down what was going on at the time of an error. Instead, we have a different step that scans all the log files across all systems and checks for error messages, dumps those to a single file that we can display in a webapp or email somewhere.

cheers,
Rich

1 Like

A quick follow up on Cinder + PM2: We had issues getting PM2 to properly launch Cinder apps. They would start w/o a taskbar icon and would receive any mouse events (those would just pass through to apps in the background).

Adding the following lines to our apps fixes this, but I can’t for sure say why:

#include <Windows.h>
// ...
void MyApp::setup() {
  ::ShowWindow((HWND)getWindow()->getNative(), SW_SHOW);
}

Everything else seems to work pretty well now, but it does feel like this is a shaky foundation. Has anyone else had success with PM2 and Cinder? Does anyone have an idea why we explicitly have to call ::ShowWindow()?

FWIW, I identified that line since toggling fullscreen on/off or launching into fullscreen seemed to fix the issue too. We haven’t seen any of these issues with Unity and Chrome.

Damn, just saw your reply @rich.e! For some reason I didn’t get a notification. Thanks for the info; have you had any issues with your Cinder apps not receiving focus?

Following up here after figuring out why PM2 doesn’t launch apps in windowed mode: They set the windowsHide flag in Node’s exec() function to true (apparently to avoid popups in Windows).

There was a PR that’s been merged into PM2 to make this configurable, but it’s completely undocumented. It looks like @gschomburg you came across this thread too last year and pointed that out to them :raised_hands:

Solutions

For anyone curious, the TLDR seems to be that if you want to launch a Cinder/Unity/… app via PM2, you have to do one of the following:

  1. Set windowsHide=false in your PM2 app config (recommended)
  2. Call ShowWindow() in your app to force show your window
  3. Launch your app in full screen mode (not always an option)

Examples

Example app config:

// ecosystem.config.js
module.exports = {
  apps: [
    {
      name: "my-app",
      script: "my-app.exe",
      windowsHide: false
    }
  ]
}

Cinder/C++ example for calling ShowWindow():

#include <Windows.h>
// ...
void MyApp::setup() {
  ::ShowWindow((HWND)getWindow()->getNative(), SW_SHOW);
}

Unity/C# example for calling ShowWindow():

using System;
using System.Runtime.InteropServices;
public class MyClass : MonoBehaviour {
    [DllImport("user32.dll")] static extern uint GetActiveWindow();
    [DllImport("user32.dll")] static extern bool ShowWindow(IntPtr hWnd, int nCmdShow);
    void Awake() {
        // See https://docs.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-showwindow
        ShowWindow((IntPtr)GetActiveWindow(), 5);
    }
}

Ben

2 Likes