Render Process Terminated Debugging

[akjana]

any suggestions on how to gather enough information to efficiently debug render process terminations ?

On my local machine i can attach the debugger and debug. Whats the best way to gather info on remote machines ?
Consider the case where GPU/Renderer runs out of memory as an example.

[magreenblatt]

Set up a crash reporting system. See https://bitbucket.org/chromiumembedded/cef/issues/1131/ for details.

[akjana]

Thanks. I'll try it out.

[akjana]

Hi again

Enabling breakpad started generating nice dumps but only in my user folder/AppData/Local/Temp. Passing the --crash-dumps-dir and --dumps-dir didn't work. Which is the correct setting?
I tested it with CefSharp with Version 43. With CefSharp both the flags were not passed as command line arguments to the sub processes, is there any logic in cef/chromium which checks for only valid commandline arguments? Perhaps i'm using the wrong command line flag.

I couldn't try the crash reporting feature with cefclient.exe by navigation to chrome://crash while starting it with the command line flags "--enable-crash-reporter=1 --crash-dumps-dir=escapedPath".

Also noticed that the info gets written out the named pipe. Whats the default named pipe name ? Does the stack trace get written there?

Thanks

[akjana]

Also I didn't enable sandboxing yet. Is there a wiki/guide on enabling sandbox.

Thanks

[magreenblatt]

See the Chrome code linked from the above issue for how the named pipe works.

[akjana]

I'm going through the code and will take the time to understand things. If anyone has some time, are there any known limitations of enabling breakpad on windows machines ?
As i understand, "crash-dumps-dir" works only on linux and sandboxing needs to be enabled for getting correct stack traces.
Are there any other restrictions/limitations, does breakpad catch all exceptions/cases where render process gets terminated ?

To be specific, I saw this piece of code and got worried, i'm yet to study the whole code"
#ifndef _WIN64
static NTSTATUS WINAPI HookNtTerminateProcess(HANDLE ProcessHandle, ExitStatus)

[akjana]

I finally went through the code, breakpad documentation and fair understanding of how things work i think and how i could integrate it but I am seeing one issue.

While testing out cefclient.exe --enable-crash-reporter and content_shell_crash_service.exe, the dump gets generated for me on both x86/x64 windows platform but the dump's crashed thread contains only this entry as the stack trace.
> ntdll.dll!770e5dae() Unknown
[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]

When i load symbols, it reduces to > ntdll.dll!__RtlUserThreadStart@8() Unknown

This is quite different from what was observed before at https://groups.google.com/a/chromium.org/forum/#!msg/chromium-dev/tJ48PfabcC8/EPqIOqa60bEJ and http://magpcss.org/ceforum/viewtopic.php?f=6&t=13990.
Any ideas what is happening ? any suggestions on what to try out.

[akjana]

I was making cefclient crash by navigating to chrome://crash and I tested cefclient with(by removing the compile flag CEF_USE_SANDBOX)/without sandbox on branch 2526.

[akjana]

Looks like non-official builds of Chromium will do in process stack printing on windows per below code for version 3.2526.1362.
This explains why breakpad's Out of Process dumping mechanism didn't work. With the line 203 commented, things work fine in cefclient for me in x64 and x86. Can anyone else facing similar issues confirm this ?
This finding also partially explains https://groups.google.com/a/chromium.org/forum/#!msg/chromium-dev/tJ48PfabcC8/EPqIOqa60bEJ which I was observing till now as well. Not sure if this should be patched in CEF/fixed in Chromium.

However, i have noticed that in a case where render process went Out of memory, dump wasn't generated.

Code: Select all

//line 203 of content_main_runner.cc
#if !defined(OFFICIAL_BUILD)
  // Print stack traces to stderr when crashes occur. This opens up security
  // holes so it should never be enabled for official builds.
  base::debug::EnableInProcessStackDumping();
#if defined(OS_WIN)
  base::RouteStdioToConsole(false);
  LoadLibraryA("dbghelp.dll");
#endif
#endif

// Stack_trace_win.cc line 192
bool EnableInProcessStackDumping() {
  // Add stack dumping support on exception on windows. Similar to OS_POSIX
  // signal() handling in process_util_posix.cc.
  g_previous_filter = SetUnhandledExceptionFilter(&StackDumpExceptionFilter);

  // Need to initialize symbols early in the process or else this fails on
  // swarming (since symbols are in different directory than in the exes) and
  // also release x64.
  return InitializeSymbols();
}

// Stack_trace_win.cc line 34
// Prints the exception call stack.
// This is the unit tests exception filter.
long WINAPI StackDumpExceptionFilter(EXCEPTION_POINTERS* info) {
  debug::StackTrace(info).Print();
  if (g_previous_filter)
    return g_previous_filter(info);
  return EXCEPTION_CONTINUE_SEARCH;
}