Page 1 of 1

Need help with random shutdow crashes

PostPosted: Fri Jun 26, 2020 11:19 am
by rdh
I have CEF in our app where we have some dialogs hosting web pages. I have been seeing a random crash during shutdown even though I have scoured the code to make sure I set the cef app pointer to nullptr and the browser pointers to nullptr when closing down. Let me note that I am running my Visual Studio 2019 app in debug but I only link to the release cef lib file and use the release DLLs. Every time I have crashed, the stack shows an audio stream being stopped. I have a stack below and the crash occurs after I set my cef app pointer to nullptr and call CefShutdown. Stepping into the source, it is the actual call to the cef DLL shutdown api that results in the crash. Very hard to reproduce with my app and appears to be related to stuff running in a worker thread. I suspect I am just not closing down correctly. My sequence is the destroy my windows hosting the browser at which time I call the GetHost()->CloseBrowser(true) and then GetHost()->GetWindowHandle() and pass that to the Windows DestroyWindow api. Finally, I set the browser to nullptr. Perhaps the order is wrong?

libcef.dll!logging::LogMessage::~LogMessage() Line 953 C++
libcef.dll!cef_log(const char * file, int line, int severity, const char * message) Line 335 C++
jengined.dll!cef::logging::LogMessage::~LogMessage() Line 186 C++
> jengined.dll!shutdown_checker::AssertNotShutdown() Line 54 C++
jengined.dll!`anonymous namespace'::life_span_handler_on_before_close(_cef_life_span_handler_t * self, _cef_browser_t * browser) Line 198 C++
libcef.dll!CefAudioHandlerCToCpp::OnAudioStreamStopped(scoped_refptr<CefBrowser> browser) Line 113 C++
libcef.dll!CefBrowserHostImpl::DestroyBrowser() Line 1537 C++
libcef.dll!CefBrowserInfoManager::DestroyAllBrowsers() Line 352 C++
libcef.dll!CefContext::FinishShutdownOnUIThread(base::WaitableEvent * uithread_shutdown_event) Line 680 C++
[Inline Frame] libcef.dll!base::OnceCallback<void ()>::Run() Line 98 C++
libcef.dll!base::TaskAnnotator::RunTask(const char * trace_event_name, base::PendingTask * pending_task) Line 142 C++
libcef.dll!base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWorkImpl(base::sequence_manager::LazyNow * continuation_lazy_now, bool * ran_task) Line 325 C++
libcef.dll!base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork() Line 250 C++
libcef.dll!base::MessagePumpForUI::DoRunLoop() Line 219 C++
libcef.dll!base::MessagePumpWin::Run(base::MessagePump::Delegate * delegate) Line 77 C++

Re: Need help with random shutdow crashes

PostPosted: Fri Jun 26, 2020 12:14 pm
by magreenblatt
Do you want to receive OnBeforeClose for all browsers before quitting the message loop and calling CefShutdown?

Re: Need help with random shutdow crashes

PostPosted: Fri Jun 26, 2020 1:50 pm
by rdh
Well magreenblatt, I am new to cef so I'm not sure. I do get those calls now. I have a CefClient object for each window for which I want a browser. I get the OnAfterCreated for each one (I have 3). I dump out the handler pointer and the browser pointer that is passed in to debug out. When I am closing my app, DoClose is called. But the odd thing is the browser passed in is rarely the same pointer received in OnAfterCreated. I'm dumping those calls too. And I can get more DoClose calls than handlers. Am I supposed to verify the browser passed in is the one passed to OnAfterCreated and ignore it if it doesn't match?

My app isn't structured at all like the cefsimple sample. I do not call CefRunMessageLoop and shutdown when it returns like the simple sample. Everything I have tested is working except for the shutdown call and I suspect it is thread related because of how inconsistent the crash is (though it always crashes at the same spot - an int 3 cpu instruction down in the cef dll.

I log creation:

Create - Hander = 0x0000000042baea80 {ref_count_={ref_count_=0x00000008 } lock_={owning_thread_ref_={id_=0x00000000 } ...} ...} browser = {ptr_=0x0000000069170710 {...} }
Create - Hander = 0x0000000042bb3d60 {ref_count_={ref_count_=0x00000008 } lock_={owning_thread_ref_={id_=0x00000000 } ...} ...} browser = {ptr_=0x000000006910a330 {...} }
Create - Hander = 0x0000000042bb2360 {ref_count_={ref_count_=0x00000008 } lock_={owning_thread_ref_={id_=0x00000000 } ...} ...} browser = {ptr_=0x000000006910a730 {...} }

I long my window destruction where I call the host and close the browser ("ptr" matches the pointer passed to my handler that I then just pass to my window as expected and I call GetHost()->CloseBrowser in that routine:

WebViewClosing - {ptr_=0x0000000069170710 {...} }
WebViewClosing - {ptr_=0x000000006910a330 {...} }
WebViewClosing - {ptr_=0x000000006910a730 {...} }

and then I log the CoClose calls coming from the cef "main UI" thread. Note none of these calls have any of the browser pointers above and one of my 3 handlers isn't called while another is called multiple times.

DoClose - Hander = 0x0000000042baea80 {ref_count_={ref_count_=0x00000003 } lock_={owning_thread_ref_={id_=0x00000000 } ...} ...} browser = {ptr_=0x000000004a3cc110 {...} }
DoClose - Hander = 0x0000000042bb2360 {ref_count_={ref_count_=0x00000003 } lock_={owning_thread_ref_={id_=0x00000000 } ...} ...} browser = {ptr_=0x000000004a3cbe10 {...} }
DoClose - Hander = 0x0000000042bb3d60 {ref_count_={ref_count_=0x00000003 } lock_={owning_thread_ref_={id_=0x00000000 } ...} ...} browser = {ptr_=0x000000004a3cbc10 {...} }
DoClose - Hander = 0x0000000042bb3d60 {ref_count_={ref_count_=0x00000003 } lock_={owning_thread_ref_={id_=0x00000000 } ...} ...} browser = {ptr_=0x000000004a3cd490 {...} }

Since I have already called CloseBrowser and set my browser variable to null in each WebViewClosing call, the handler doesn't really do anything in this case as my window object detaches from the handler. Also, I started logging these calls because sometimes I saw one of my browser pointers get passed to DoClose and sometimes they didn't match and I thought I must have written down a pointer wrong.

Should I expect DoClose to be called on my handler when the input isn't the same pointer it was passed during creation?

Also, to more directly answer your short question, I assumed I needed those and do get them. Just been running again and see the come in. But, I haven't crashed the last few times I ran while writing this post. I'll log those calls too and see if they fail to arrive when the app crashes. I'm mostly worried that I just don't quite have the wires connected correctly due to the browser pointers that don't match. Not matching was a clue I would crash, I thought. But I see that that doesn't always hold.

Aha, just duplicated it and DoClose was called but no calls to OnBeforeBrowserClosed. What does that tell me?

Re: Need help with random shutdow crashes

PostPosted: Fri Jun 26, 2020 2:11 pm
by rdh
I know it is anecdotal but so far if the OnBeforeClose gets called for each window I have, no crash. Could it be that cef is waiting on a web response when I am closing down and that's why this is so "random"? I'm thinking about keeping track of each web view I have and blocking my cef shutdown call until I do get OnBeforeClose for each web view. I assume cef will give up on a response if a server fails and do so in a timely manner (30s ?).

Re: Need help with random shutdow crashes

PostPosted: Fri Jun 26, 2020 2:32 pm
by magreenblatt
magreenblatt wrote:Do you want to receive OnBeforeClose for all browsers before quitting the message loop and calling CefShutdown?

Sorry, typo here. You should wait for OnBeforeClose before shutting down.

Re: Need help with random shutdow crashes

PostPosted: Fri Jun 26, 2020 3:47 pm
by rdh
Thanks for the info. I don't let the user see any "your app has crashed" message from the OS by calling SetErrorMode(SEM_FAILCRITICALERRORS|SEM_NOGPFAULTERRORBOX) but I'd rather always have a clean close sequence.

Re: Need help with random shutdow crashes

PostPosted: Fri Jun 26, 2020 5:58 pm
by rdh
Well I added synchronization code. But, I create a CefClient I pass into CefBrowserHost::CreateBrowser so when OnBeforeClose is called, it didn'tt look like that is enough. But, after that call, the CefClient object I created has its destructor called from the cef thread. Since that CefClient code is in my DLL, I have to wait until it is called. Normally that is right after OnBeforeClose. But, at least one time I found myself waiting forever and when I did a debug break, I found there was still one outstanding client. I eventually forced the synchronization so the app would close down and of course, kaboom.

Re: Need help with random shutdow crashes

PostPosted: Fri Jun 26, 2020 8:50 pm
by magreenblatt
I don’t follow the complexity that you’re describing. OnBeforeClose is called after the browser is destroyed due to CloseBrowser or the window being destroyed. Release all object references in OnBeforeClose. Keep a count of the # of browsers that you’ve created, and when the count reaches 0 quit the message loop. Finally, don’t call CefShutdown from a call stack that originates in CEF. This way there should be no existing browsers (or references) when CefShutdown is called.

Re: Need help with random shutdow crashes

PostPosted: Sat Jun 27, 2020 9:30 am
by rdh
I have a ref to the browser and cef has a ref to my client object. That is how cef makes call backs as my client object derives from CefClient. The ref count on my client goes up by one when CefBrowserHost::CreateBrowser is called with my client as a parameter. So I do release my refs but I have to return and wait for cef to release the refs it holds too.

Re: Need help with random shutdow crashes

PostPosted: Sat Jun 27, 2020 10:37 am
by magreenblatt
A reference to the CefClient object or other objects implemented by your client are OK during CefShutdown. Just make sure you release any references to CefBrowser or other objects that libcef creates that you may be holding.