Startup crash on macOS base::allocator::StoreMallocZone

[weq]

When i strip "debug information" from my binary by removing a compile flag, my CEF app no longer starts up and throws this error. Here is the backtrace / thread list from lldb. Can i please get some guidance on whats going wrong here or any info i could relay to the Mono devs who build the framework im using to integrate with CEF.

MacOS chromium 86.0.4240.18

Process 91159 stopped
* thread #1, name = 'tid_307', queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x0)
frame #0: 0x000000010ea203da libcef`base::allocator::StoreMallocZone(_ChromeMallocZone*) [inlined] base::allocator::StoreZoneFunctions(zone=0x0000000100db0000, functions=<unavailable>) at malloc_zone_functions_mac.cc:27:3 [opt]
Target 0: (JanisonReplay) stopped.
(lldb) bt all
* thread #1, name = 'tid_307', queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x0)
* frame #0: 0x000000010ea203da libcef`base::allocator::StoreMallocZone(_ChromeMallocZone*) [inlined] base::allocator::StoreZoneFunctions(zone=0x0000000100db0000, functions=<unavailable>) at malloc_zone_functions_mac.cc:27:3 [opt]
frame #1: 0x000000010ea203d9 libcef`base::allocator::StoreMallocZone(zone=0x0000000100db0000) at malloc_zone_functions_mac.cc:82 [opt]
frame #2: 0x000000010ea1f43d libcef`base::allocator::StoreFunctionsForAllZones() at allocator_interception_mac.mm:330:5 [opt]
frame #3: 0x000000010ea30d7d libcef`base::allocator::InitializeAllocatorShim() at allocator_shim.cc:351:3 [opt]
frame #4: 0x000000011115d675 libcef`service_manager::MainInitialize(params=0x0000000102275080) at main.cc:288:5 [opt]
frame #5: 0x000000010e6faf31 libcef`CefMainRunner::ContentMainInitialize(this=0x0000000102275520, args=<unavailable>, windows_sandbox_info=<unavailable>, no_sandbox=<unavailable>) at main_runner.cc:391:10 [opt]
frame #6: 0x000000010e6fad37 libcef`CefMainRunner::Initialize(this=0x0000000102275520, settings=0x0000000102277308, application=<unavailable>, args=0x00007ffeefbfedd0, windows_sandbox_info=<unavailable>, initialized=0x0000000102277300, context_initialized=base::OnceClosure @ 0x00007ffeefbfea80)>) at main_runner.cc:235:7 [opt]
frame #7: 0x000000010e6d55c4 libcef`CefContext::Initialize(this=0x0000000102277300, args=0x00007ffeefbfedd0, settings=<unavailable>, application=<unavailable>, windows_sandbox_info=<unavailable>) at context.cc:356:24 [opt]
frame #8: 0x000000010e6d507b libcef`CefInitialize(args=0x00007ffeefbfedd0, settings=0x00007ffeefbfec08, application=<unavailable>, windows_sandbox_info=0x0000000000000000) at context.cc:214:21 [opt]
frame #9: 0x000000010ba1905b libcef`::cef_initialize(args=<unavailable>, settings=<unavailable>, application=0x0000000100e078a0, windows_sandbox_info=0x0000000000000000) at libcef_dll.cc:112:7 [opt]
frame #10: 0x00000001151b67e4
frame #11: 0x000000010ba0fb5b
frame #12: 0x000000010ba060a4
frame #13: 0x000000010a8d2ab3
frame #14: 0x000000010a8d1b03
frame #15: 0x0000000105bacf93
frame #16: 0x0000000100127c6e JanisonReplay`mono_jit_runtime_invoke(method=0x0000000100e078a0, obj=<unavailable>, params=0x000000010226f260, exc=0x0000000000000000, error=<unavailable>) at mini-runtime.c:3191:12 [opt]
frame #17: 0x000000010025cd78 JanisonReplay`mono_runtime_invoke_checked [inlined] do_runtime_invoke(method=0x0000000102212738, obj=0x0000000000000000, params=0x00007ffeefbff608, exc=<unavailable>, error=0x00007ffeefbff640) at object.c:3052:11 [opt]
frame #18: 0x000000010025cd42 JanisonReplay`mono_runtime_invoke_checked(method=0x0000000102212738, obj=0x0000000000000000, params=0x00007ffeefbff608, error=0x00007ffeefbff640) at object.c:3220 [opt]
frame #19: 0x00000001002640a5 JanisonReplay`mono_runtime_exec_main_checked [inlined] do_exec_main_checked(method=0x0000000102212738, args=<unavailable>, error=0x00007ffeefbff640) at object.c:0 [opt]
frame #20: 0x0000000100264069 JanisonReplay`mono_runtime_exec_main_checked(method=0x0000000102212738, args=<unavailable>, error=0x00007ffeefbff640) at object.c:5284 [opt]
frame #21: 0x000000010008403c JanisonReplay`mono_jit_exec at driver.c:1383:13 [opt]
frame #22: 0x000000010008402e JanisonReplay`mono_jit_exec(domain=<unavailable>, assembly=<unavailable>, argc=1, argv=0x000000010213cf98) at driver.c:1328 [opt]
frame #23: 0x0000000100087136 JanisonReplay`mono_main [inlined] main_thread_handler(user_data=<unavailable>) at driver.c:1465:3 [opt]
frame #24: 0x00000001000870ef JanisonReplay`mono_main(argc=4, argv=<unavailable>) at driver.c:2714 [opt]
frame #25: 0x000000010003c05c JanisonReplay`xamarin_main(argc=1, argv=0x00007ffeefbff950, launch_mode=XamarinLaunchModeApp) at launcher.m:656:9
frame #26: 0x000000010003ced4 JanisonReplay`main(argc=1, argv=0x00007ffeefbff950) at launcher.m:675:9
frame #27: 0x00007fff69225cc9 libdyld.dylib`start + 1
thread #2
frame #0: 0x00007fff693684ce libsystem_kernel.dylib`__workq_kernreturn + 10
frame #1: 0x00007fff69426aa1 libsystem_pthread.dylib`_pthread_wqthread + 390
frame #2: 0x00007fff69425b77 libsystem_pthread.dylib`start_wqthread + 15
thread #3
frame #0: 0x00007fff693684ce libsystem_kernel.dylib`__workq_kernreturn + 10
frame #1: 0x00007fff69426aa1 libsystem_pthread.dylib`_pthread_wqthread + 390
frame #2: 0x00007fff69425b77 libsystem_pthread.dylib`start_wqthread + 15
thread #4, name = 'JavaScriptCore bmalloc scavenger'
frame #0: 0x00007fff69369882 libsystem_kernel.dylib`__psynch_cvwait + 10
frame #1: 0x00007fff6942a425 libsystem_pthread.dylib`_pthread_cond_wait + 698
frame #2: 0x00007fff664f8592 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
frame #3: 0x00007fff33987274 JavaScriptCore`void std::__1::condition_variable_any::wait<std::__1::unique_lock<bmalloc::Mutex> >(std::__1::unique_lock<bmalloc::Mutex>&) + 84
frame #4: 0x00007fff3398ba7b JavaScriptCore`bmalloc::Scavenger::threadRunLoop() + 299
frame #5: 0x00007fff3398b649 JavaScriptCore`bmalloc::Scavenger::threadEntryPoint(bmalloc::Scavenger*) + 9
frame #6: 0x00007fff3398dd27 JavaScriptCore`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)(bmalloc::Scavenger*), bmalloc::Scavenger*> >(void*) + 39
frame #7: 0x00007fff6942a109 libsystem_pthread.dylib`_pthread_start + 148
frame #8: 0x00007fff69425b8b libsystem_pthread.dylib`thread_start + 15
thread #5, name = 'SGen worker'
frame #0: 0x00007fff69369882 libsystem_kernel.dylib`__psynch_cvwait + 10
frame #1: 0x00007fff6942a425 libsystem_pthread.dylib`_pthread_cond_wait + 698
frame #2: 0x0000000100315641 JanisonReplay`thread_func [inlined] mono_os_cond_wait(cond=<unavailable>, mutex=<unavailable>) at mono-os-mutex.h:219:8 [opt]
frame #3: 0x000000010031562e JanisonReplay`thread_func at sgen-thread-pool.c:165 [opt]
frame #4: 0x0000000100315563 JanisonReplay`thread_func(data=0x0000000000000000) at sgen-thread-pool.c:196 [opt]
frame #5: 0x00007fff6942a109 libsystem_pthread.dylib`_pthread_start + 148
frame #6: 0x00007fff69425b8b libsystem_pthread.dylib`thread_start + 15
thread #6, name = 'Finalizer'
frame #0: 0x00007fff69366e36 libsystem_kernel.dylib`semaphore_wait_trap + 10
frame #1: 0x00000001001e379f JanisonReplay`finalizer_thread [inlined] mono_os_sem_wait(sem=<unavailable>, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-os-semaphore.h:84:8 [opt]
frame #2: 0x00000001001e3794 JanisonReplay`finalizer_thread at mono-coop-semaphore.h:41 [opt]
frame #3: 0x00000001001e377a JanisonReplay`finalizer_thread(unused=<unavailable>) at gc.c:965 [opt]
frame #4: 0x00000001002b5e33 JanisonReplay`start_wrapper_internal(start_info=0x0000000000000000, stack_ptr=<unavailable>) at threads.c:1233:3 [opt]
frame #5: 0x00000001002b5c7e JanisonReplay`start_wrapper(data=0x000000010220e4d0) at threads.c:1308:8 [opt]
frame #6: 0x00007fff6942a109 libsystem_pthread.dylib`_pthread_start + 148
frame #7: 0x00007fff69425b8b libsystem_pthread.dylib`thread_start + 15
thread #8
frame #0: 0x00007fff69425b68 libsystem_pthread.dylib`start_wqthread
thread #9
frame #0: 0x00007fff693684ce libsystem_kernel.dylib`__workq_kernreturn + 10
frame #1: 0x00007fff69426aa1 libsystem_pthread.dylib`_pthread_wqthread + 390
frame #2: 0x00007fff69425b77 libsystem_pthread.dylib`start_wqthread + 15


Process 91159 stopped
* thread #1: tid = 0x8d7d06, 0x000000010ea203da libcef`base::allocator::StoreMallocZone(_ChromeMallocZone*) [inlined] base::allocator::StoreZoneFunctions(zone=0x0000000100db0000, functions=<unavailable>) at malloc_zone_functions_mac.cc:27:3, name = 'tid_307', queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x0)
thread #2: tid = 0x8d7d23, 0x00007fff69425b68 libsystem_pthread.dylib`start_wqthread
thread #3: tid = 0x8d7d24, 0x00007fff69425b68 libsystem_pthread.dylib`start_wqthread
thread #4: tid = 0x8d7d25, 0x00007fff69369882 libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'JavaScriptCore bmalloc scavenger'
thread #5: tid = 0x8d7d27, 0x00007fff69369882 libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'SGen worker'
thread #6: tid = 0x8d7d29, 0x00007fff69366e36 libsystem_kernel.dylib`semaphore_wait_trap + 10, name = 'Finalizer'
thread #8: tid = 0x8d7d44, 0x00007fff69425b68 libsystem_pthread.dylib`start_wqthread
thread #9: tid = 0x8d8092, 0x00007fff693684ce libsystem_kernel.dylib`__workq_kernreturn + 10

[Czarek]

Does it reproduce with CEF sample applications?

[weq]

-

[weq]

Im using a Xamarin to access CEF via an API approach that mirrors CEF sample but takes the form: C# / Xamarin.macos -> Obj-C -> CEF

Ive been kind of stumped on whats happening and why when i remove the debug symbols i get this crash. ive been reading up on the mechanics of the function and its supposed to be doing some tricks to redirect memory. i have no idea how this intereacts with the debug symbols and ive reached out to the mono maintainers and they dont know enough about CEF etc to help me out.

I was hoping moving from v84 -> v86 would help me out, and it solved the majority of my other problems but not this specific one.

this is a basic impl my private apps architecture
https://github.com/captainjono/Chromely

this sample app doesnt suffer the problem... when this projects debug symbols are turned off, it STILL WORKS! but when my private projects debug symbols are turned off, it crashes with the above.

I can really only think of is things like "reflection" to fail because debug symbols are missing. but i have no idea why CEF would be needing to reflect on my assemblies how call cef init causes reflection to happen in my app at a point where this function is called.
this thread here https://www.magpcss.org/ceforum/viewtop ... 17&t=16471 indicates another person who experienced the crash - and they patched the cef source code. it doesnt seem logical to me but maybe i have a good repo now and this could help cef authors?? @magreenblatt

here are are some other similiar crashes:
https://youtrack.jetbrains.com/issue/GO-9720
https://pastebin.com/8XKXjKdK

Im tooo rusty with C to really step through the cef source and make the changes myself.. but i may attempt to repo that guys changes if i must. (i need to remove the symbols so i can pass pentest).. i can readily repo in lldb though if there are any commands u can think of that would help me pinpoint the issue here?

@Czarek You name actually is familiar with a contractor who works with me. if u thought u could help out if u had access to the whole source reach out through rahul@rjvtech my company would compensate$

[magreenblatt]

You can try building CEF/Chromium with some combination of use_allocator=none and use_allocator_shim=false in GN_DEFINES. See also https://bitbucket.org/chromiumembedded/cef/issues/3095 which may or may not be an issue on Mac.