Crash in CefSimple When Specifying Cache Folder on Linux

[Mayhew]

Before I file a bug I wanted to get some info to be sure it is a real issue. I built the 3865 (at 77.0.3865.90) branch via the automated build method on Linux and am seeing a crash on launch of our app. I can repro this by setting a cache path in CefSettings in CefSimple as well. If I leave the cache path empty both applications run normally. I have built the local builds on Debian and Ubuntu 14 and get the same result. I tried this with the builds from Spotify and the crash does not occur which is confusing. If I use the spotify builds libcef.so in my local build, both apps work. So the issue is in libcef.so itself. A friend helped me narrow this down and it looks like the crash is happening in GlibcGetSizeEstimate at the return call. This code was added here in May of this year. If we revert that change in our local builds everything works normally. The callstack is below.

I was thinking that maybe this is an issue with how my local builds are setting use_allocator or use_allocator_shim but I am using the stock build settings for my local builds. When we analyze the assembly for the GlibcGetSizeEstimate method between my build and the Spotify build, they are indeed different. I'm not sure what is going on.

To repro in CefSimple I just added these lines to cefsimple_linux.cc

CefString(&settings.cache_path) = CefString("<some valid path on disk>");
CefString(&settings.root_cache_path) = CefString("<some valid path on disk>");

Any insight would be greatly appreciated. If anyone else can repro, I'm happy to file a bug.

Thanks,
John

Call Stack

Code: Select all

#0  0x00007faa5cb66a94 in _ZN12_GLOBAL__N_120GlibcGetSizeEstimateEPKN4base9allocator17AllocatorDispatchEPvS5_$bd8608a96952500691d600ccbd08bddd.cfi () at ../../base/allocator/allocator_shim_default_dispatch_to_glibc.cc:67
#1  0x00007faa5e0cb308 in sqlite3MallocSize () at ../../third_party/sqlite/amalgamation/sqlite3.c:26852
#2  0x00007faa5e0cb308 in mallocWithAlarm () at ../../third_party/sqlite/amalgamation/sqlite3.c:26786
#3  0x00007faa5e0cb308 in sqlite3Malloc () at ../../third_party/sqlite/amalgamation/sqlite3.c:26808
#4  0x00007faa5e188cf2 in sqlite3MallocZero () at ../../third_party/sqlite/amalgamation/sqlite3.c:27013
#5  0x00007faa5e188cf2 in pthreadMutexAlloc () at ../../third_party/sqlite/amalgamation/sqlite3.c:25650
#6  0x00007faa5e0dbe04 in sqlite3MutexAlloc () at ../../third_party/sqlite/amalgamation/sqlite3.c:25193
#7  0x00007faa5e0dbe04 in chrome_sqlite3_initialize () at ../../third_party/sqlite/amalgamation/sqlite3.c:23822
#8  0x00007faa5e0ba5c9 in EnsureSqliteInitialized() () at ../../sql/initialization.cc:55
#9  0x00007faa5e0b3b69 in OpenInternal() () at ../../sql/database.cc:1371
#10 0x00007faa5e0b3a4a in Open() () at ../../sql/database.cc:270
#11 0x00007faa5e845b01 in InitializeDatabase() () at ../../net/extras/sqlite/sqlite_persistent_store_backend_base.cc:99
#12 0x00007faa5e83d767 in LoadAndNotifyInBackground() () at ../../net/extras/sqlite/sqlite_persistent_cookie_store.cc:637
#13 0x00007faa5e844077 in Invoke<void (net::SQLitePersistentCookieStore::Backend::*)(base::OnceCallback<void (std::__1::vector<std::__1::unique_ptr<net::CanonicalCookie, std::__1::default_delete<net::CanonicalCookie> >, std::__1::allocator<std::__1::unique_ptr<net::CanonicalCookie, std::__1::default_delete<net::CanonicalCookie> > > >)>, const base::Time &), scoped_refptr<net::SQLitePersistentCookieStore::Backend>, base::OnceCallback<void (std::__1::vector<std::__1::unique_ptr<net::CanonicalCookie, std::__1::default_delete<net::CanonicalCookie> >, std::__1::allocator<std::__1::unique_ptr<net::CanonicalCookie, std::__1::default_delete<net::CanonicalCookie> > > >)>, base::Time> ()
    at ../../base/bind_internal.h:499
#14 0x00007faa5e844077 in MakeItSo<void (net::SQLitePersistentCookieStore::Backend::*)(base::OnceCallback<void (std::__1::vector<std::__1::unique_ptr<net::CanonicalCookie, std::__1::default_delete<net::CanonicalCookie> >, std::__1::allocator<std::__1::unique_ptr<net::CanonicalCookie, std::__1::default_delete<net::CanonicalCookie> > > >)>, const base::Time &), scoped_refptr<net::SQLitePersistentCookieStore::Backend>, base::OnceCallback<void (std::__1::vector<std::__1::unique_ptr<net::CanonicalCookie, std::__1::default_delete<net::CanonicalCookie> >, std::__1::allocator<std::__1::unique_ptr<net::CanonicalCookie, std::__1::default_delete<net::CanonicalCookie> > > >)>, base::Time> ()
    at ../../base/bind_internal.h:599
#15 0x00007faa5e844077 in RunImpl<void (net::SQLitePersistentCookieStore::Backend::*)(base::OnceCallback<void (std::__1::vector<std::__1::unique_ptr<net::CanonicalCookie, std::__1::default_delete<net::CanonicalCookie> >, std::__1::allocator<std::__1::unique_ptr<net::CanonicalCookie, std::__1::default_delete<net::CanonicalCookie> > > >)>, const base::Time &), std::__1::tuple<scoped_refptr<net::SQLitePersistentCookieStore::Backend>, base::OnceCallback<void (std::__1::vector<std::__1::unique_ptr<net::CanonicalCookie, std::__1::default_delete<net::CanonicalCookie> >, std::__1::allocator<std::__1::unique_ptr<net::CanonicalCookie, std::__1::default_delete<net::CanonicalCookie> > > >)>, base::Time>, 0, 1, 2> () at ../../base/bind_internal.h:672
#16 0x00007faa5e844077 in RunOnce() () at ../../base/bind_internal.h:641
#17 0x00007faa5cacd8a8 in Run () at ../../base/callback.h:98
#18 0x00007faa5cacd8a8 in RunTask() () at ../../base/task/common/task_annotator.cc:142
#19 0x00007faa5cb00182 in base::internal::TaskTracker::RunBlockShutdown(base::internal::Task*) ()
    at ../../base/task/thread_pool/task_tracker.cc:762
#20 0x00007faa5caff966 in RunTaskWithShutdownBehavior () at ../../base/task/thread_pool/task_tracker.cc:777
#21 0x00007faa5caff966 in RunOrSkipTask() () at ../../base/task/thread_pool/task_tracker.cc:615
#22 0x00007faa5cb65143 in RunOrSkipTask() () at ../../base/task/thread_pool/task_tracker_posix.cc:24
#23 0x00007faa5cafeee9 in RunAndPopNextTask() () at ../../base/task/thread_pool/task_tracker.cc:473
#24 0x00007faa5cb0bc4a in RunWorker() () at ../../base/task/thread_pool/worker_thread.cc:322
#25 0x00007faa5cb0b9d4 in base::internal::WorkerThread::RunPooledWorker() () at ../../base/task/thread_pool/worker_thread.cc:223
#26 0x00007faa5cb65ade in ThreadFunc() () at ../../base/threading/platform_thread_posix.cc:81
#27 0x00007faa56b22c73 in start_thread (arg=<optimized out>) at pthread_create.c:486
#28 0x00007faa56854def in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

[Mayhew]

It looks like my local builds set use_allocator=none. I wonder if the Spotify build system changes that to use_allocator=tcmalloc for Linux.

[amaitland]

The builder config is detailed at https://bitbucket.org/chromiumembedded/ ... figuration

[Mayhew]

Okay, good to know. That indicates the setting used is use_allocator=none which is what I'm using for my local builds as well. Is this difference possibly due to a glibc version difference?

[magreenblatt]

If you’re not building with use_sysroot=true then there is likely to be many library version differences.

[Mayhew]

If I am simply following the automated build steps I assume use_sysroot=true would be used. I didn't make any changes to the build process for my local builds.

[magreenblatt]

Well, if you’re following the automated build steps then your local build should be identical to the Spotify build. If that’s not the case then you likely have some deviation in your local configuration.

[chrisdz]

This was caused by Clang's Clang's Control Flow Integrity (CFI) checks that prevented the compiler from generating code calling a function pointer that it could not reason about because it was loaded dynamically. Instead, it generated a ud2 "Undefined Instruction" opcode to kill the process instead:

Code: Select all

  00000000055eaa50 <_ZN12_GLOBAL__N_120GlibcGetSizeEstimateEPKN4base9allocator17AllocatorDispatchEPvS5_$bd8608a96952500691d600ccbd08bddd.cfi>:
   55eaa50:   55                      push   %rbp
   55eaa51:   48 89 e5                mov    %rsp,%rbp
   55eaa54:   8a 05 9e e9 dd 05       mov    0x5dde99e(%rip),%al        # b3c93f8 <_ZGVZN12_GLOBAL__N_120GlibcGetSizeEstimateEPKN4base9allocator17AllocatorDispatchEPvS5_E6fn_ptr>
   55eaa5a:   84 c0                   test   %al,%al
   55eaa5c:   75 36                   jne    55eaa94 <_ZN12_GLOBAL__N_120GlibcGetSizeEstimateEPKN4base9allocator17AllocatorDispatchEPvS5_$bd8608a96952500691d600ccbd08bddd.cfi+0x44>
   55eaa5e:   48 8d 3d 93 e9 dd 05    lea    0x5dde993(%rip),%rdi        # b3c93f8 <_ZGVZN12_GLOBAL__N_120GlibcGetSizeEstimateEPKN4base9allocator17AllocatorDispatchEPvS5_E6fn_ptr>
   55eaa65:   e8 56 39 d6 ff          callq  534e3c0 <__cxa_guard_acquire>
   55eaa6a:   85 c0                   test   %eax,%eax
   55eaa6c:   74 26                   je     55eaa94 <_ZN12_GLOBAL__N_120GlibcGetSizeEstimateEPKN4base9allocator17AllocatorDispatchEPvS5_$bd8608a96952500691d600ccbd08bddd.cfi+0x44>
   55eaa6e:   48 8d 35 79 e9 95 fb    lea    -0x46a1687(%rip),%rsi        # f493ee <_ZN5blinkL36kDefaultNativeMemorySamplingIntervalE+0x1d8c8e>
   55eaa75:   48 c7 c7 ff ff ff ff    mov    $0xffffffffffffffff,%rdi
   55eaa7c:   e8 8f 9d 76 05          callq  ad54810 <dlsym@plt>
   55eaa81:   48 89 05 68 e9 dd 05    mov    %rax,0x5dde968(%rip)        # b3c93f0 <_ZZN12_GLOBAL__N_120GlibcGetSizeEstimateEPKN4base9allocator17AllocatorDispatchEPvS5_E6fn_ptr>
   55eaa88:   48 8d 3d 69 e9 dd 05    lea    0x5dde969(%rip),%rdi        # b3c93f8 <_ZGVZN12_GLOBAL__N_120GlibcGetSizeEstimateEPKN4base9allocator17AllocatorDispatchEPvS5_E6fn_ptr>
   55eaa8f:   e8 dc 39 d6 ff          callq  534e470 <__cxa_guard_release>
   55eaa94:   0f 0b                   ud2
   55eaa96:   cc                      int3
   55eaa97:   cc                      int3
   55eaa98:   cc                      int3
   55eaa99:   cc                      int3
   55eaa9a:   cc                      int3
   55eaa9b:   cc                      int3
   55eaa9c:   cc                      int3
   55eaa9d:   cc                      int3
   55eaa9e:   cc                      int3
   55eaa9f:   cc                      int3


It's not clear why other users aren't hitting this. It may have something to do with the toolchain that we're using, but it seems to be hit when using a compiler that supports -fsanitize=cfi-icall and when Chromium is built with is_cfi=true (the default for Chromium and CEF Linux x64) and use_allocator=none (not the default for Chromium Linux x64, but the default for CEF Linux x64).

See https://crrev.com/c/1868513 for more details and the fix that is being upstreamed to Chromium.

[magreenblatt]

Thanks for the update. The Spotify builders are additionally using "is_cfi=false use_thin_lto=false" for the 64-bit Linux builds only, which likely explains why they aren't experiencing the CFI issue. Looks like these flags were added for issue #2484.

We can try removing these flags once the Chromium change you've linked above has merged.

[Mayhew]

Thanks to Chris for all the hard work debugging this,figuring out the cause and the correct fix!