crash in cefclient on ubuntu 18.04 with proprietary codecs?

[DanKegel]

On ubuntu 18.04, this works:

CEFBRANCH=3440
python automate-git.py --force-clean --x64-build \
--branch="$CEFBRANCH" \
--build-target=cefsimple \
--download-dir="$(pwd)/tmp/downloadedsrc" \
--no-release-build \
#
mkdir btmp
cd btmp
cmake ../cef_binary_*64 -GNinja -DCMAKE_BUILD_TYPE=Debug
ninja -v
btmp/tests/cefclient/Debug/cefclient

But doing it with the following options set fails on startup of cefclient with an illegal instruction:

export CEF_USE_GN=1
export GN_DEFINES="ffmpeg_branding=Chrome is_official_build=true proprietary_codecs=true remove_webcore_debug_symbols=true symbol_level=1 use_allocator=none use_sysroot=true"
export GYP_DEFINES="buildtype=Official disable_nacl=1 use_allocator=none use_sysroot=true"
export CEF_ARCHIVE_FORMAT=tar.bz2

I'm bisecting those options now.

[magreenblatt]

If you run with gdb you should be able to get a symbolized call stack for the failure. Please post the call stack.

[DanKegel]

Here's the stack dump (from my earlier post; it hasn't changed):

Program received signal SIGILL, Illegal instruction.
AddRef () at ../../cef/libcef_dll/ctocpp/ctocpp_ref_counted.h:33
33 ../../cef/libcef_dll/ctocpp/ctocpp_ref_counted.h: Datei oder Verzeichnis nicht gefunden.
(gdb) bt
#0 0x00007ffff1bb812b in AddRef() () at ../../cef/libcef_dll/ctocpp/ctocpp_ref_counted.h:33
#1 0x00007ffff1bb844c in AddRef () at ../../base/memory/scoped_refptr.h:274
#2 0x00007ffff1bb844c in scoped_refptr () at ../../base/memory/scoped_refptr.h:176
#3 0x00007ffff1bb844c in Wrap () at ../../cef/libcef_dll/ctocpp/ctocpp_ref_counted.h:125
#4 0x00007ffff1bb844c in cef_initialize() () at ../../cef/libcef_dll/libcef_dll.cc:229

I've done three runs of a bisect so far.
The problem does seem to appear when I enable the proprietary codecs.
This works (i.e. lets me build a working cefclient):

CEFBRANCH=3440
export CEF_USE_GN=1
export GN_DEFINES="remove_webcore_debug_symbols=true symbol_level=1 use_allocator=none use_sysroot=true"
export GYP_DEFINES="disable_nacl=1 use_allocator=none use_sysroot=true"
export CEF_ARCHIVE_FORMAT=tar.bz2
python automate-git.py --force-clean --x64-build \
--branch="$CEFBRANCH" \
--build-target=cefsimple \
--download-dir="$(pwd)/tmp/downloadedsrc" \
--no-release-build \
#

but this fails (i.e. cefclient crashes):

CEFBRANCH=3440
export CEF_USE_GN=1
export GN_DEFINES="ffmpeg_branding=Chrome is_official_build=true proprietary_codecs=true remove_webcore_debug_symbols=true symbol_level=1 use_allocator=none use_sysroot=true"
export GYP_DEFINES="buildtype=Official disable_nacl=1 use_allocator=none use_sysroot=true"
export CEF_ARCHIVE_FORMAT=tar.bz2
python automate-git.py --force-clean --x64-build \
--branch="$CEFBRANCH" \
--build-target=cefsimple \
--download-dir="$(pwd)/tmp/downloadedsrc" \
--no-release-build \
#

This works fine on ubuntu 18.04 with branch 3282, so the problem happened somewhere between 3282 and 3440. (I'll try it with 3359...)

[DanKegel]

3359 seems ok, i.e. the following lets me build a cefclient that doesn't crash on startup on ubuntu 18.04:

CEFBRANCH=3359
export CEF_USE_GN=1
export GN_DEFINES="ffmpeg_branding=Chrome is_official_build=true proprietary_codecs=true remove_webcore_debug_symbols=true symbol_level=1 use_allocator=none use_sysroot=true"
export GYP_DEFINES="buildtype=Official disable_nacl=1 use_allocator=none use_sysroot=true"
export CEF_ARCHIVE_FORMAT=tar.bz2
python automate-git.py --force-clean --x64-build \
--branch="$CEFBRANCH" \
--build-target=cefsimple \
--download-dir="$(pwd)/tmp/downloadedsrc" \
--no-release-build \
#

[DanKegel]

3396 seems ok, too, in that cefclient doesn't crash on startup.

So the problem appears to have been introduced after 3396 and is in 3440.

Is there a recipe somewhere for how to use git bisect run with cef ?
( https://bitbucket.org/chromiumembedded/ ... ckStart.md looks pretty close, but it's not quite clear how to inject git bisect run into that procedure.)

[magreenblatt]

You're probably looking for a Chromium change. I don't know of any way to do it automatically with git bisect, but you can use the automate-git.py script to build the Chromium updates on master branch between 3396 and 3440: https://bitbucket.org/chromiumembedded/ ... nch/master. Add `--checkout=<git hash>` to your automate-git.py command-line and remove `--branch`.

[DanKegel]

OK, starting with the commit closest to 3396, with the command

#!/bin/sh
# Manual bisecting a la
# viewtopic.php?f=6&t=16089
HASH=f24f885eeb7fe358a41e5544e29d76f74210c787
set -ex
export CEF_USE_GN=1
export GN_DEFINES="ffmpeg_branding=Chrome is_official_build=true proprietary_codecs=true remove_webcore_debug_symbols=true symbol_level=1 use_allocator=none use_sysroot=true"
export GYP_DEFINES="buildtype=Official disable_nacl=1 use_allocator=none use_sysroot=true"
export CEF_ARCHIVE_FORMAT=tar.bz2
python automate-git.py --x64-build \
--checkout=$HASH \
--build-target=cefsimple \
--download-dir="$(pwd)/tmp/downloadedsrc" \
--no-release-build \
#

I suppose I should add --force-clean again.

I have an i7-8700 coming tomorrow, so it's a bit of a race to see if the bisect finds the problem on my "old" i7-6700 before that's up and running...

[DanKegel]

That ran ok. And the next one seems to be re-using the local git clients sensibly... fingers crossed.

[DanKegel]

I have to say, the i7-8700 really helps. It builds cef in 2 hours (vs. 3 hours with an i7-6700).

The first bad commit appears to be
https://github.com/chromiumembedded/cef ... 2408760086

The previous commit is ok, though cefclient failed to build with it at first, giving the error

tmp/downloadedsrc/chromium/src/cef/binary_distrib/cef_binary_3.3403.1779.g84e2286_linux64/tests/cefclient/browser/views_menu_bar.h:115:26: error
: ‘char16’ is not a member of ‘base’
typedef std::map<base::char16, int> MnemonicMap;

requiring the following patch (cherrypicked from the next commit) after building cef:

--- ./tmp/downloadedsrc/chromium/src/cef/binary_distrib/cef_binary_3.3409.1780.g240ba80_linux64/include/base/cef_string16.h 2018-06-22 12:19
:07.109957450 -0700
+++ ./tmp/downloadedsrc/chromium/src/cef/binary_distrib/cef_binary_3.3409.1780.g240ba80_linux64/include/base/cef_string16.h.new 2018-06-22 12:19
:03.621930593 -0700
@@ -164,6 +164,7 @@

namespace base {

+typedef cef::base::char16 char16;
typedef cef::base::string16 string16;

extern std::ostream& operator<<(std::ostream& out, const string16& str);

[magreenblatt]

The Chromium changes in that range are: https://chromium.googlesource.com/chrom ... er&n=10000

I'm suspecting this commit: https://crrev.com/a6889a090458a2653794c ... db743dc7f9 ("Enable cfi-icall for official Linux x64 builds"). According to https://www.chromium.org/developers/tes ... -integrity:

By default, a program compiled with CFI will crash with SIGILL if it detects a CFI violation.
For better error messages (but not for production use) add use_cfi_diag=true to your args.gn

However, use_cfi_diag=true cannot be used in combination with is_official_build=true so that doesn't really help to debug this issue.

I'm currently creating a local official build and will debug further with GDB.