Chromium crash on JCEF shutdown

[Rigner]

Hey,

We've been working for a while on adding JCEF to our app. We're 1 bug away from releasing it.

When the app is being shut down, if a browser instance was created previously, a crash will happen within Chromium, on macOS only.
Would you have any idea about what could be causing it ? I checked a bit the chromium code on the version we're using (official Spotify release build) and that scenario isn't supposed to happen, there's even a `DCHECK()` there...

Thanks!

Our shutdown code (Java):

Code: Select all

   public void destroy() {
      this.cefClient.dispose();
      this.cefApp.dispose();

      long start = System.currentTimeMillis();
      int i = 0;
      while (CefManager.openedBrowsers != 0) {
         if (System.currentTimeMillis() - start > 1_000) {
            CefManager.LOGGER.info("Failed to clean up before timeout. Pumped events {} times", i);
            return;
         }
         this.doEvents();
         i++;
      }

      CefManager.LOGGER.info("Cleanup successful");
   }

Stack trace with symbols:

Code: Select all

content::SiteInstanceImpl::GetProcess() (in Chromium Embedded Framework) (site_instance_impl.cc:436)
content::SiteInstanceImpl::GetProcess() (in Chromium Embedded Framework) (site_instance_impl.cc:436)
content::ServiceWorkerProcessManager::Shutdown() (in Chromium Embedded Framework) (service_worker_process_manager.cc:60)
content::ServiceWorkerContextWrapper::Shutdown() (in Chromium Embedded Framework) (service_worker_context_wrapper.cc:335)
content::StoragePartitionImpl::OnBrowserContextWillBeDestroyed() (in Chromium Embedded Framework) (storage_partition_impl.cc:1243)
content::StoragePartitionImplMap::ForEach(base::FunctionRef<void (content::StoragePartition*)>) (in Chromium Embedded Framework) (storage_partition_impl_map.cc:440)
content::BrowserContextImpl::NotifyWillBeDestroyed() (in Chromium Embedded Framework) (browser_context_impl.cc:0)
Profile::MaybeSendDestroyedNotification() (in Chromium Embedded Framework) (profile.cc:499)
AlloyBrowserContext::Shutdown() (in Chromium Embedded Framework) (alloy_browser_context.cc:195)
CefBrowserContext::RemoveCefRequestContext(CefRequestContextImpl*) (in Chromium Embedded Framework) (browser_context.cc:249)
CefRequestContextImpl::~CefRequestContextImpl() (in Chromium Embedded Framework) (request_context_impl.cc:171)
CefRequestContextImpl::~CefRequestContextImpl() (in Chromium Embedded Framework) (request_context_impl.cc:163)
CefRequestContextImpl::~CefRequestContextImpl() (in Chromium Embedded Framework) (request_context_impl.cc:171)
CefRequestContextImpl::~CefRequestContextImpl() (in Chromium Embedded Framework) (request_context_impl.cc:163)
CefRequestContextImpl::Release() const (in Chromium Embedded Framework) (request_context_impl.h:228)
content::BrowserMainLoop::ShutdownThreadsAndCleanUp() (in Chromium Embedded Framework) (browser_main_loop.cc:0)
content::BrowserMainRunnerImpl::Shutdown() (in Chromium Embedded Framework) (browser_main_runner_impl.cc:196)
CefMainRunner::Shutdown(base::OnceCallback<void ()>, base::OnceCallback<void ()>) (in Chromium Embedded Framework) (main_runner.cc:333)
CefShutdown() (in Chromium Embedded Framework) (context.cc:365)

[Rigner]

I forgot to send it, here's the settings we're passing when starting the app:

Code: Select all

builder.getCefSettings().windowless_rendering_enabled = true;
builder.addJcefArgs("--disable-in-process-stack-traces");
builder.addJcefArgs("--no-sandbox");
builder.addJcefArgs("--single-process");

We had to enable single process because we had issues where some antiviruses (Malwarebytes ?) were blocking jcef Helper process from starting up, causing Chromium to abort().

That's why it's also weird that it's crashing when shutting down processes when it's not supposed to have any.

[Rigner]

Oh great I see, it's actually because single process mode has never been fully implemented...

This code here is not supposed to run in single process, but the flag is never set except on mobile devices...
https://github.com/chromium/chromium/bl ... ger.cc#L57
https://github.com/chromium/chromium/bl ... op.cc#L830

What would you recommend me to try then ? Removing this code and making my own Chromium build ? Using different flags ?

Like I said, jcef Helper.exe is flagged as a virus by many antiviruses (which is stupid), so we have a decent amount of users that would get the issue (we have like 100k+ unique users a day at least, more on weekends). And since Chromium is aborting in that scenario, we can't really catch it and just disable the feature properly...

cc @magreenblatt

[magreenblatt]

jcef Helper.exe is flagged as a virus by many antiviruses

Are you code signing the binaries?

[Rigner]

That is a very good question. We're using the jcefbuild / jcefmaven repos to generate the dependency, and I don't think It's signing them on windows... I'm gonna add that and test if it works better, thanks!

[Rigner]

Hey, looping back on this, it seems that after codesigning the dlls / exe, the affected people stop getting the abort(). Although, there is a new one for a very small portion of users (only 2 so far).

Code: Select all

[0220/201420.791:ERROR:icu_util.cc(237)] Failed to initialize ICU with data file: U_INVALID_FORMAT_ERROR

We asked them to send us the icudtl.dat file to see if it has been modified / corrupted and we're waiting on them to respond.

Have you ever seen that one happen and know what could be at fault here ? Like an antivirus once again ? it's weird because it's not a executable file so they shouldn't be interfering...

I also noticed one has a chinese locale (zh-TW) and the other a norwegian one (nb-NO), just saying in case it could affect anything since they might be using non-ASCII / standard characters.

EDIT: I just received the ICU dat file from one of the users, it's just full of null bytes (funnily it has the right size). I guess that most likely means an antivirus edited it ?
EDIT 2: Seems like the jdk binaries we were using (17.0.8) were not signed. Asking the user to switch to jdk21 (signed by oracle) fixed the issue. I will try to update our software to latest jdk 17 which is signed and add a safety in our code to force reinstall the files if it detects it's corrupted.