Browser subprocess not shutting down on Win7

[magreenblatt]

Do you mean separate sub-process instances? That should be fine.

[ScottKevill]

Is it possible that your app is not calling CefShutdown in all cases (perhaps exiting early with exit() or similar)? Does the problem reproduce with 'cefclient --external-message-pump'? Are you implementing CefDoMessageLoopWork as shown in cefclient?

Is that still the case? I assumed it had just been because I was using an older version of CEF3.

I had to create a watchdog thread in each child process that would WaitForSingleObject() on the main process handle and forcibly self-terminate, just to make sure the child processes would exit 100% of the time. ie. Even if the main process crashed before (or during) CefShutdown.

Relying on CefShutdown to guarantee cleanup of child processes seems like an unrealistic requirement.

[magreenblatt]

This is not a new issue with Chromium, or even Windows-specific. If your workaround gives you consistent results on all versions of Windows then I suggest sticking with it.

[Mikael]

In the process of testing out PR 158 we upgraded from 3.3112.1650 to 3.3325.1752 and we've had this issue appear very frequently since then.

Basically, when stopping a debugging session (thereby killing the browser process being debugged) in Visual Studio, or for some other reason not cleanly exiting with a CefShutdown (like an exit(0)), you often get a renderer process that's just hanging around and consumes one virtual CPU core as it's constantly retrying the task RenderWidgetCompositor::RequestNewLayerTreeFrameSink.

I was under the assumption that this stemmed from changes related to the PR, but it turned out to be something related to the upgrade. Using the archive of Spotify CEF builds I've narrowed it down to being introduced at CEF's switch to Chromium major 64. I'm able to reproduce this issue almost 100% with any build greater or equal to CEF 3.3282.1728.g2171fc7 / Chromium 64.0.3282.119 and not at all with any build preceding it (starting with CEF 3.3239.1723.g071d1c1 / Chromium 63.0.3239.132).

I'll probably end up employing the same kind of fix/hack as described by ScottKevill, but I share the sentiment that having to rely on CefShutdown for cleanup of the child processes is somewhat unrealistic, if this is by design that is.

Anyway, I mostly just want to drive home the point that this issue has (in my experience) a point in time where it's become significantly more prevalent.

You can view a call stack of the zombie processes on this ticket here. Let me know if there's any more information you need.

[ndesktop]

I have noticed this way back.
Killing the browser does not allow normal renderer termination via CefShutdown, parent process being abruptly killed etc.

What I did is this:
- browser process creates a shutdown event, unsignaled
- spawn a cleanup process from the browser process, which is waiting on the shutdown event
- browser process set the shutdown event on termination.

If WaitForSingleObject(hShutdownEvent, INFINITE) gives you WAIT_TIMEOUT, WAIT_FAILED (basically anything else than WAIT_OBJECT_0), and having the PID of browser process, you know from the cleanup process that the browser did not ended normally.

The cleanup process does this in such a case:
- check if browser process term result (GetExitCodeProcess) - I need this to be absolutely sure the browser died from unnatural causes
- CreateToolhelp32Snapshot to get all processes with
(a) parent PID = killed browser PID
(b) same executable path as browser PID (or at least exe name)
- NtQueryInformationProcess might get involved if you need the full command line of active PIDs to parse for --type=<subprocess-type>, that is, if you need fine tuning
- for each such subprocess OpenProcess(... PROCESS_TERMINATE ... ) and TerminateProcess on the orphaned renderers.

I've noticed for utility, GPU that they (usually) closes themselves when the browser dies (for some reason), they might maintain a some form of ping/wait on browser process, or simply terminate if they have no work, I'm not sure.
Hope this helps.

[callum]

Thanks ndesktop - I'm working on a companion issue right now but I've bookmarked your post for later this week.

Many thanks.

[magreenblatt]

In the process of testing out PR 158 we upgraded from 3.3112.1650 to 3.3325.1752 and we've had this issue appear very frequently since then.

Basically, when stopping a debugging session (thereby killing the browser process being debugged) in Visual Studio, or for some other reason not cleanly exiting with a CefShutdown (like an exit(0)), you often get a renderer process that's just hanging around and consumes one virtual CPU core as it's constantly retrying the task RenderWidgetCompositor::RequestNewLayerTreeFrameSink.

I was under the assumption that this stemmed from changes related to the PR, but it turned out to be something related to the upgrade. Using the archive of Spotify CEF builds I've narrowed it down to being introduced at CEF's switch to Chromium major 64. I'm able to reproduce this issue almost 100% with any build greater or equal to CEF 3.3282.1728.g2171fc7 / Chromium 64.0.3282.119 and not at all with any build preceding it (starting with CEF 3.3239.1723.g071d1c1 / Chromium 63.0.3239.132).

I'll probably end up employing the same kind of fix/hack as described by ScottKevill, but I share the sentiment that having to rely on CefShutdown for cleanup of the child processes is somewhat unrealistic, if this is by design that is.

Anyway, I mostly just want to drive home the point that this issue has (in my experience) a point in time where it's become significantly more prevalent.

You can view a call stack of the zombie processes on this ticket here. Let me know if there's any more information you need.

Can you file a CEF issue for this? Thanks.

[Mikael]

Can you file a CEF issue for this? Thanks.

Will do.

In the meantime this code seems to work well for us (in our CEF sub process), albeit with very limited testing so far:

Code: Select all

#include <thread>
#include <tlhelp32.h>

HANDLE GetParentProcess()
{
    HANDLE Snapshot = CreateToolhelp32Snapshot(TH32CS_SNAPPROCESS, 0);

    PROCESSENTRY32 ProcessEntry = {};
    ProcessEntry.dwSize = sizeof(PROCESSENTRY32);

    if (Process32First(Snapshot, &ProcessEntry))
    {
        DWORD CurrentProcessId = GetCurrentProcessId();

        do
        {
            if (ProcessEntry.th32ProcessID == CurrentProcessId)
                break;
        } while (Process32Next(Snapshot, &ProcessEntry));
    }

    CloseHandle(Snapshot);

    return OpenProcess(SYNCHRONIZE, FALSE, ProcessEntry.th32ParentProcessID);
}

int WINAPI WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPSTR lpCmdLine, int ShowCmd)
{
    HANDLE ParentProcess = GetParentProcess();

    std::thread([ParentProcess]()
    {
        WaitForSingleObject(ParentProcess, INFINITE);
        ExitProcess(0);
    }).detach();

    CefMainArgs MainArgs(hInstance);
    return CefExecuteProcess(MainArgs, /* ... */);
}


[callum]

@Mikael - I'm working through it now but can you highlight which of those calls is 32bit only? Our application is 64 bit and I want to try a version of this in our code to see if it helps.

[Mikael]

@Mikael - I'm working through it now but can you highlight which of those calls is 32bit only? Our application is 64 bit and I want to try a version of this in our code to see if it helps.

It works just fine in 64-bit. In fact I've only tested it in 64-bit.