Crash in IPC::ChannelPosix::ProcessOutgoingMessages()

[ronenmiz]

We have been getting the crash below with 2526:

#0 0x00007fa3810f6b9f in IPC::ChannelPosix::ProcessOutgoingMessages() ()
from /usr/local/tversity/x64/lib/libcef.so
#1 0x00007fa3810f70aa in IPC::ChannelPosix::Send(IPC::Message*) ()
from /usr/local/tversity/x64/lib/libcef.so
#2 0x00007fa3810f917f in IPC::ChannelProxy::Context::OnSendMessage(scoped_ptr<IPC::Message, base::DefaultDeleter<IPC::Message> >) ()
from /usr/local/tversity/x64/lib/libcef.so
#3 0x00007fa3810fa1c7 in base::internal::Invoker<base::IndexSequence<0ul, 1ul>, base::internal::BindState<base::internal::RunnableAdapter<void (content::NavigatorConnectContextImpl::*)(scoped_ptr<content::NavigatorConnectServiceFactory, base::DefaultDeleter<content::NavigatorConnectServiceFactory> >)>, void (content::NavigatorConnectContextImpl*, scoped_ptr<content::NavigatorConnectServiceFactory, base::DefaultDeleter<content::NavigatorConnectServiceFactory> >), base::internal::TypeList<content::NavigatorConnectContextImpl*, base::internal::PassedWrapper<scoped_ptr<content::NavigatorConnectServiceFactory, base::DefaultDeleter<content::NavigatorConnectServiceFactory> > > > >, base::internal::TypeList<base::internal::UnwrapTraits<content::NavigatorConnectContextImpl*>, base::internal::UnwrapTraits<base::internal::PassedWrapper<scoped_ptr<content::NavigatorConnectServiceFactory, base::DefaultDeleter<content::NavigatorConnectServiceFactory> > > > >, base::internal::InvokeHelper<false, void, base::internal::RunnableAdapter<void (content::NavigatorConnectContextImpl::*)(scoped_ptr<content::NavigatorConnectServiceFactory, base::DefaultDeleter<content::NavigatorConnectServiceFactory> >)>, base::internal::TypeList<content::NavigatorConnectContextImpl* const&, scoped_ptr<content::NavigatorConnectServiceFactory, base::DefaultDeleter<content::NavigatorConnectServiceFactory> > > >, void ()>::Run(base::internal::BindStateBase*)
() from /usr/local/tversity/x64/lib/libcef.so
#4 0x00007fa37f4b382f in base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask const&) () from /usr/local/tversity/x64/lib/libcef.so
#5 0x00007fa37f4cd2fb in base::MessageLoop::RunTask(base::PendingTask const&)
() from /usr/local/tversity/x64/lib/libcef.so
#6 0x00007fa37f4cd5f9 in base::MessageLoop::DeferOrRunPendingTask(base::PendingTask const&) () from /usr/local/tversity/x64/lib/libcef.so
#7 0x00007fa37f4cd7dd in base::MessageLoop::DoWork() ()
from /usr/local/tversity/x64/lib/libcef.so
#8 0x00007fa37f4aacb9 in base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) () from /usr/local/tversity/x64/lib/libcef.so
#9 0x00007fa37f4de950 in base::RunLoop::Run() ()
from /usr/local/tversity/x64/lib/libcef.so
#10 0x00007fa37f4ccc3a in base::MessageLoop::Run() ()
from /usr/local/tversity/x64/lib/libcef.so
#11 0x00007fa381cb6496 in content::BrowserThreadImpl::IOThreadRun(base::MessageLoop*) () from /usr/local/tversity/x64/lib/libcef.so
#12 0x00007fa381cb65c7 in content::BrowserThreadImpl::Run(base::MessageLoop*)
() from /usr/local/tversity/x64/lib/libcef.so
#13 0x00007fa37f4fd3a8 in base::Thread::ThreadMain() ()
from /usr/local/tversity/x64/lib/libcef.so
#14 0x00007fa37f4f9dcd in base::(anonymous namespace)::ThreadFunc(void*) ()
from /usr/local/tversity/x64/lib/libcef.so
#15 0x00007fa37cb9fe9a in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#16 0x00007fa37c2c338d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#17 0x0000000000000000 in ?? ()

The termination is with signal SIGSEGV, Segmentation fault.

A ticket for a similar issue has been opened a few months now:
https://bitbucket.org/chromiumembedded/ ... ult-in-ipc

To reproduce one needs to create and destroy browsers repeatedly for a while (sometime happens after a few minutes/hours and sometimes after days) hence it cannot be reproduced with cefclient.

[amaitland]

It appears you already posted on the bitbucket issue, is there any reason your posting here now? Do you have additional information to add?

[ronenmiz]

I did not open the ticket, I responded to it saying that I see the same issue. However after further examination there is a slight difference between the two, the first frame of the task in the ticket is different from the stack I provided. Therefore I figured I should post this stack as well. Moreover, there was no response from devs to the ticket which also prompted me to post to the forums, where responses seem more immediate.

In the meantime, I did some analysis and it seems like the message is corrupted and the crash occurs in the line:
size_t amt_to_write = msg->size() - message_send_bytes_written_;

Apparently in msg->size() where the header_ field is referenced.

I also found that already in Send() (previous stack frame) the message is corrupted.

To me this suggests that there may be some kind of buffer overflow in 2526 that leads to this. I tried using ASAN and the issue reproduces however ASAN does not provide any indication of a buffer overflow that preceded it, so for now I have no idea how to proceed, other than switch to an older version. Any suggestions?

[ronenmiz]

It appears that a heap user after free is causing this. See issue 1881:
https://bitbucket.org/chromiumembedded/ ... er-free-in

I am still testing the fix but initial results look good.

[ronenmiz]

It turns out there were two issues in browser_message_filter, the first was indeed a heap use after free of the RenderProcessHost object (Issue #1881) and the second is the use of RenderProcessHost:Send from the IO thread. It can only be used from UI thread or else a race condition occurs and it leads to corruption of the queue of IPC messages managed by RenderProcessHost. This corruption lead to the crash I describe here and I believe it also lead to Issue #1838.

A fix for both issues is provided in pull request #58.

[ronenmiz]

Correction, the fix is provided in pull request #59