v8 crash, Allocation failed - process out of memory ?

[heshiming]

Hi,

I've been testing out CEF to run my application. And I always encounter a crash that points back to V8Proxy after minutes of automatic navigating.

I haven't been able to create a minimal set of code to reproduce it yet. But my javascript is quite simple. Each "onload" there's a function to declare an array:

var myarray = new Array();
myarray['map'] = "mapped string";
myarray['othermap'] = "other mapped string";

then there are calls to "document.getElementById" to read out some hidden form values, and print formatted values to the page by accessing innerHTML.

The program that hosts CEF will attempt to refresh the page filled with random content, served by a custom scheme handler, then execute that javascript, at the timer of 1 second per refresh. Within several minutes, there will be a crash that points back to line 226 of V8Proxy.cpp, the call stack is below:

> libcef.dll!WebCore::reportFatalErrorInV8(const char * location=0x5b03b7b4, const char * message=0x5b022ad0) Line 226 C++
libcef.dll!v8::internal::V8::FatalProcessOutOfMemory(const char * location=0x5b03b7b4) Line 179 C++
libcef.dll!v8::internal::Factory::NewJSObject(v8::internal::Handle<v8::internal::JSFunction> constructor={...}, v8::internal::PretenureFlag pretenure=TENURED) Line 647 + 0x20 bytes C++
libcef.dll!v8::internal::Genesis::CreateRoots(v8::Handle<v8::ObjectTemplate> global_template={...}, v8::internal::Handle<v8::internal::Object> global_object={...}) Line 779 + 0x11 bytes C++
libcef.dll!v8::internal::Genesis::Genesis(v8::internal::Handle<v8::internal::Object> global_object={...}, v8::Handle<v8::ObjectTemplate> global_template={...}, v8::ExtensionConfiguration * extensions=0x0301f3bc) Line 1592 C++
libcef.dll!v8::internal::Bootstrapper::CreateEnvironment(v8::internal::Handle<v8::internal::Object> global_object={...}, v8::Handle<v8::ObjectTemplate> global_template={...}, v8::ExtensionConfiguration * extensions=0x0301f3bc) Line 380 C++
libcef.dll!v8::Context::New(v8::ExtensionConfiguration * extensions=0x0301f3bc, v8::Handle<v8::ObjectTemplate> global_template={...}, v8::Handle<v8::Value> global_object={...}) Line 2789 + 0x15 bytes C++
libcef.dll!WebCore::V8Proxy::createNewContext(v8::Handle<v8::Object> global={...}, int extensionGroup=0) Line 990 C++
libcef.dll!WebCore::V8Proxy::initContextIfNeeded() Line 1084 + 0x11 bytes C++
libcef.dll!WebCore::V8Proxy::updateDocument() Line 844 C++
libcef.dll!WebCore::Frame::setDocument(WTF::PassRefPtr<WebCore::Document> newDoc={...}) Line 292 C++
libcef.dll!WebCore::FrameLoader::begin(const WebCore::KURL & url={...}, bool dispatch=false, WebCore::SecurityOrigin * origin=0x00000000) Line 830 C++
libcef.dll!WebCore::FrameLoader::receivedFirstData() Line 749 C++
libcef.dll!WebCore::FrameLoader::setEncoding(const WebCore::String & name={...}, bool userChosen=false) Line 1466 C++
libcef.dll!WebKit::WebFrameImpl::commitDocumentData(const char * data=0x0979c4e0, unsigned int dataLen=2107) Line 879 C++
libcef.dll!WebKit::FrameLoaderClientImpl::committedLoad(WebCore::DocumentLoader * loader=0x098d5140, const char * data=0x0979c4e0, int length=2107) Line 1030 C++
libcef.dll!WebCore::FrameLoader::committedLoad(WebCore::DocumentLoader * loader=0x098d5140, const char * data=0x0979c4e0, int length=2107) Line 3240 + 0x16 bytes C++
libcef.dll!WebCore::DocumentLoader::commitLoad(const char * data=0x0979c4e0, int length=2107) Line 343 C++
libcef.dll!WebCore::DocumentLoader::receivedData(const char * data=0x0979c4e0, int length=2107) Line 354 + 0xf bytes C++
libcef.dll!WebCore::MainResourceLoader::addData(const char * data=0x0979c4e0, int length=2107, bool allAtOnce=false) Line 143 + 0x10 bytes C++
libcef.dll!WebCore::ResourceLoader::didReceiveData(const char * data=0x0979c4e0, int length=2107, __int64 lengthReceived=2107, bool allAtOnce=false) Line 252 C++
libcef.dll!WebCore::MainResourceLoader::didReceiveData(const char * data=0x0979c4e0, int length=2107, __int64 lengthReceived=2107, bool allAtOnce=false) Line 375 C++
libcef.dll!WebCore::ResourceLoader::didReceiveData(WebCore::ResourceHandle * __formal=0x0da72188, const char * data=0x0979c4e0, int length=2107, int lengthReceived=2107) Line 398 + 0x18 bytes C++
libcef.dll!WebCore::ResourceHandleInternal::didReceiveData(WebKit::WebURLLoader * __formal=0x0d869ad8, const char * data=0x0979c4e0, int dataLength=2107) Line 141 + 0x20 bytes C++
libcef.dll!webkit_glue::WebURLLoaderImpl::Context::OnReceivedData(const char * data=0x0979c4e0, int len=2107) Line 479 + 0x17 bytes C++
libcef.dll!`anonymous namespace'::RequestProxy::NotifyReceivedData(int bytes_read=2107) Line 203 C++
libcef.dll!RunnableMethod<URLRequestFtpJob,void (__thiscall URLRequestFtpJob::*)(int),Tuple1<int> >::Run() Line 289 + 0xf bytes C++
libcef.dll!MessageLoop::RunTask(Task * task=0x02aea400) Line 321 C++
libcef.dll!MessageLoop::DoWork() Line 435 + 0xb bytes C++
libcef.dll!base::MessagePumpForUI::DoRunLoop() Line 210 C++
libcef.dll!base::MessagePumpWin::Run(base::MessagePump::Delegate * delegate=0x0301f8c8) Line 78 + 0x3e bytes C++
libcef.dll!MessageLoop::RunInternal() Line 205 + 0xb bytes C++
libcef.dll!MessageLoop::Run() Line 156 C++
libcef.dll!ThreadHandlerUI(void * lpParam=0x003e4de8) Line 249 C++
kernel32.dll!767feccb()
[Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]
ntdll.dll!775fd24d()
ntdll.dll!775fd45f()

In the crashing function: "static void reportFatalErrorInV8(const char* location, const char* message)", location = "CALL_AND_RETRY_2", message = "Allocation failed - process out of memory".

The process started out using about 160MB-ish of RAM, and crashed at some 175MB-ish. The crash is random and I can't link it to a particular piece of javascript of page content. But it always happens after several minutes of running.

I synced chromium to the latest and it's still there.

Is there any way I could debug it and find out about the cause?

[heshiming]

Hi, I'm able to create a minimal set of code that could reproduce this problem. It's based on existing cefclient code.

In cefclient.cpp, around line 710's case WM_TIMER, change:

Code: Select all

    case WM_TIMER:
      if(g_handler.get() && g_handler->GetBrowserHwnd())
      {
        // Retrieve the current navigation state
        bool isLoading, canGoBack, canGoForward;
        // Update the status of child windows
        EnableWindow(editWnd, TRUE);
        EnableWindow(backWnd, canGoBack);
        EnableWindow(forwardWnd, canGoForward);
        EnableWindow(reloadWnd, !isLoading);
        EnableWindow(stopWnd, isLoading);
      }


to:

Code: Select all

    case WM_TIMER:
   switch(wParam)
   {
   case 1:
      if(g_handler.get() && g_handler->GetBrowserHwnd())
      {
         // Retrieve the current navigation state
         bool isLoading, canGoBack, canGoForward;
         g_handler->GetNavState(isLoading, canGoBack, canGoForward);
         // Update the status of child windows
         EnableWindow(editWnd, TRUE);
         EnableWindow(backWnd, canGoBack);
         EnableWindow(forwardWnd, canGoForward);
         EnableWindow(reloadWnd, !isLoading);
         EnableWindow(stopWnd, isLoading);
      }
      break;
   case 100:
      {
         CefRefPtr<CefBrowser> browser;
         if(g_handler.get())
            browser = g_handler->GetBrowser();
         if(browser.get())
            RunSchemeTest(browser);
      }
      break;
   }


and around line 816, change the WM_COMMAND id handler, from:

Code: Select all

        case ID_TESTS_SCHEME_HANDLER: // Test the scheme handler
          if(browser.get())
            RunSchemeTest(browser);
          return 0;


to:

Code: Select all

        case ID_TESTS_SCHEME_HANDLER: // Test the scheme handler
      ::SetTimer(hWnd, 100, 200, NULL);


and then in scheme_test.cpp, add the following after #includes:

Code: Select all

int g_passes = 0;


then at line 36, change the html code to something like this:

Code: Select all

      html_ = "<html><head><title>Client Scheme Handler</title>"
"<script language=\"javascript\">"
"var myarray = new Array();"
"myarray['abc'] = 'abfdasfasdcsdfasdf';"
"myarray['abcd'] = 'abafdsasdfcd';"
"myarray['abcde'] = 'abadsfasdfcde';"
"myarray['abcdef'] = 'abadfasdfcdef';"
"myarray['abca'] = 'abadfser54ca';"
"myarray['abcab'] = 'abcdsertyy54yab';"
"myarray['abcabc'] = 'abcjkuyjk7abc';"
"myarray['abcg'] = 'abktyk67tkcg';"
"myarray['abcgh'] = 'abck6k67ui6trugh';"
"myarray['abcghi'] = 'abcfgncheyw454w5y544ghi';"
"myarray['abcj'] = 'abcdjjytu5674567j';"
"myarray['abcjk'] = 'abc25j54645tygk';"
"myarray['abcjkl'] = 'abcjsty4t34kl';"
"myarray['abc1'] = 'ab543554c1';"
"myarray['abc12'] = '343wrfaseerabc12';"
"myarray['abc123'] = '45tst4etabc123';"
"myarray['1abc'] = '45s4tgtgsabc';"
"myarray['1abcd'] = '56t5e4ytsdrgabcd';"
"myarray['1abcde'] = 'ab6s55rtys4st6srdcde';"
"myarray['1abcdef'] = 'asrtys4tdrserbcdef';"
"myarray['1abca'] = 'absetsretgs44tsewrtgca';"
"myarray['1abcab'] = 'ab4t55s4t45sretgscab';"
"myarray['1abcabc'] = 'abtse4tsetseertscabc';"
"myarray['1abcg'] = 'abs4t5s545s4et4s4ecg';"
"myarray['1abcgh'] = 'as4st44s4etswbcgh';"
"myarray['1abcghi'] = 'as4tst4rs4ts4tbcghi';"
"myarray['1abcj'] = 'as4ts4ts4ts4tbcj';"
"myarray['1abcjk'] = 'abst43335txer5srtgcjk';"
"myarray['1abcjkl'] = 'abregerrt544ysdtrysdretcjkl';"
"myarray['1abc1'] = 'abc4twetsdrgdhrtyhd1';"
"myarray['1abc12'] = 'ab56r5etydtyhc12';"
"myarray['1abc123'] = 'abc5tyrydthcf123';"
"myarray['2abc'] = 'ae4t76iuy8ioguyhkfgubc';"
// ... etc
"</script>"
"</head><body><div id=\"content\"></div>"


finally, change RunSchemeTest into:

Code: Select all

   wchar_t url[256];
   swprintf_s(url, 256, L"client://tests/handler.html%d", g_passes);
   g_passes++;
  browser->GetMainFrame()->LoadURL(url);


The main idea here is to let cefclient refresh the page with a new URL each 200 milliseconds. And in each of the pages, there are the same javascript to create a page-global string hash (array). And as you can see there are no other scripts.

If you now run the scheme test, cefclient will always crash at around "client://tests/handler.html903". It takes about 3 minutes to get there. And the crash points to the V8Proxy.cpp file illustrated in the previous post.

I've tried to change the Array in javascript into a switch statement that does the same thing. This has no effect. The crash still happens the same way.

Both changed files are available in the attachment.

[heshiming]

The compiled cefclient executable can be obtained here. Choose "Scheme test" from the menu to run.

[heshiming]

And there's one more thing. When the automatic refreshing is running, dragging the window will cause the refresh to be blocked. And when you release the mouse button, it crashes AsyncResolver::Resolve of scheme_impl.cc, because |owner_| = NULL. The error is: Access violation reading location 0x00000008 .

To fix this problem, I made the following changes in scheme_impl.cc :

Line 181, changed "void Resolve(const GURL& url)" to "void Resolve(const GURL url)".
Line 183, after the lock, put "if (owner_ && owner_loop_)" to wrap the rest.
Line 212, put the lock above "owner_ = NULL"
Line 217, change "void ReturnResults(const GURL& url)" to "void ReturnResults(const GURL url)"

Like the one attached (like the ones before, there are some coding style issues though).

I wish you could review it, and make it to the official tree.

[magreenblatt]

Thank you for reporting the crash in AsyncResolver::Resolve and taking the time to develop a possible solution. Please create an issue for it in the CEF issue tracker and post a link to that issue here so that we can better track its resolution.

Are you able to reproduce the out-of-memory situation in the test_shell application? Are you seeing any CEF-created objects that are not being properly released after each automatic refresh? It could be that you have (re)discovered a bug in the V8 proxy layer.

Regards,
Marshall

[heshiming]

Hi Marshall,

I've filed issue 65: http://code.google.com/p/chromiumembedd ... tail?id=65 for the AsyncResolver::Resolve issue.

I don't know whether the v8 problem is related to CEF or not, but I discovered another problem, that could be a memory leak.

It's easy to see if you take the compiled executable in my earlier post and run it. Select the scheme handler test, cefclient.exe's memory consumption rises quickly as navigating happens. It started at about 25MB, and ended at 140MB due to v8 crash, with about 900 navigates, which comes to around 130KB increase per navigate (LoadURL).

And it's not only happening to scheme handlers. If I set a slower rate timer to let cefclient automatically navigate to http://www.google.com/handler.html%d (you'll get 404), the process memory also increases at a similar rate. I'm entering this as issue 66: http://code.google.com/p/chromiumembedd ... tail?id=66 .

I'm wondering if it's some kind of chromium's internal memory cache? Is there any way to flush it?

Thanks for looking into this.

[magreenblatt]

Chromium maintains a number of internal memory caches. V8 caching in particular is problematic and there doesn't seem to be a clear way to ensure that all cached V8 objects are released. You may want to explore how Peter Kasting implemented his memory purger as described in this thread:
http://groups.google.com/group/chromium ... 12ae1a1d26

[heshiming]

Thanks for the info on memory purger. I"m still not sure if it's a problem of v8 or chromium. Here is what I discovered today though, if I comment out all the javascript code, and only let CEF navigate to a pure HTML page, it still crashes at around 900 navigates, with trace pointing back to the same V8Proxy.cpp . I guess I have to try reproducing it on test_shell and maybe file a bug report to chromium.

[heshiming]

Hi Marshall,

I've tried reproducing it on test_shell, and I didn't succeed.

I changed small parts in test_shell_win.cc , added a WM_TIMER in TestShell::WndProc , like:

Code: Select all

case WM_TIMER:
   if(nav_count%2 == 0)
      shell->LoadURL(GURL(std::string("test-shell-resource://inspector/inspector.html")));
   else
      shell->LoadURL(GURL(std::string("test-shell-resource://inspector/devtools.html")));
   nav_count++;
   break;


|nav_count| is a global int, initialized with 0. Protocol "test-shell-resource" is built-in, and the two html files are there by default. The timer is set to fire at 1 sec.

So test_shell.exe starts with process memory: 35MB as Working Set (WS), and 6MB as Private Working Set(PWS). And I trigger the timer to start, the process memory starts to climb until 80MB WS or 65MB PWS. Then it shrinks back to 68MB WS / 54MB PWS. For the first half an hour, it's been a round trip, it never gets very high like the cefclient case. And V8 never crashes.

Therefore, I think the v8 crash problem, together with the memory leak issue, is particular to the implementation of CEF. I'll try and look into this.

Now, after 3 hours of running, test_shell's process memory increased from the 68-80MB round trip, to about 73MB-90MB. It kept increasing but at a very slow rate. I guess chromium has some kind of memory leak too. But since v8 is still not crashing after 3 hours (about 10000 navigates), it might not be that serious as the CEF one.

[heshiming]

Hi Marshall,

So I discovered at least the leak point of this problem.

I compared CEF to test_shell, and found that the leak is happening in CefHandler::HandleJSBinding . This is called upon each navigation, from webview delegate's didClearWindowObject. It calls CefV8Value::CreateObject . I don't know the exact objects being leaked. However, by commenting out the calls to CefV8Value::CreateObject, that is, to remove the "v8 function handler test (function binding)", the memory leak disappears. I was able to get cefclient to stay at 43MB-47MB of process memory, similar to test_shell .

The weird thing is that I tapped into CefV8ValueImpl, and discovered that it's always properly destroyed. So CEF itself might not be responsible for the leak. It could be somewhere inside WebKit or V8.

So as I said, to prevent this from happening, simply don't use the "v8 function handler".