Native/javascript shared ArrayBuffer crash

[sdiverdi]

Ah and I see now that what's going wrong in the trace in that image, ReleaseBuffer is where the error is raised (inside the backing store deleter) because at that point the ReleaseCallback object has been deleted. Switching it to a singleton object so the pointer never goes bad.

[magreenblatt]

I suggest you wait for the proper fix in the PR.

[sdiverdi]

Hi! I was super excited to see that the PR above went live, thanks for that! Yesterday I patched it into my CEF build (branch 4324) aaaand...still crashing on ReleaseBuffer! What the heck? I've done more investigating and I'm pretty unclear what's going on, but it looks like some memory is getting corrupted somehow.

To recap, I am using an ArrayBuffer to inject video frames from c++ into javascript. In my renderer helper process I create an ArrayBuffer in my js context, and in js, I draw the ArrayBuffer into a canvas. Every once in a while, the renderer helper process crashes. The backtrace shows what causes the crash is at some point the GC decides to delete a stale ArrayBuffer, and the BackingStore destructor calls the ArrayBuffer custom deleter, which tries to call a bad function pointer.

trace.jpg (343.61 KiB) Viewed 4769 times

In that image, I've also confirmed that the buffer_ pointer is incorrect as well (or at least it's not the pointer of a buffer that I've allocated). Note that I'm just allocating the one ArrayBuffer and then writing into the buffer repeatedly, not re-allocating it, so I'm not sure what ArrayBuffer is being deleted anyway. Also my CefV8ArrayBufferReleaseCallback is a static singleton, so it's not like the function pointer goes bad when the object goes away.

Any ideas on what might be going on here, or how to investigate?

[magreenblatt]

Are you navigating away from the frame that's using the ArrayBuffer? Are you keeping a global reference to the ArrayBuffer in JS? Is your CefV8ArrayBufferReleaseCallback being called at all?

[magreenblatt]

Can you also set a breakpoint in ~V8TrackArrayBuffer() and see what call stack is deleting the object?

[sdiverdi]

Thanks for getting back to me! I did some more testing to get rid of things that don't affect the crash. So: no navigating away from the frame, I have one browser/frame that I do the array buffer injection into with no navigation (and one other OSR browser that also does not navigate). To trigger allocations I change the resolution of the OSR browser repeatedly, each time causing a new array buffer to be created of the correct size. I do this a bunch and eventually a GC sweep occurs, which called the buffer release callback for all the old array buffers, and then crashes. Here you can see the list of allocations and releases:

Code: Select all

ALLOCATE ARRAY BUFFER prefix=preview size=8294400 ptr=0x116800000
ALLOCATE ARRAY BUFFER prefix=preview size=3686400 ptr=0x1211e9000
ALLOCATE ARRAY BUFFER prefix=preview size=921600 ptr=0x120500000
ALLOCATE ARRAY BUFFER prefix=preview size=3686400 ptr=0x122384000
ALLOCATE ARRAY BUFFER prefix=preview size=8294400 ptr=0x120a00000
ALLOCATE ARRAY BUFFER prefix=preview size=3686400 ptr=0x122000000
ALLOCATE ARRAY BUFFER prefix=preview size=921600 ptr=0x121700000
ALLOCATE ARRAY BUFFER prefix=preview size=3686400 ptr=0x1230e9000
ALLOCATE ARRAY BUFFER prefix=preview size=8294400 ptr=0x122900000
ALLOCATE ARRAY BUFFER prefix=preview size=3686400 ptr=0x123600000
ALLOCATE ARRAY BUFFER prefix=preview size=921600 ptr=0x1205e1000
ALLOCATE ARRAY BUFFER prefix=preview size=3686400 ptr=0x124e84000
ALLOCATE ARRAY BUFFER prefix=preview size=8294400 ptr=0x12416d000
ALLOCATE ARRAY BUFFER prefix=preview size=3686400 ptr=0x125208000
RELEASE ARRAY BUFFER ptr=0x12416d000 prefix=preview buffer=0x12416d000 size=8294400
RELEASE ARRAY BUFFER ptr=0x124e84000 prefix=preview buffer=0x124e84000 size=3686400
RELEASE ARRAY BUFFER ptr=0x1205e1000 prefix=preview buffer=0x1205e1000 size=921600
RELEASE ARRAY BUFFER ptr=0x123600000 prefix=preview buffer=0x123600000 size=3686400
RELEASE ARRAY BUFFER ptr=0x122900000 prefix=preview buffer=0x122900000 size=8294400
RELEASE ARRAY BUFFER ptr=0x1230e9000 prefix=preview buffer=0x1230e9000 size=3686400
RELEASE ARRAY BUFFER ptr=0x121700000 prefix=preview buffer=0x121700000 size=921600
RELEASE ARRAY BUFFER ptr=0x122000000 prefix=preview buffer=0x122000000 size=3686400
RELEASE ARRAY BUFFER ptr=0x120a00000 prefix=preview buffer=0x120a00000 size=8294400
RELEASE ARRAY BUFFER ptr=0x122384000 prefix=preview buffer=0x122384000 size=3686400
RELEASE ARRAY BUFFER ptr=0x120500000 prefix=preview buffer=0x120500000 size=921600
RELEASE ARRAY BUFFER ptr=0x1211e9000 prefix=preview buffer=0x1211e9000 size=3686400
RELEASE ARRAY BUFFER ptr=0x116800000 prefix=preview buffer=0x116800000 size=8294400


I've matched them up and all but the last allocate (which is still live) are matched with a correct release (the "RELEASE" statements are printed in my release callback). The stack for one of those ~V8TrackArrayBuffer calls (one for each array buffer) looks like this:

tracker.jpg (324.91 KiB) Viewed 4765 times

and the crash occurs immediately after the last ~V8TrackArrayBuffer, with this trace:

crash.jpg (356.23 KiB) Viewed 4765 times

One thing I've noticed: the crash occurs when freeing an ArrayBufferContents or an ArrayBufferExtension, which seem to own BackingStores that are then deleted. I'm not sure what those types are in relation to the ArrayBuffer type that I create. It feels like a BackingStore is getting stored in two places that are both trying to delete it?

Oh to your last question I've tried both storing and not storing a global reference to the array buffer in js (it's injected by attaching it to the global window object, but I've created a separate variable to hold it as well) and haven't been able to see a change in behavior but I can investigate that a bit more too.

[magreenblatt]

Thanks for the info. I’ll try to fix it next week.

[sdiverdi]

Awesome, looking forward to it! I hacked up cefsimple to reproduce the error if that helps, attached. I ran this with my libcef build 4324 with your commit patched in ( https://github.com/chromiumembedded/cef ... 3fd4243a1d ), and it will reliably crash the renderer process pretty quickly (less than 30sec). The crash is slightly different, it actually gets into the ReleaseBuffer callback but raises a malloc error that the pointer entry vanished. I wonder if it's a race to see if ~BackingStore or ~V8TrackArrayBuffer calls ReleaseBuffer first?

simple.jpg (248.33 KiB) Viewed 4757 times

[magreenblatt]

Very helpful, thanks

[magreenblatt]

Please try the new fix and let me know if you're still having issues. Thanks.