Transfer large amount of data to Javascript

Having problems with building or using CEF's C/C++ APIs? This forum is here to help. Please do not post bug reports or feature requests here.

Transfer large amount of data to Javascript

Postby mohamedm » Tue Jan 16, 2018 5:50 pm

Hello everyone,

I would like to transfer a large amount of data (float* around 60MB) from the renderer process C++ to JS. I'm looking for a way to do this in a frame rate friendly kind of speed (i.e a few times a second). If there is a way to do this from the browser process, I could adapt.

So far, I modified CEF to add support for V8 ArrayBuffer and have been using that with some success (See the fork https://github.com/mmakhalaf/cef which only adds ArrayBuffer support in CefV8Value in branch 3239 based on this pull request https://bitbucket.org/chromiumembedded/ ... pport/diff ). Using ABs, I could send any amount of data almost instantaneously. However, there are some stability issues where an external AB crashes when used as WebGL buffer, and internal ABs crash during V8 garbage collection, which make me worried. This may well be due to how I implemented it, but I'm not sure what the solution is.

So the question is, how do i send data that big very quickly from Cpp to JS and vice versa?

Thank you in advance

Mohamed
mohamedm
Techie
 
Posts: 18
Joined: Mon Oct 16, 2017 3:25 am

Re: Transfer large amount of data to Javascript

Postby Czarek » Tue Jan 16, 2018 6:21 pm

Maintainer of the CEF Python, PHP Desktop and CEF C API projects. My LinkedIn.
User avatar
Czarek
Virtuoso
 
Posts: 1927
Joined: Sun Nov 06, 2011 2:12 am

Re: Transfer large amount of data to Javascript

Postby mohamedm » Tue Jan 16, 2018 7:04 pm

Thank you. I saw a lot of mentions about XMLHttpRequest. Also, in the latest versions of Cef, there is CefServer which allows us to send through WebSockets. These are for the browser process though, which I'm trying to steer away from if there is another way.

Do you or anyone have any timings for those method and/or knows about a way to communicate with the renderer Cpp as opposed to the browser?
mohamedm
Techie
 
Posts: 18
Joined: Mon Oct 16, 2017 3:25 am

Re: Transfer large amount of data to Javascript

Postby Czarek » Wed Jan 17, 2018 7:58 am

How about starting a simple http server (e.g. Mongoose) in renderer process?
Maintainer of the CEF Python, PHP Desktop and CEF C API projects. My LinkedIn.
User avatar
Czarek
Virtuoso
 
Posts: 1927
Joined: Sun Nov 06, 2011 2:12 am

Re: Transfer large amount of data to Javascript

Postby mohamedm » Wed Jan 17, 2018 9:50 am

I will have to benchmark that to see the performance. Though, judging by the CefServer using WebSockets to send the data, it may still be too slow unless a special / streaming-like protocol which isn't applicable at the moment.

I suppose what I would really be grateful for is to get help from anyone with chromium experience on the causes of the crash that occurs in V8 and WebKit when using ArrayBuffers.

Here is the workflow,
  • A CefV8Value is created using the CreateArrayBuffer() with some memory (which is allocated with malloc/realloc) and passed into a V8 function. The memory should now be owned by V8
  • JS function is executed taking the AB and stores it
  • ...
  • This is repeated over and over again assigning the incoming AB over the old JS ArrayBuffer
  • At some point (I'm guessing), the garbage collector kicks in and attempts to free memory, and that's when it crashes. I'm not sure how the GC works, so perhaps it's the first run or it was run multiple times

This is the code I use to create the ArrayBuffer in CEF ( from https://github.com/mmakhalaf/cef/blob/3 ... l.cc#L1328 ).
For reference, I looked at NodeJS's source to see how they use ArrayBuffers. They seem to return a TypedArray that's wrapped around the AB (not the AB), but I'm not sure why (See https://github.com/nodejs/node/blob/mas ... er.cc#L352 )
Code: Select all
CefRefPtr<CefV8Value> CefV8Value::CreateArrayBufferInternalized(
   void* data,
   size_t length) {
  CEF_V8_REQUIRE_ISOLATE_RETURN(NULL);

  v8::Isolate* isolate = GetIsolateManager()->isolate();
  v8::HandleScope handle_scope(isolate);

  v8::Local<v8::Context> context = isolate->GetCurrentContext();
  if (context.IsEmpty()) {
    NOTREACHED() << "not currently in a V8 context";
    return NULL;
  }

  // Create a tracker object that will cause the user data reference to be
  // released when the V8 object is destroyed.
  V8TrackObject* tracker = new V8TrackObject(isolate);
 
  // Create the new V8 array to own the given data
  v8::Local<v8::ArrayBuffer> arr = v8::ArrayBuffer::New(
      isolate,
      data,
      length,
      v8::ArrayBufferCreationMode::kInternalized);
 
  // Attach the tracker object.
  tracker->AttachTo(context, arr);

  CefRefPtr<CefV8ValueImpl> impl = new CefV8ValueImpl(isolate);
  impl->InitObject(arr, tracker);
  return impl.get();
}


A couple of assertions occur just before the crash (from the CEF debug.log),
Code: Select all
[0117/141039.613:FATAL:partition_alloc.h(595)] Check failed: super_page_offset < kSystemPageSize + (kNumPartitionPagesPerSuperPage * kPageMetadataSize).
[0117/141533.552:FATAL:partition_alloc.h(601)] Check failed: partition_page_index < kNumPartitionPagesPerSuperPage - 1.


The assertions are followed by an access violation crash. They occur with the same stack trace. It doesn't seem to be occurring from the main renderer thread.
Code: Select all
    libcef.dll!WTF::ArrayBufferContents::FreeMemory() Line 158   C++
    libcef.dll!v8::internal::LocalArrayBufferTracker::Free<<lambda_74d483d3add1142e410cc7f2df77873c> >() Line 63   C++
    libcef.dll!v8::internal::MarkCompactCollector::Sweeper::RawSweep() Line 3570   C++
    libcef.dll!v8::internal::MarkCompactCollector::Sweeper::ParallelSweepPage() Line 4431   C++
    libcef.dll!v8::internal::MarkCompactCollector::Sweeper::ParallelSweepSpace() Line 4402   C++
    libcef.dll!v8::internal::PagedSpace::RawSlowAllocateRaw() Line 3214   C++
    libcef.dll!v8::internal::PagedSpace::AllocateRawUnaligned() Line 335   C++
    libcef.dll!v8::internal::PagedSpace::AllocateRaw() Line 391   C++
    libcef.dll!v8::internal::LocalAllocator::Allocate() Line 50   C++
    libcef.dll!v8::internal::FullEvacuator::RawEvacuatePage() Line 3305   C++
    libcef.dll!v8::internal::Evacuator::EvacuatePage() Line 3231   C++
    libcef.dll!v8::internal::PageEvacuationTask::RunInParallel() Line 3409   C++
    libcef.dll!v8::internal::ItemParallelJob::Task::RunInternal() Line 98   C++
    libcef.dll!base::OnceCallback<void __cdecl(void)>::Run() Line 64   C++
    libcef.dll!base::debug::TaskAnnotator::RunTask() Line 58   C++
    libcef.dll!base::internal::TaskTracker::RunOrSkipTask() Line 412   C++
    libcef.dll!base::internal::TaskTracker::RunNextTask() Line 312   C++
    libcef.dll!base::internal::SchedulerWorker::Thread::ThreadMain() Line 72   C++
    libcef.dll!base::`anonymous namespace'::ThreadFunc() Line 91   C++


Any help or pointers would be be appreciated. ArrayBuffers to me seem to provide the quickest way to transfer data between the renderer process Cpp and JS, and I would like to submit this to CEF once I can get it to work reliably.

Many thanks

Mohamed
mohamedm
Techie
 
Posts: 18
Joined: Mon Oct 16, 2017 3:25 am

Re: Transfer large amount of data to Javascript

Postby Czarek » Wed Jan 24, 2018 6:29 am

Just saw an another PR that adds V8 ArrayBuffer:
https://bitbucket.org/chromiumembedded/ ... ized-array
Maintainer of the CEF Python, PHP Desktop and CEF C API projects. My LinkedIn.
User avatar
Czarek
Virtuoso
 
Posts: 1927
Joined: Sun Nov 06, 2011 2:12 am

Re: Transfer large amount of data to Javascript

Postby mohamedm » Mon Aug 06, 2018 4:14 pm

ArrayBuffer support has been added in 3359.

Thank you!
mohamedm
Techie
 
Posts: 18
Joined: Mon Oct 16, 2017 3:25 am


Return to Support Forum

Who is online

Users browsing this forum: No registered users and 50 guests