Crashing with large file inputs (FileReader)

[solarnoise]

Hi all,

I've been working very long on a CEF 3 tool for Adobe Photoshop. The primary feature requires file input. Everything has been working perfectly until I happened to try a file larger than 200 MB, and the extension appears to restart immediately (like hitting reload in browser).

(What's even weirder is if I had the DevTools open at the time, the extension page will load in the DevTools window and the original window goes blank. This might be Photoshop specific but thought I'd mention it if it's useful.)

I can see the File object is there, but as soon as I try creating an ArrayBuffer from it using FileReader.readAsArrayBuffer(), there is a 80-90% chance that the restart/crash happens. What's strange is it is not a guarantee that it will fail. Memory? FileReader bug?

Here's what I've tried:

1. Create an array of smaller blobs (50 MB) using File.slice
2. Loop through these to create the ArrayBuffer comprised of all the desired bytes, so instead of calling readAsArrayBuffer on the original File, it's done gradually on the smaller blobs
3. Create a DataView on this ArrayBuffer and proceed with the rest of the app (sometimes fails at this step now instead of earlier)

Unfortunately, all this did was reduce the failure rate to about 50%.

What can I do?? I've exhausted my JavaScript knowledge at this point (not an engineer by trade), but really want to see this tool through.

[magreenblatt]

The renderer process is likely crashing, perhaps running out of memory. The next step will be reproducing the problem in the cefclient sample application and providing a symbolized call stack for the crash. Downloads are available from http://cefbuilds.com. Debugging tips are available from viewtopic.php?f=6&t=10567 #19.

[solarnoise]

The renderer process is likely crashing, perhaps running out of memory. The next step will be reproducing the problem in the cefclient sample application and providing a symbolized call stack for the crash. Downloads are available from http://cefbuilds.com. Debugging tips are available from http://magpcss.org/ceforum/viewtopic.php?f=6&t=10567 #19.

Thank you magreenblatt. I have not had a chance to do the debugging but I can verify some results.

First of all, I figured out that the CEF implementation I'm targeting in Photoshop is a 2013 version, the Chromium version is 27.0.1453.110

I downloaded the oldest CEF test client available from the link you provided, and can confirm that the same crash happens with the same file. I tried a newer client and the crash did not happen, so at some point this was fixed.

At this point would the call stack still be helpful? Or does the fact that the crash is not present in newer clients mean that I may be out of luck for the CEF version I'm currently targeting?

[magreenblatt]

At this point would the call stack still be helpful?

Nope, no need to provide a call stack. Good to know that it's already fixed in newer versions.

[solarnoise]

At this point would the call stack still be helpful?

Nope, no need to provide a call stack. Good to know that it's already fixed in newer versions.

Sorry maybe I should clarify... I downloaded test clients from the CEF builds link, and could see that in a newer test client, the crash does not happen, but that doesn't help me as the CEF version Photoshop uses is from 2013. And I was able to repro the crash with the corresponding 2013 test client.

I have no control over what CEF version my tool is developed for.

So I guess what I'm asking is, am I out of luck with my tool entirely? Unless Adobe patches their software and upgrades the CEF version they're using, I'm stuck with what I have and unfortunately it's crashing with files about 150-200 MB in size.

Are there any creative workarounds I can try? As I detailed in the first post, I've tried slicing the File into smaller blobs and then reading them as array buffers, but even this causes the crash.

What's frustrating is, I can read a file that is 100 MB as array buffer just fine, but reading small chunks of ~10 MB in size eventually crashes once I've read 15-20 of them, so the chunking approach doesn't appear to be a good solution.

[solarnoise]

Anyone have any crafty ideas on how to get around this bug?

Is there some combination of slice/dataview that might allow me to view pieces of the file at a time but without incurring the crash?

I cannot know ahead of time where to slice up the file, I have to actually read through to find out where the pieces are that I need. So if I were to use a sliced blob approach, I would need to constantly check if I'm at the border of the blob/dataview and then go to a new slice. I'm worried even this would cause the crash I've been experiencing though.

[magreenblatt]

It's hard to say since you're using such an old CEF version. I think trial and error is your best option (unless you can perform the work using a separate process that you control and then provide the result to Photoshop).

[solarnoise]

It's hard to say since you're using such an old CEF version. I think trial and error is your best option (unless you can perform the work using a separate process that you control and then provide the result to Photoshop).

Good point.

One approach I am trying now is to actually use Photoshop's own JavaScript VM (their ancient "extendscript" they use for scripting directly with the host application) to read and split up the file into many smaller ones... and then use my CEF app to read in these smaller files individually. It's messy because now I've spread out the IO to two different layers of JavaScript but it's the only way I can see (for now) to get around this limitation.