-
Notifications
You must be signed in to change notification settings - Fork 760
Large file save freeze app for seconds #364
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Where is I have an app where I write a rather large amount of data to a file, upwards of 50mb in size and I don't really see a huge freeze. |
I download png files with xhr and save them without any processing. |
When you download your png files, are you using blobs or are you referencing that data via strings? |
I download it as blob. |
Very strange indeed. If possible, if you can create a sample reproduction app that reproduces the issue, it might help move this ticket along, though this may very well be device dependent. |
Hello again! Click on load button for load file, then click save button for save. =) |
Thanks, I'll try to take a quick look at this this weekend and report what I find on my devices. |
Hi, any update on this? |
My apologies, I've been really tied up with work trying to meet a deadline and I completely forgot about this. I'll take a look now just to at least confirm that I can reproduce the issue, but I'm not sure when I'll be able to take a deep look. Edit: Definitely can see the ~4 second pause when I click the "Save Image" button. I'll have to investigate further to see exactly where the bottleneck is, but I'm not sure when I'll have the time to do so since I've been doing some overtime at my workplace. |
The line that blocks the UI thread for a significant amount of time here I found to be:
When blobs are used, it reads the blob as an arraybuffer. Then it converts the array buffer to a base64 encoded string. I assume it does this because the JavasciptInterface I believe can only accept strings. Unfortunately, I don't think there is any way around that, based on some brief googling around. This is a weak workaround since this plugin is deprecated, you can try using the file transfer plugin which allows you to download a file and save it to the file system completely on the native side. I do still use the file transfer plugin myself in my own apps and it still works, but it is a plugin that cordova doesn't provide updates or support for. Possible ideas for solutions: Cordova doesn't support web workers, but I think this would be a good case where web workers would be beneficial, to at least move the bridge execution calls off the UI thread. It won't improve performance, but at least it will make Another idea is to bring the concept of file transfers into the |
My knowledge of web workers is very limited. I think i cant make cordova calls from worker, so i need to pass my file to worker, convert it to base64 in worker, pass it back to main and call save() ? |
You are correct, this is something that cordova doesn't currently support.
You can try this, but I'm not certain if that will help as there is a bottleneck of copying the data to and from the worker. If you're downloading a file and saving it to disk, you're best/quickest work around is probably using the File Transfer Plugin despite it being deprecated. Because it downloads the file and saves it to the filesystem all on the native side. The primary reason why the transfer plugin is deprecated to my understanding is because it is possible to download binary files using |
We can convert the blob into chunks and write the file in parts:
Implemented this in a Capacitor project which works very nicely:
Edit: 20MB took arround 2 seconds to write |
I have My code for that: self.requestFileSystem(LocalFileSystem.PERSISTENT, 0, function (fs) {
fs.root.getFile(file_name, { create: true, exclusive: false }, function (fileEntry) {
var reader = new FileReader();
reader.onloadend = function () {
fileEntry.createWriter(function (fileWriter) {
fileWriter.onwriteend = function () {};
fileWriter.onerror = function (e) {};
dataObj = new Blob([reader.result], { type: 'text/plain' });
// freeze here
fileWriter.write(dataObj);
postMessage('ok');
});
};
reader.readAsDataURL($(file_id)[0].files[0]);
}, onErrorCreateFile);
}, onErrorLoadFs); But this code freeze my UI on device (ios, android) up to 10 seconds when file is 30MB. I discovered the world of Web Workers, but there is no access to How can I save selected file ( |
I have a similar issue when downloading a large file (100Mb) in iOS. but in my case the app completely freezes and goes to white screen of death. |
We are also hitting this issue when downloading large video files (>25MB) on iOS. Even worse, wkwebview decides that the page is unresponsive, and reloads it, terminating our download task before it completes successfully. |
We tried the chunking method described above, but unfortunately wkwebview still eventually decides that the page is unresponsive. We experimented with chunk sizes from 512KB to 15MB. |
File transer plugin was somewhat "undeprecated", was upgraded to be available for latest cordova and shows good performance and results. Solved most of the issues my app was experiencing and it can also download when the app is not inn the front which is a nice plus. I recommend using it. |
We just tested the file transfer plugin and it works perfectly for us. Thank you. |
We were running into some of the same issues. We try to download files via XMLHttpRequest, and write the contents via a fileWriter. Writing files larger than a few megabytes would consistently crash the whole application. We switched to writing the files in chunks, which alleviated the problem, but I was still confused why writing files less than 50MiB would result in crashes or white-screens. I profiled with chromes debugging tools, and found that the garbage collector is running like crazy, and the function uses a lot of memory. You can see here, that while invoking As already mentioned by @breautek, this part in cordova is really slow: I wanted to understand what is happening here: function uint8ToBase64 (rawData) {
var numBytes = rawData.byteLength;
var output = '';
var segment;
var table = b64_12bitTable();
for (var i = 0; i < numBytes - 2; i += 3) {
segment = (rawData[i] << 16) + (rawData[i + 1] << 8) + rawData[i + 2];
output += table[segment >> 12];
output += table[segment & 0xfff];
}
if (numBytes - i === 2) {
segment = (rawData[i] << 16) + (rawData[i + 1] << 8);
output += table[segment >> 12];
output += b64_6bit[(segment & 0xfff) >> 6];
output += '=';
} else if (numBytes - i === 1) {
segment = (rawData[i] << 16);
output += table[segment >> 12];
output += '==';
}
return output;
} Ouch. JavaScript has immutable strings. Using Note that if you want to write 1MiB, you would have to allocate several gigabytes of memory! This explains the memory usage, and why writing binary files is slow. The garbage collector needs to clean up millions of strings, and if it can't keep up, we run out of memory. This not only affects writing files, but ANY cordova plugin that wants to pass binary data from JS to the native layer. The conversion via I thought it would be reasonable to support So I started to change the I am also not sure why @digaus solution works. When I pass a string to FileWriter.write(), it should write the strings as-is to disk, instead of interpreting it as binary data encoded in base64?
In order to get Does I will spend some more time on this. I would really like to solve this issue and make a PR, once I get this to work. |
I have managed to change FileWriter, such that it can convert ArrayBuffers to base64 encoded strings itself. To get the correct base64 encoded string, it is important to call
Otherwise you would encode the string representation of the ArrayBuffer, which is not at all what you want. I have used this to write 1MiB and smaller files, and the performance looks better for now. Especially the memory usage for each file or chunk is only about twice the filesize/chunksize. There is less garbage collection happening. I will test this with bigger files and measure the timing in comparison to the original version, to see if it actually performs better, and implement chunking, if that seems necessary. |
Thanks for the detailed analysis and potential fix(es) @LightMind - looking forward to hearing more from you. |
I have implemented the chunking behaviour in FileWriter.js. This does not go so well for several megabyte of data. I tried with chunks of 128KiB, 1MiB, and 5MiB. Larger chunks are faster to write, having less downtime between writes, but the memory spikes are worse. Memory usage is mostly due to When writing 1MiB chunks, there is more downtime, but much less memory pressure. Writing smaller chunks does not make sense, as the downtime between writes is too much, halving the write speed at chunks of 128KiB. I have some questions, but I'll take that when I make a PR, which I will do soon. |
If this is an issue with cordova-common, did you report it? I'd like the see this issue's status in the repo that owns it. |
@kputh This cant be an issue from cordova-common. cordova-common is not bundled with any app. From the content of this thread, it sounds like you might mean cordova-js. This file specifically: https://github.com/apache/cordova-js/blob/master/src/common/base64.js |
I filed a PR in cordova-js that tries to improve the performance for big files as much as is possible with the current architecture. However, I think the only solid solution to this problem is to not process big files in the browser's main thread. It seems the file-transfer-plugin caters well to the common use case of simply downloading and storing files. Chunked reading and writing seems to be a reasonable alternative if you really need to transfer a lot of bytes over the web/native bridge. However, I'm not quite sure if we should promote that by including it into this plugin. One thing that we should definitely consider for this plugin is the move from the synchronous @LightMind Thanks so much for your detailed analysis of the problem. You really pushed this issue forward! |
@raphinesse Thank you so much, I really appreciate it :) I am also a little bit worried about changing the behaviour of This makes me unsure about if or how to continue #461 Would it be reasonable to provide a Something like
|
@LightMind First off: sorry for the late reply.
Agreed. Maybe a
If we want a chunked writer in this plugin, I think that would be the way to go. Another interesting approach can be seen in capacitor-blob-writer. It streams the data out of the WebView by means of network requests. Seems more reasonable than pushing everything over the bridge in base64 form. Moreover, this approach seems to grant great performance. A pity that we actually need a local server running on the device for that. |
Hi All! I read this thread, and some advanced concepts are being discussed here I see! (Sorry for being a lay man developer, please go easy on me) I am using this implementation, and I can download up to ~30 MB of data on my A53 Samsung Android device with it:
The app works and downloads the file when trying to download ~30 MB of data. The app crashes when trying to download ~60 MB of data. I was looking for a bullet proof solution capable to download files (.mp3) up to ~ +300 MB. I'm using this for the request: cordova-plugin-advanced-http I'm using this for the download: cordova-plugin-file (uses cordova.file). Could I just use: https://github.com/sgrebnov/cordova-plugin-background-download ? All help or recommendations would be appreciated. Extra information:
|
As mentioned in other posts, you should be using file-transfer-plugin to download and save the file without holding it in memory and causing the app to crash. I'm able to download and save 100Mb+ files suing this method. |
Another approach would be to download files with the fetch api. This is a native browser feature and most modern browsers support it. |
This is what I got from ChatGPT, I haven't tested it though... Here's how to do it step-by-step: ✅ Use Case: Efficiently Download a Large File via Streamingasync function downloadLargeFile(url, filename) {
const response = await fetch(url);
if (!response.ok) throw new Error(`HTTP error! Status: ${response.status}`);
const contentLength = response.headers.get('Content-Length');
const total = contentLength ? parseInt(contentLength, 10) : 0;
const reader = response.body.getReader();
const stream = new ReadableStream({
async start(controller) {
let received = 0;
while (true) {
const { done, value } = await reader.read();
if (done) break;
controller.enqueue(value);
received += value.length;
console.log(`Received ${received} of ${total} bytes`);
}
controller.close();
reader.releaseLock();
}
});
const blob = await new Response(stream).blob();
// Create a temporary download link and trigger download
const blobUrl = URL.createObjectURL(blob);
const a = document.createElement('a');
a.href = blobUrl;
a.download = filename;
document.body.appendChild(a);
a.click();
a.remove();
URL.revokeObjectURL(blobUrl);
} 🔍 Why this keeps memory usage low:
🧠 Bonus Tips
Would you like a version that saves directly to the user's disk using the File System Access API (Chrome/Edge support)? That allows real streaming to disk, ideal for very large files. |
I'll do more research and update it here. |
Keep in mind that in android, there is a rather conservative heap size. It does varies by device, where devices with more total memory will have a larger heap, but in my experience most heap sizes are between 128-256mb, or upwards to 512mb on very high end devices. What this means, that the android app can only allocate approximately 128mb in the JVM before memory pressure gets too high and risking the OOM error. If you can't keep your chunk sizes low and require allocating larger amounts of memory, android does expose a largeHeap flag which puts the JVM heap size closer to hardware limitations, though if you're actually allocating large amount of memory, it will destroy the performance of the application and be very inefficient with energy consumption. More objects to be tracked, longer GC sweeps, etc... Cordova has no option to enable this flag, but you can use the edit-config directive to apply it. You can also bypass the JVM limitations by passing the data chunks through to Android NDK, so that you actually do the processing in pure native land (using C), as memory allocated in NDK is outside of the JVM's heap and thus not included in that 128-256mb limit. If you're doing audio conversion, you probably would benefit greatly from android NDK since audio codecs typically isn't lightweight, and of course you get the raw hardware performance as well, but there is a cost to maintaining NDK code. Speaking from my own experience in with maintaining GIS-like systems. |
I found the solution. I still need to decipher what exactly is going on. These are my first impressions. Cordova is able to download big (+ ~300 MB) files from a server. To do this, what worked for me was editing the API calls in the method (tool/lib/package/...). I randomly came across this article dated 18th October 2017 that shows how to make API calls the right way, in order to retrieve a file IMO. https://cordova.apache.org/blog/2017/10/18/from-filetransfer-to-xhr2.html You will find an implementation example on the webpage. DM me on Github if you want the exact implementation I used to do so. I essentially was able to implement a version to download an MP3 file from a server in Blob format. I identified that the response had a valid blob format of the MP3 file by inspecting it through code and logs. After this add-on, the Cordova app was still crashing. Get this! I realized the crash was not from the API call, but from the save file process that saves "large" files onto the Android file system. The saving process kept making the Cordova Android app crash. So I am still not sure if it was my previous API call method or both it and the save process that make the Cordova Android app crash. (I had to make a bunch of detailed logs provided in real time from the Android OS level to identify that the save operation was making the app crash. Again, you can DM me to know the exact error if you want.) The "hack" to fix this is to essentially save the file in chunks. (DM for more code details) The diagnosis and solution are too long to explain here in detail, but the gist of the solution is here. I find it curious that Cordova doesn't have an easy-to-find, simple built-in lib/tool/api to save large files? I also find it curious that Cordova doesn't have easy-to-find documentation showing the best standard way to download and save a large file? I'll be posting a detailed solution for downloading and saving large files in Cordova Android apps on Stack Overflow. Anyhow, I thought I would share my solution for others that are struggling and wondering why their app is silently crashing when downloading a large file on an Android Cordova app. Thanks. |
You keep ignoring an existing solution to this problem with the file transfer plugin, I wonder why... |
My intuition told me it would not work tbh |
I have it in production for a few years now. Working great. |
Bug Report
Problem
What is expected to happen?
On large file save (> 5mb) app freeze for seconds.
Its pause render in my app, and animations start lag and jump.
What does actually happen?
No freeze or lag in render on file save.
Information
Command or Code
Environment, Platform, Device
Version information
Checklist
The text was updated successfully, but these errors were encountered: