Crashed to upload large files ( > 500MB): [Terminating app due to uncaught exception 'NSMallocException', reason: 'Failed to grow buffer']

[nikolaytsigvintsev]

Hi.

I'm trying to upload a large file. My application crashes after launching the upload with the error:

• Terminating app due to uncaught exception 'NSMallocException', reason: 'Failed to grow buffer'

On a small file size works well.

iPad Pro
Deployment Target : 12.1
React-Native v0.58

Code:

      const upload = new tus.Upload({ uri: 
        `${RNFS.DocumentDirectoryPath}/images/${item.image}`}, {
        endpoint: 'http://192.168.1.5:1080/files/',
        retryDelays: [0, 1000, 3000, 5000, 10000, 15000, 20000],
        resume: true,
        metadata: {
          name: item.image,
          filetype: item.image.substring(item.image.lastIndexOf('.') + 1).toUpperCase(),
          copyright: 30
        },
        onError: (error) => {
          console.log('Error upload:', error)
        },
        onProgress: (uploadedBytes, totalBytes) => {
          console.log(`Progress: ${uploadedBytes / totalBytes}%`)
        },
        onSuccess: () => {
          console.log('Upload URL:', upload.url)
        }
      })
      upload.start()

[Acconut]

That's interesting, thanks for reporting. It seems to me as if tus-js-client is trying to read the entire file in memory before uploading it. This works fine if enough memory is available to cover a smaller file but obviously fails for bigger uploads.

I am not sure how to fix this inside tus-js-client out of my head now but maybe @arturi knows if React Native has a better way for streaming uploads.

@nikolaytsigvintsev: As a workaround you might want to add the chunkSize: 100 * 1024 * 1024 option when you create the tus.Upload instance. This will cause the upload to be split into multiple PATCH requests where at most 100MiB are uploaded per request. Therefore, not more than 100MiB should be necessary. Of course, this is only a workaround for now and we need to find a better solution.

[nikolaytsigvintsev]

Hi.

Thanks for the answer. I tried to install chunkSize, but it did not help.
Exactly the same mistake.

  const upload = new tus.Upload({ uri: `${RNFS.DocumentDirectoryPath}/images/${item.image}`}, {
        endpoint: 'http://192.168.1.5:1080/files/',
        retryDelays: [0, 1000, 3000, 5000, 10000, 15000, 20000],
        chunkSize: 100 * 1024 * 1024,
        resume: true,
        metadata: {
          name: item.image,
          filetype: item.image.substring(item.image.lastIndexOf('.') + 1).toUpperCase(),
          copyright: 30
        },
        onError: (error) => {
          console.log('Error upload:', error)
        },
        onProgress: (uploadedBytes, totalBytes) => {
          console.log(`Progress: ${uploadedBytes / totalBytes}`)
        },
        onSuccess: () => {
          console.log('Upload URL:', upload.url)
      })
      upload.start()

[Acconut]

Oh, I forgot. tus-js-client apparently tries to load the file into memory before it slices it using the chunkSize option. That explains why setting this parameter does not change anything for your situation. Frankly, I am not sure how to resolve this. I am not experienced enough with React Native to know if there is a way to read a fixed amount of bytes from a file at a specific offset. Not sure if @arturi had time to look into this yet.

In the meantime you might want to look into using https://github.com/vinzscam/react-native-tus-client which is a tus-js-client-like wrapper around the native Android and iOS tus clients. I have never tried them by myself but they should be able to handle bigger files.

[nikolaytsigvintsev]

Hi.

Thank you for taking the time to respond.

The tus-js-client documentation says that you can start the upload with the Readable stream option. I think that one of the ways out of this problem is to create a readable stream and send it to the upload function. Only I cannot create a readable stream (react-native) that I can send in the Upload function.
I created a question on Stakoverflow.

Thanks for help.

[Acconut]

The tus-js-client documentation says that you can start the upload with the Readable stream option.

We only implemented uploading from readable stream for the browsers and Node.js, so I doubt that this would work without modifications for React Native. We are happy to extend the support for streaming uploads to React Native if you would like to give it a try.

I created a question on Stakoverflow.

That's great, would you mind sharing a link with us?

[nikolaytsigvintsev]

Hi.

Thanks for the answer.

I do not know how I can help you. I am new to the react-native. I can try to make a function to upload file by stream, but tell me where to start.

link: https://stackoverflow.com/questions/55496609/tus-js-client-for-react-native-crashed-to-upload-large-files

[kvz]

Hi @nikolaytsigvintsev, we're working on this. We'll have a (very basic) example/version online over at the Uppy site, hopefully today.

[nikolaytsigvintsev]

Hi @kvz.

I am very glad that your work goes forward.
I noticed that @arturi added an example response-native-expo to the Uppy site (https://github.com/transloadit/uppy/tree/master/examples/react-native-expo). I took the code from the example and compiled it for 'react-native'. I think this is what I need.

Thanks and good luck.

[kvz]

Hi @nikolaytsigvintsev, I noticed you said it wasn't working for you in a different thread. it seems we had a typo in there earlier 😱 sad that we're not using TypoScript (pun intended :) ) for examples yet maybe 😹 @ifedapoolarewaju discovered and fixed it here: transloadit/uppy@854654b

We've tested this and confirmed it works on an Android simulator.

Sorry for the long and dangerous road we're putting you on! Would you mind seeing if this fixes it for you too?

Thanks for bearing with us!

[nikolaytsigvintsev]

Hi @kvz.

Thanks for the answer.
The program works on the simulator, because there may be more memory on the simulator that real device ?
On real device iPad Pro all works good until size file < 500 - 700MB
I tried upload file 542MB result good, on upload file 2.4G result failed.

Videoreport:
https://www.dropbox.com/s/1z8yy5v85mmy72m/ScreenRecording_05-08-2019%2013-31-57.MP4?dl=0

I'm trying to understand how the tusFileReader function works.
I inserted console.log() statements into the getTusFileReader function and into the slice() function.
The function slice() is not called.
As I understand it, the slice() function works for streaming uploading.

Thanks.

[ifedapoolarewaju]

@nikolaytsigvintsev there might be something off with your code. How are you instantiating Tus? Are you passing the tusFileReader into your TusOptions?

[nikolaytsigvintsev]

Hi @ifedapoolarewaju

Yes.

const upload = new tus.Upload({uri: ${RNFS.DocumentDirectoryPath}/images/${node.asset.url}}, {
endpoint: 'http://192.168.1.5:1080/files/',
retryDelays: [0, 1000, 3000, 5000],
resume: true,
fileReader: getTusFileReader,
metadata: {
name: node.asset.url,
filetype: node.image.substring(node.image.lastIndexOf('.') + 1).toUpperCase(),
copyright: 30
},
onError: (error) => {
console.log('Error upload:', error)
this.setState({isUploading: false})
},
onSuccess: () => {
console.log('Upload Success:')
this.setState({isUploading: false})
}
})
upload.start()

The getTusFileReader function is called, but slice () in class TusFileReader does not call

[ifedapoolarewaju]

what is the content of getTusFileReader ? Also what version of tus-js-client are you using this with?