Skip to content

download of compressed files on arm-plattforms #110

Closed
@simonhbw

Description

@simonhbw

hi, ive a problem using tools like @azure/storage-blob or node-fetch on armv7 devices with any node-version, which uses llhttp instead of the legacy parser. when downloading a compressed file, the response is always decompressed what cause an data corruption failure. i can reproduce the issue using whatever @azure/storage-blob or node-fetch directly.

the interessting thing is, that the issue only ocurres on arm(v7)-plattforms. running the same code on an x64/86 plattform, doesnt cause any issues.
changin the node-version to 11.x or 12.x, with "http-parser=legacy" also doesnt cause any issues

code sample:

const { BlockBlobClient } = require('@azure/storage-blob');
const path = require('path')
const fs = require('fs')
const cliProgress = require('cli-progress');

const repositoryPath = '../repository'
const uri = 'placeUrlWithSasHere'

async function startDownload(sasUri) {
try {
    console.log('starting download from url: %s', sasUri);
    const blobClient = new BlockBlobClient(sasUri);
    const props = await blobClient.getProperties();
    // console.log({ props });
    
    const fileName = blobClient._name;
    const filePath = path.join(repositoryPath, fileName);
    const bar = new cliProgress.SingleBar({}, cliProgress.Presets.shades_classic);

    const downloadBlockBlobResponse = await blobClient.download(null, null,
        {
            onProgress: ((ev) => {
                bar.update(ev.loadedBytes);
            }),
        });
    
    const readStream = downloadBlockBlobResponse.readableStreamBody;
    const writeStream = fs.createWriteStream(filePath);
    
    bar.start(props.contentLength, 0);
    readStream
        // .pipe(throttler)
        .pipe(writeStream);
    
    readStream.on('error', (e) => { 
        bar.stop()
        console.error(e.message)
    });
    writeStream.on('error', (e) => { 
        bar.stop()
        console.error(e.message)
    });

    readStream.on('end', () => {
        bar.stop()
        console.log('readstream finished')
    })
} catch (error) {
   console.error(error)
}
}

startDownload(uri)

result on armv7:

Data corruption failure: received less data than required and reached maxRetires limitation. Received data offset: 1473222655, data needed offset: 10063157247, retries: 5, max retries: 5

result on x64

readstream finished

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions