Buffer size used for staging write data

[bhawesh-khandelwal]

Hey

I am working with gcsfuse and while writing to a file, all the data is kept in the temp dir until a close is called. Even if I just want to append some data to a file, would it download the entire data, then append and then upload. The doc does mention that it would reupload only the appended data in case the appended data size is more than 2MB but does it download the entire object and then upload only the appended data or does it append the data to the existing object without downloading the object.

[raj-prince]

Apology for the late reply, here!

Currently, the entire object is downloaded to the staging location (controlled with --temp-dir flag) in both modify and append flows.

I totally understand your concern, that GCSFuse shouldn't download the complete data in the append flow.
Actually, we haven't invested much on the append flow as of now, but there are plans to do this.

[bhawesh-khandelwal]

I see, also the data is sent to gcs bucket only when the flush operation is issued by the client application. We should better keep writing the data to gcs bucket using resumable upload gcs writer at certain predefined buffer thresholds. Keeping everything in temp-dir is not a good approach from memory management perspective.

[vadlakondaswetha]

Agreed on the buffering part. We do have improvements planned in the write flow which should reduce the disk usage and start uploading to GCS before the flush call.

[bk-rubrik]

Is there any timeline for this @vadlakondaswetha ?

[bk-rubrik]

@vadlakondaswetha I see there's been some recent developments related to this. Can I request to know the timelines of the rollout ? Also, can you share any perf numbers around this?

[raj-prince]

Thanks for the interest, @bhawesh-khandelwal!

Feature with the perf numbers will be available in February.