Skip to content

Compress Wasm bytecodes when publishing #2382

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Aug 19, 2024

Conversation

jvff
Copy link
Contributor

@jvff jvff commented Aug 17, 2024

Motivation

Sometimes publishing application fails because the bytecodes are too large, and hit gRPC message size limits when sending the block proposal.

Proposal

Compress the bytecodes using Zstandard when including them in the SystemOperation::PublishBytecode. Use a new CompressedBytecode type to represent the compressed bytecode, and make it simple to convert to and from an uncompressed Bytecode.

Test Plan

CI should catch any regressions, since end-to-end tests cover publishing bytecodes and using them in applications.

Release Plan

A new major version must be released, because this breaks backward compatibility with bytecode publishing.

Links

@jvff jvff added the enhancement New feature or request label Aug 17, 2024
@jvff jvff added this to the Testnet #1 milestone Aug 17, 2024
@jvff jvff requested review from ma2bd, ndr-ds and afck August 17, 2024 02:15
@jvff jvff self-assigned this Aug 17, 2024
@jvff jvff force-pushed the wasm-bytecode-compression branch from 44ae7d3 to dc15211 Compare August 17, 2024 02:19
Instead of specifying the full path.
@jvff jvff force-pushed the wasm-bytecode-compression branch from dc15211 to d6ae9dd Compare August 17, 2024 15:37
jvff added 4 commits August 17, 2024 18:25
The type was not a tuple.
Use Zstandard to compress Wasm bytecodes.
Only accept bytecodes published if they are compressed.
Use `CompressedBytecode` instead of `Bytecode`.
@jvff jvff force-pushed the wasm-bytecode-compression branch from d6ae9dd to cd9ffdb Compare August 17, 2024 18:26
Copy link
Contributor

@christos-h christos-h left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice! One thing I would say is that not all OS's ship with zstd. Since the crate used is a binding to zstd, perhaps we should have a fallback if it's not installed (gzip and then no compression at all).

Nevermind! Looks like we're already linking against it.

@Twey
Copy link
Contributor

Twey commented Aug 19, 2024

These are our current dynamically-linked libraries:

	librocksdb.so.9
	libgcc_s.so.1
	libm.so.6
	libc.so.6
	ld-linux-x86-64.so.2
	libsnappy.so.1
	libz.so.1
	libbz2.so.1
	liblz4.so.1
	libzstd.so.1
	liburing.so.2
	libstdc++.so.6
	libpthread.so.0

I'm not sure which of these are included with the standard Rust install, which would be okay since we currently require source builds, and which we should list as dependencies in the manual.

@jvff jvff merged commit a4d6fb4 into linera-io:main Aug 19, 2024
7 checks passed
@jvff jvff deleted the wasm-bytecode-compression branch August 19, 2024 14:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

4 participants