The road to the native runtime free world

[bkchr]

The Vision

We want to get rid off the native runtime. Initially the native runtime was added to overcome the performance "problems" of using wasmi for the runtime execution. However, with wasmtime we are around 2x slower than native. While this still isn't perfect, it is way faster than wasmi that is around 10x slower (all depending on the work load etc). While the performance plays an important role here there are also other things to consider, like ensuring that the native and wasm runtime are deterministic. We have already run into problems with Polkadot when it comes to this determinism problems. While we can ensure to use the exact same compiler version for building the Polkadot binary in our release process, we can not ensure that everyone else is also using the same compiler version as we used for the wasm build.

There is also a nice write up and some discussions here: paritytech/substrate#7288

The Plan

To remove the native runtime, we need to fix some things before:

1. Make the wasm memory growable. We have already run into a situation of a OOM. This happened because we only allow a fixed amount of memory per wasm instance. We solved this problem temporarily by running the native runtime which can just use the entire system memory and then afterwards we released a new Polkadot node with an increased memory budget. So, we were lucky because we had the native runtime that saved our ass, but if we remove it it would not be there anymore. So, we should support growing of the wasm memory. It can still in the end allocate in maximum 4GB, but that should be way enough for all our use cases and combined with some proper monitoring we should be able to be informed before we hit this upper limit.
2. Remove runtime api requirement of statically checking that the runtime api is implemented. Currently when trying to use a runtime api, it is required to tell the compiler that a certain type is required that needs to implement a runtime api. This was done on purpose to give people better hints at compile time that they don't have implemented a certain api, but this only works by checking the native runtime (because we can not tell the compiler to check the wasm file). So, if we want to remove the native runtime we also need to remove this requirement on having the runtime api being implemented for a type. This can be solved by implementing the runtime apis for a different trait that will then be able to call into the runtime. We will also need to change some stuff, like checking that a runtime api is implemented on startup instead of failing later when the node is running. Remove sp-api requirement on compile time bounds #27
3. Genesis is currently being created using the native runtime. We will also need to fix this. Maybe with an extra tool or whatever. GenesisConfig in a native runtime free world #25

Open Questions

If you want to help us out and contribute to this issue, in this section you can find open questions and tasks where we would appreciate any input.

There are maybe still some other things to consider and to fix on the way, like only compiling the runtime for wasm instead of the current native & wasm build.

---

Here you can find the board with specific sub-tasks to this milestone:
SDK Node (view)

[burdges]

As an aside not strictly relevant here, we do envision having "runtime libraries" eventually, like SPREEs or perhaps large verifier keys ala halo2, which might be implemented in another way like a flat file of data, or if they undergo closer auditing then eBPF or native code loaded via dlopen, or a native generator for a verifier key. Afaik all node should use identical deployment scheme, so we'd never walk back the consensus improvement described here.

[bkchr]

Okay, sounds interesting. I always thought that SPREE would be different wasm files that are just linked at runtime?

Do we really want to native load code? Doesn't this destroy the entire advantage of using wasm? I mean then we could have done that from the beginning and could have distributed the runtime as a native library.

[burdges]

I've too little grasp about what "wasm files linked at runtime" means, but the SPREE would require process-like separation from the main PVF, and the SPREE invokes hostcalls that modify the SPREE state, for which the main PVF lacks permission.

At least in my mind that's impossible with "linked at runtime" so instead the SPREE exposes some crate linked at build time, but then itself creates new hostcalls or more SPREE-calls available to the whole PVF including its own part.

Yes, SPREEs could be WASM but maybe 2x matters enough that a common enough SPREE would be native and well audited, or maybe doing those native builds limits our hardware choices too much? I donno..

Do we really want to native load code?

I donno, it's plausible we'd never want native code loaded, but much here remains unclear to me, like the performance hit eBPF takes.

I only raised the comment in case anything of this impacts how you remove the native runtime. :)

[bkchr]

I've too little grasp about what "wasm files linked at runtime" means, but the SPREE would require process-like separation from the main PVF, and the SPREE invokes hostcalls that modify the SPREE state, for which the main PVF lacks permission.

Yeah that makes sense and I did not thought about it, but this could just be a host call to communicate with SPREE and the SPREE could be anything.

Aka we could have SPREE by default in wasm and could provide native compiled SPREE that ships with the node. If the requested SPREE version would not be supported by the node, we could just run the wasm version. However, I think whatever we do it is orthogonal to removing the native runtime.

I only raised the comment in case anything of this impacts how you remove the native runtime. :)

That is always a good idea nevertheless :)

[Wizdave97]

@bkchr Would host functions be removed entirely along side the native runtime?
Or would there be a workaround for that?

[bkchr]

Host functions can not be removed and will not be removed.

[Garandor]

A question on this popped up at Manta-Network/Manta#701

The gist of it is:
There already exists a commandline flag to force WASM execution: --execution wasm

While stopping validators on a nondeterminism issue is certainly bad and should be avoided ( e.g. using that same CLI flag ), it might be tolerable for other full node network participants - as a downtime on their functions ( serving RPC in our case ) is not nearly as critical to the network as a chain stall.

We might want to actually optimize non-critical nodes for the happy case where everything works and have the 2x speedup in runtime execution result in a higher number of RPCs served per node ( got no numbers on the actual performance impact yet, using the 2x number from @bkchr's initial description here, actual gain I guess will also depend on the weight used per block on average )

native runtime free as on the tin would remove this optionality of tolerating stability issues for performance gain, so IMHO it's not desirable, since you can already force the WASM runtime with the above CLI flag.

[bkchr]

Did you benchmark the execution? And do you have that many calls into the runtime on your RPC nodes?