proposal: NeoFS block storage format #194

AliceInHunterland · 2025-04-08T03:48:16Z

No description provided.

neofs-blocks.mediawiki

roman-khimov

Index files? Synchronization algorithm?

roman-khimov · 2025-04-08T16:32:24Z

neofs-blocks.mediawiki

+Its structure is described further below.
+
+<pre>
+block_binary


I think you can just refer to "standard Neo block serialization", that's what it is in its essence.

roman-khimov · 2025-04-08T16:33:13Z

neofs-blocks.mediawiki

+
+==Abstract==
+
+This proposal outlines the methodology and specifications for storing Neo blockchain blocks within


methodology is too strong word for this.

roman-khimov · 2025-04-08T16:33:34Z

neofs-blocks.mediawiki

+
+==Motivation==
+
+Neo node synchronization via P2P requires all block headers (~700 bytes each, over 1 GB per year with


headers and blocks

roman-khimov · 2025-04-08T16:34:41Z

neofs-blocks.mediawiki

+Neo node synchronization via P2P requires all block headers (~700 bytes each, over 1 GB per year with
+15-second blocks) to remain available, limiting scalability as chains grow. Storing blocks in NeoFS
+provides a distributed alternative, reducing local storage, enabling on-demand fetches, and speeding
+up sync with snapshots or checkpoints. This proposal defines a NeoFS storage schema, building on


snapshots and checkpoints are for a different proposal.

roman-khimov · 2025-04-08T16:34:58Z

neofs-blocks.mediawiki

+15-second blocks) to remain available, limiting scalability as chains grow. Storing blocks in NeoFS
+provides a distributed alternative, reducing local storage, enabling on-demand fetches, and speeding
+up sync with snapshots or checkpoints. This proposal defines a NeoFS storage schema, building on
+experiments (e.g., nspcc-dev/neo-go#3496) and future enhancements (#1526).


Experiments are for a different proposal section (rationale).

roman-khimov · 2025-04-08T16:37:00Z

neofs-blocks.mediawiki

+has network magic attribute (`Magic:56753`). Each block is stored in a binary
+form as a separate object with a unique OID and a set of attributes:
+ - block object identifier with block index value (`Block:1`)
+ - primary node index (`Primary:0`)


We have it now, but I'm still thinking if its useful. I can imagine searches for block hashes/timestamps, but primary index is not so obvious (especially without a verification script hash).

roman-khimov · 2025-04-08T16:38:05Z

neofs-blocks.mediawiki

+A single NeoFS container is used to store blocks. Each container
+has network magic attribute (`Magic:56753`). Each block is stored in a binary
+form as a separate object with a unique OID and a set of attributes:
+ - block object identifier with block index value (`Block:1`)


Define field names first, then describe them. Examples are for a different section.

roman-khimov · 2025-04-08T16:39:03Z

neofs-blocks.mediawiki

+form as a separate object with a unique OID and a set of attributes:
+ - block object identifier with block index value (`Block:1`)
+ - primary node index (`Primary:0`)
+ - block hash in the LE form (`Hash:5412a781caf278c0736556c0e544c7cfdbb6e3c62ae221ef53646be89364566b`)


LE/BE are a bit different in C#, we can say "in hex hash representation"

roman-khimov · 2025-04-08T16:39:57Z

neofs-blocks.mediawiki

+
+These attributes are stored as NeoFS object metadata alongside the binary block data.
+Multibyte data items are always stored in little-endian order, where the low bytes come first.
+Its structure is described further below.


Add a not that the object MAY have additional data after the block, it can be useful for future extensions.

roman-khimov · 2025-04-08T16:40:38Z

neofs-blocks.mediawiki

+  var-byte transactions[];     // Array of serialized transactions
+}
+</pre>
+


We need an example of object.

Jim8y · 2025-04-11T02:30:09Z

@AliceInHunterland please update accordingly, this is not likely can be reviewed with any useful suggestions from other coredevs since we know little about neofs, please just follow roman's suggestion

AnnaShaleva · 2025-06-24T09:01:36Z

neofs-blocks.mediawiki

+
+===Block Storage Schema===
+A single NeoFS container is used to store blocks and index files per network. Each container
+has network magic attribute (`Magic:56753`). Each block is stored in a binary


Replace direct example of Magic:56753 with description of attribute name and the form of attribute value, because this section is a specification description, examples shouldn't be placed here.

AnnaShaleva · 2025-06-24T09:05:09Z

neofs-blocks.mediawiki

+===Block Storage Schema===
+A single NeoFS container is used to store blocks and index files per network. Each container
+has network magic attribute (`Magic:56753`). Each block is stored in a binary
+form as a separate object with a unique OID and a set of attributes:


in a binary form

in a binary form following standard Neo block serialization rules

AnnaShaleva · 2025-06-24T09:14:00Z

neofs-blocks.mediawiki

+form as a separate object with a unique OID and a set of attributes:
+* Block: block object identifier with block index value.
+* Hash: block hash in the in hex hash representation.
+* PrevHash: previous block hash in the in hex hash representation.
+* BlockTime: millisecond-precision block creation timestamp.
+* Timestamp: second-precision block uploading timestamp.


Ref. #194 (comment), from this description it's not quite clear that Block, Hash, etc. are attribute names and they are strictly fixed. I'd say that we either need additional note about it or may be even a table, something like that:

Attribute name Attribute value

Block String representation of block index in the base of 10.

...

The same stile can be applied to container attributes.

The same stile can be applied to container attributes.

We have, for now, only one. Would you like a separate table for it?

It's nice to have things unified, so table also suits even for one attribute. If we won't use table, then we need at least to adjust description wrt #194 (comment).

AnnaShaleva · 2025-06-24T09:17:17Z

neofs-blocks.mediawiki

+An [https://rest.fs.neo.org/v1/objects/3RCdP3ZubyKyo8qFeo7EJPryidTZaGCMdUjqFJaaEKBV/by_id/Fu7yQzspvLJwSGJNK64xeeyMdWXtU5B5b1es6KSxUag1 example]
+of a block object is as follows:
+<pre>
+ID: Fu7yQzspvLJwSGJNK64xeeyMdWXtU5B5b1es6KSxUag1
+CID: 3RCdP3ZubyKyo8qFeo7EJPryidTZaGCMdUjqFJaaEKBV
+Owner: NVvY1FF67XJ2GTVhy9FqiZGC4jEQtvjmHt


Is it correct to attach object link as an example? This object/container may be removed in future whereas we expect NEP to be consistent. @roman-khimov?

Example can be a proper example with this data if needed. It doesn't imply any of the IDs are relevant for mainnet or whatever else.

AnnaShaleva · 2025-06-24T09:23:28Z

neofs-blocks.mediawiki

+* BlockTime: millisecond-precision block creation timestamp.
+* Timestamp: second-precision block uploading timestamp.
+
+These attributes are stored as NeoFS object metadata alongside the binary block data and can be extended.


Suggested change

These attributes are stored as NeoFS object metadata alongside the binary block data and can be extended.

These attributes are stored as NeoFS object metadata; the list of attributes MAY be extended.

AnnaShaleva · 2025-06-24T09:40:01Z

neofs-blocks.mediawiki

+==Rationale==
+This proposal defines a NeoFS storage schema, building on experiments (e.g., nspcc-dev/neo-go#3496)
+and future enhancements (#1526).


It's not the proper section for the rationale. Also, the text itself may be improved, see as an example https://github.com/neo-project/proposals/blob/master/nep-25.mediawiki#user-content-Rationale.

AnnaShaleva · 2025-06-24T09:41:45Z

neofs-blocks.mediawiki

+
+
+== Implementation ==
+* Go: https://github.com/nspcc-dev/neo-go/blob/61992405983ddfb5547ae3afab75c22186dc966f/cli/util/uploader.go


Do we need to attach an example of NeoFSBlockFetcher as far? I think it will be useful since synchronisation algorythm is also a part of this proposal. @roman-khimov?

Can be referenced.

AnnaShaleva · 2025-06-24T09:42:46Z

neofs-blocks.mediawiki

+* [https://github.com/neo-project/neo/issues/2373 Neo Issue#2373]
+* [https://github.com/neo-project/neo/issues/1526 Neo Issue#1526]
+* [https://github.com/nspcc-dev/neo-go/issues/3496 NeoGo Issue#3496]
+* [https://github.com/nspcc-dev/neofs-api/issues/285 NeoFS API Issue#285]


I don't think that this issue is directly related to the proposal. @roman-khimov?

Yeah, a different story. Even quite old one.

AnnaShaleva · 2025-06-24T09:45:41Z

neofs-blocks.mediawiki

+* [https://github.com/neo-project/neo/issues/3463 Neo Issue#3463]
+* [https://github.com/neo-project/neo/issues/2373 Neo Issue#2373]
+* [https://github.com/neo-project/neo/issues/1526 Neo Issue#1526]
+* [https://github.com/nspcc-dev/neo-go/issues/3496 NeoGo Issue#3496]


These issues should have readable description instead of just Issue#.... See https://github.com/neo-project/proposals/blob/master/nep-23.mediawiki#references as an example.

AnnaShaleva · 2025-06-24T09:47:17Z

neofs-blocks.mediawiki

+  NEP: TBD
+  Title: NeoFS block storage format
+  Author: Ekaterina Pavlova <[email protected]>
+  Type: Informational


To me the type of this standard looks more like Standard rather than Informational because we directly standardize NeoFS storage schema and building node synchronization mechanism over it. Or may be even Meta standard, since it's not directly related to the Neo protocol, ref https://github.com/neo-project/proposals/blob/master/nep-1.mediawiki#nep-types. @roman-khimov?

To me it's Informational since there is no obligation for N3 protocol to store blocks in NeoFS and be able to synchronize from it. It's for interoperability if anyone wants to fetch blocks this way, but it's far from P2P getblocks for example.

Signed-off-by: Ekaterina Pavlova <[email protected]>

shargon reviewed Apr 8, 2025

View reviewed changes

neofs-blocks.mediawiki Outdated Show resolved Hide resolved

roman-khimov requested changes Apr 8, 2025

View reviewed changes

AliceInHunterland force-pushed the neofs-blocks branch from 1c6106c to 674a1c2 Compare April 17, 2025 14:36

AliceInHunterland requested a review from roman-khimov April 17, 2025 14:37

AliceInHunterland force-pushed the neofs-blocks branch 2 times, most recently from 1231891 to 6c12fbe Compare April 18, 2025 05:11

AnnaShaleva reviewed Jun 24, 2025

View reviewed changes

proposal: NeoFS block storage format

bf44b35

Signed-off-by: Ekaterina Pavlova <[email protected]>

AliceInHunterland force-pushed the neofs-blocks branch from 6c12fbe to bf44b35 Compare June 25, 2025 09:27

AliceInHunterland requested a review from AnnaShaleva June 25, 2025 09:30

Jim8y mentioned this pull request Jun 26, 2025

June 26 Core Dev Regular Meeting Proposals neo-project/neo#4020

Closed


		==Abstract==

		This proposal outlines the methodology and specifications for storing Neo blockchain blocks within


		==Motivation==

		Neo node synchronization via P2P requires all block headers (~700 bytes each, over 1 GB per year with

Attribute name	Attribute value
Block	String representation of block index in the base of 10.
...

	These attributes are stored as NeoFS object metadata alongside the binary block data and can be extended.
	These attributes are stored as NeoFS object metadata; the list of attributes MAY be extended.



		== Implementation ==
		* Go: https://github.com/nspcc-dev/neo-go/blob/61992405983ddfb5547ae3afab75c22186dc966f/cli/util/uploader.go

proposal: NeoFS block storage format #194

Are you sure you want to change the base?

proposal: NeoFS block storage format #194

Uh oh!

Conversation

AliceInHunterland commented Apr 8, 2025

Uh oh!

Uh oh!

roman-khimov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jim8y commented Apr 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!