[WIP] IPLD Composites (formerly IPLD Generics (formerly IPLD multi-block types))

[mikeal]

No description provided.

[rvagg]

really needs @warpfork to help us find a meeting point between schemas and types, or at least to agree that there is some place where they intersect and our various visions for these things is mutually understood and sane

[warpfork]

I'm not sure this is a particularly direct response to any of the diff content, but it's on the topic space as a whole, so as a top level comment:

Perhaps one way we could consider tersely describing the contract of these things is:

function (LinkLoader, Node) -> (Node)

Such a definition is:

• terse;
• can lean nicely on the properties and enumerations we already established in the Data Model docs;
• clearly points out this may be multi-block;
• gives us a lot of nice structure to communicate what this feature is and is not meant to encompass.

It might not be literally what the code is doing in our implementations, and as documentation it also still needs surrounding explanation, but structurally, does this seem about right?

[rvagg]

function (LinkLoader, Node) -> (Node) is conceptually accurate insofar as reads are concerned. Does that break down when you bring writes into it though? And therefore, is it even useful here as anything more than just a description for how traversal extended with these advanced types/generics?

[warpfork]

The write side is a bit different, but if you dream of it being based around a NodeBuilder, it's not so different: function (Storer) -> (NodeBuilder): where Storer is a function that serialize nodes, writes them to disk, and returns their links, and it's used to take care of all the internal nodes.

(In go-ipld-prime, it'll be even more similar: function (Storer, NodeBuilder) -> (NodeBuilder), because the first NodeBuilder is used to define the memory representations used for the internal node. Although I should tag this as a "probably"; haven't written it yet.)

(Aside: I wrote a whole first draft of this comment based on function (Storer, NodeBuilder) -> (Node) before realizing I was doing too much in one step... that approach came out much worse. Such a signature, in addition to doing too much in one step, would've been fairly terrible on the eyes for documentation/explanation purposes -- f(a, b) -> (b) is nice because it makes clear implications about how an abstraction is staying the same on both sides; f(a, b) -> (c) says roughly nothing.)

But as to whether or not this is good as a description: well, maybe. I think we'll get most of the descriptive benefits from putting function (LinkLoader, Node) -> (Node) pretty high in the documentation, right under it's own "H1" tag... and maybe if we do document function (Storer, NodeBuilder) -> (NodeBuilder), it would be much further on down the page, under an "H3" tag of additional clarifying information, or possibly tucked in some notes and suggestions for implementers of new IPLD libraries in new languages.

[mikeal]

Big changes were pushed:

• Removal of reserved property
• Addition of “definition” section
• Added “open problem” regarding fat pointers
• Remove the entire operations section and moved, temporarily, to the README in the experimental implementation. It’s not yet ready to be spec’d in a language agnostic way.

[mikeal]

I’m going to merge this in 24h if there are no objections.

[warpfork]

I would prefer to move this to a design history document, write up some other learnings afterwards, and later make a fresher, more limited, and clarified doc for spec drafts.

I like all the changes in the latest set, but there are several things in this file that make me uncomfortable, particularly with merging it as a spec-track document:

• There are several matters discussed in the doc that are interesting to our design process, but we haven't concretely proposed solutions to yet, nor necessarily are we even sure they're part of this topic. This is fine for design exploration notes, but not good for a spec document.
• We have not resolved that this new name is by any consensus superior to the old name. It may be, but we haven't polled for agreement on that; and, since the existing name exists in several other widely circulated documents already, we should be engaging this as a rename and take it with suitable aplomb.
• We have a lot of very similar concepts, approached with different granularity of definition, in issue Identifying the critical decisions in IPLD advanced data structure design #130. Could we write a better document by starting again and selecting a hybrid of the best parts from that issue and this document?
• ... and several other comments raised inline.

Proposal: let's move this file to the ./design-history/exploration-reports folder, date it, and call it lessons learned -- I like to see things land too, and I'd be more than happy to see it merge there, as it more accurately represents the state of play.

[rvagg]

If we're going with the design-history thing then I agree that this might be better there (I'm still not convinced about design-history but it's a thing for now). The main hint in here is the digression into _type which we seem to be agreeing is just a placeholder for some superior mechanism yet defined. So it's more of a just-make-it-work-for-now description than a spec.

[mikeal]

Proposal: let's move this file to the ./design-history/exploration-reports folder, date it, and call it lessons learned -- I like to see things land too, and I'd be more than happy to see it merge there, as it more accurately represents the state of play.

The entire purpose of having an “Exploratory” stage is to avoid having a bunch of exploratory documents in notes or “design history.”

The goal of this document is to write out and align effort going into Composites, as it will soon include at last 3 people on the team. Having people aligned on terminology and direction is the primary purpose of the document, as is the purpose of an “Exploratory” stage.

A document being in an exploratory stages does not solidify any of the approach, it’s all still subject to change or even to be discarded entirely. But, while people are actively trying to make forward progress, we need a way to write down and align that work, which is what the document is attempting to do. You may not agree with where all of the discussions ended up, and we may still return to them later, but in the interest of making forward progress we need to “disagree and commit” for the purposes of continuing to be able to do more exploratory development.

So it's more of a just-make-it-work-for-now description than a spec.

This is accurate, and should be what we assume the state of any Exploratory specification is.

We need to stop having PR’s open indefinitely because we lack 100% alignment on topics we have yet to even fully implement. We can continue to have discussions and make changes to Exploratory specifications, in fact, that’s one of the reason why we have a staging system at all.

[warpfork]

Regarding the describing this is "spec-track": even if labeled 'exploratory', it is taking up a slot in the file tree that makes it hard to propose anything else in parallel. I think whether or not something takes up a 'slot' in such a way is the crux of the matter in communication and development flow in practice, and that's why I describe this as 'spec-track'.

I think we may actually want to update the governance doc to suggest that exploratory docs simply go in a different directory (as we've started doing already), and we don't use the main directories until things are at least at the 'draft' stage.

[mikeal]

I think we may actually want to update the governance doc to suggest that exploratory docs simply go in a different directory (as we've started doing already), and we don't use the main directories until things are at least at the 'draft' stage.

+1000, I’ll work this into my next spec change.

[rvagg]

This was merged with the file named 'generics.md', the title "Composites" and an ongoing discussion about whether even that's the right name. That's strong proof to me that this belongs in design/ history/ or whatever that directory is going to be.

[mikeal]

Ya, that was an oversight, it’s fixed in the “rename PR” that moves it to ‘design/composites-A.md’

…

-Mikeal

________________________________ From: Rod Vagg <[email protected]> Sent: Tuesday, July 16, 2019 9:49 PM To: ipld/specs Cc: Mikeal Rogers; State change Subject: Re: [ipld/specs] [WIP] IPLD Composites (formerly IPLD Generics (formerly IPLD multi-block types)) (#126) This was merged with the file named 'generics.md', the title "Composites" and an ongoing discussion about whether even that's the right name. That's strong proof to me that this belongs in design/ history/ or whatever that directory is going to be. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub<#126?email_source=notifications&email_token=AAAAEQ54AJHMZFSX4DHHZS3P72QENA5CNFSM4HUEFIR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2DASSY#issuecomment-512100683>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAAAEQ6LFI2WX2NVEP7SPZTP72QENANCNFSM4HUEFIRQ>.