encoding/json: Encoder internally buffers full output

[extemporalgenome]

What does 'go version' print?

go version devel +2f6b9f80be36 Fri Apr 25 09:46:07 2014 -0600 linux/amd64

What steps reproduce the problem?
If possible, include a link to a program on play.golang.org.

1. Use json.NewEncoder(writer).Encode(value)

What happened?

Each call to json.Encoder.Encode uses an internal bytes.Buffer to buffer all encoded
output prior to writing any of that output.

What should have happened instead?

Output should use little or no internal buffering. Encoder should be able to efficiently
encode (and incrementally output) very large inputs.

[bradfitz]

Comment 1:

Yeah, suppose it could use a bufio.Writer instead of a bytes.Buffer.

Labels changed: added release-go1.4, repo-main.

Status changed to Accepted.

[gopherbot]

Comment 2 by nicolashillegeer:

> Yeah, suppose it could use a bufio.Writer instead of a bytes.Buffer.
This could indeed save a useless copy, as also noted in this issue:
https://golang.org/issue/5683
However, there are some small things. For example, I'm using an Encoder to encode
straight into a go.net websocket. Due to the current implementation, the Encoder issues
a Write once the entire message is completed. With a bufio.Writer, there could be
several writes. The go.net WebSocket implementation takes every write as an entire
message (as far as I can see from perusing the source). This would make my json possibly
arrive in arbitrary pieces.
Note that what I'm doing is not necessarily the appropriate usage of go.net WebSockets,
it has websocket.JSON.Receive/Send methods to send and receive entire messages at a
time. And I should be using that. But those methods use json.Marshal/json.Unmarshal
under the hood, which cause buffer churn when used a lot (my use case is relatively
high-throughput). Ideally the go.net WebSockets package would use Encoder/Decoder with a
bytes.Buffer under the hood for the JSON.(Send|Receive) calls, reusing the buffer every
time (either using a sync.Pool for the buffers or storing them in the websocket
connection itself, since I believe those don't support concurrent writing anyway).
I would not mourn too much if my current usage of go.net WebSockets stops working.

[rsc]

Comment 3:

I don't believe we can change this today. The buffering ensures that if an error is
encountered, no partial JSON is emitted. The only thing we could do is add some method
to encoder to allow people to turn it off themselves.

Labels changed: added release-go1.5, removed release-go1.4.

[emidoots]

I was rather shocked when I discovered this, actually. If my understanding is correct this means that JSON cannot be streamed to a file without it consuming large amounts of memory.

The only thing we could do is add some method to encoder to allow people to turn it off themselves.

I highly agree with this approach.

[bradfitz]

@slimsag, I don't understand. Each call to encoding/json.Encoder.Encode produces one Write. Even if Encode wrote more eagerly to its Writer, you still have the entire argument to Encode (the whatever interface{}) in memory. So it's at worst a linear factor of memory. This isn't the difference of being able to stream or not.

Realistically if you want to stream JSON, you'd do something like:

     e := json.NewEncoder(f)
     io.WriteString(f, "[")
     for .... {
           dataToStream := ....
           e.Encode(dataToStream)
           io.WriteString(f, ",")
     }
     io.WriteString(f, "]")

... and that pattern works fine even today.

Changing the internal buffering (this bug) has nothing to do with making it possible to programmatically generate the data to stream. That would a much more invasive API change, and not obviously better than the loop above.

[ydnar]

This should satisfy the existing requirements (truncation on error), and limited buffering to enable streaming structures of unbounded/unknown size: https://github.com/ydnar/go/compare/jsonio

[emidoots]

@bradfitz I can see now that I misunderstood what this bug is about, my apologies!

From the title "Encoder internally buffers full output" I had incorrectly thought all calls to Encode were buffered. I have re-read the issue understand now that only the data written by a single call to Encode is buffered, and so streaming works fine.