big_http_download stuck during download on qemu_x86

[rlubos]

After receiving first packet with the downloaded package application get's stuck and keeps idling. I'm using current Zephyr master with default configuration on qemu_x86. I attach screenshot with logs from Wireshark, there is some mess due to TCP window size, but I guess it is correct in case of small window? Not sure if am I doing something wrong or if there is an problem with the application. Any ideas what could be wrong?

[pfalcon]

or if there is an problem with the application.

I'd like to start with a methodological point. The question above, taken literally, is very easy to answer: all socket samples shipped with Zephyr to date can be built on a POSIX system (besides Zephyr). Just do:

make -f Makefile.posix

on a Linux system and you'll see that there's nothing wrong "with the application", it just works. Thus, we can strike that thru and proceed with posing next questions, specifically:

2. Is there a problem with a small, thin shim which implements BSD Sockets API on top of native Zephyr API?
3. Is there a problem with Zephyr IP stack?

This very simple, just as very important, principle was downplayed and rejected when discussing #7118. Users of that "API" won't have a freedom to triage application vs stack issues so easily. (Without introducing head-over-heels tricks, which I'm sure will be introduced.)

Anyway, not our case here, and we can concentrate on qq. 2 and 3 here.

[pfalcon]

For reference, my Wireshark window:

[rlubos]

Thank you for the feedback. In no means I wanted to blame the "sample" part of the application, by application I meant entire solution, perhaps I should revisit my vocabulary.

Anyway, I did some investigation and it appears that the root cause for the issue described is handling of Zero Window Probe, that can be seen in the Wireshark log provided. Apparently TCP implementation on the server I was connecting to, as well as Wireshark's one expect, that after zero window length was notified by the peer, it will drop any data received. Therefore the server ignored the ACK sent by Zephyr app, which indicated that byte was consumed and stopped transfering data.

Whether this is correct behaviour or not is something to discuss, I saw there are different opinions in the web (for example here). Anyway, as a proof of concept I have a fix that was verified to make communication work again (https://github.com/rlubos/zephyr/tree/zero-window-probe-handling). It's up to discussion whether we want to handle this corner case in Zephyr.

Other solution helping with this issue was doubling TCP window size, so we're simply not hitting this Zero Window Probe problem. It's more like a workaround, but it helped as well. It would require some changes as well, as currently TCP window size is not something configurable (which I think should be though, as it's tighly related to nuber of buffers we have, and that is configurable).

[pfalcon]

the root cause for the issue described is handling of Zero Window Probe, that can be seen in the Wireshark log provided

Right, so the next step in my queue is submitting a ticket titled "net: Zero Windows Probes are not handled properly", and discussing the current behavior and how it should work actually. Feel free to beat me on submitting such a ticket ;-).

[pfalcon]

I'm trying to reproduce the peer sending me a Zero Window Probe, and I can't. I never saw it before, and I try to trick into doing that with following patch:

--- a/samples/net/sockets/big_http_download/src/big_http_download.c
+++ b/samples/net/sockets/big_http_download/src/big_http_download.c
@@ -221,7 +221,7 @@ int main(void)
        hints.ai_socktype = SOCK_STREAM;
 
        while (resolve_attempts--) {
-               st = getaddrinfo(host, port, &hints, &res);
+               st = getaddrinfo("91.189.88.152", port, &hints, &res);
 
                if (st == 0) {
                        break;
diff --git a/subsys/net/lib/sockets/sockets.c b/subsys/net/lib/sockets/sockets.c
index 5bcd771f5..5f672245e 100644
--- a/subsys/net/lib/sockets/sockets.c
+++ b/subsys/net/lib/sockets/sockets.c
@@ -443,7 +443,7 @@ static inline ssize_t zsock_recv_stream(struct net_context *ctx,
        } while (recv_len == 0);
 
        if (!(flags & ZSOCK_MSG_PEEK)) {
-               net_context_update_recv_wnd(ctx, recv_len);
+//             net_context_update_recv_wnd(ctx, recv_len);
        }
 
        return recv_len;

I.e. using the exactly same IP as in @rlubos' screenshot above, and never increasing receive window, which makes sure it stays at 0, which should trick the peer into sending Zero Window Probe. But that never happens, instead I get only TCP Keep-Alive.

The other difference which can be seen on 2 screenshots above is that for me, TCP segments are capped to the length of 640, whereas @rlubos receives a segment of length 1280 right away.

[pfalcon]

For reference, the interface on the host is:

wlan0     Link encap:Ethernet   
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

[pfalcon]

The handshake uses packets of the same size, except I receive from the peer SYN/ACK with MSS option = 1440, whereas @rlubos - 1460. We don't send MSS option. So, how it happens that based on this, in @rlubos case, the peer selects MSS of 1280, while in my case - of 640, is a mystery.

[pfalcon]

Ah, right MSS is asymmetrical actually...

[pfalcon]

Even by sending explicit MSS option in SYN, I can't get peer to send anything else but 640 bytes data segments...

[pfalcon]

@rlubos , can you tell what version of Wireshark you use (and on what distro). I have a feeling that mine simply can't decode Zero Window Probe, as it decodes as TCP Keep-Alive even obvious retransmits. Mine is 2.2.6 on ubuntu 16.04.