Data loss in Secnetperf tool transmitting data

[baiyesong]

When I was testing with the Secnetperf tool, the amount of data received by the Server side was different from the amount of data that the Client side confirmed was successfully sent.

The size of the data received by the Server side is different from the amount of data successfully sent by the Client side.

I only added the logic for outputting and calculating the total data volume in the server side (PerfServer.cpp) of secnetpef.

The startup command and results of the Client side are as follows.

root@bys:~/msquic/build/bin/Release# ./secnetperf -target:localhost -port:8500 -cc:bbr -conns:1 -streams:1 -ptput:1 -up:1048576000 -iosize:1048576 
Started!

Result: Upload 1048576000 bytes @ 2316433 kbps (3621.347 ms).

The printed information of the Server side is as follows.

QUIC_STREAM_EVENT_RECEIVE
write file : 63090, total:1048381314
QUIC_STREAM_EVENT_RECEIVE
write file : 63096, total:1048444410
QUIC_STREAM_EVENT_RECEIVE
write file : 43024, total:1048487434

The modified code of the Server side is as follows:

QUIC_STATUS
PerfServer::StreamCallback(
    _In_ StreamContext* Context,
    _In_ HQUIC StreamHandle,
    _Inout_ QUIC_STREAM_EVENT* Event
    ) {
    switch (Event->Type) {
    case QUIC_STREAM_EVENT_RECEIVE:
    WriteOutput("QUIC_STREAM_EVENT_RECEIVE\n");    
    totalRecv += (int)Event->RECEIVE.Buffers[0].Length; //caculate read size!   
    WriteOutput("write file : %u, total:%ld\n",Event->RECEIVE.Buffers[0].Length,totalRecv);
        if (!Context->ResponseSizeSet) {
            uint8_t* Dest = (uint8_t*)&Context->ResponseSize;
   ...........

The printout shows that the server received 1,048,487,434 bytes of data, which is different from the expected 1,048,576,000 bytes. When the transmission is repeated multiple times, the data received by the server is always less than the expected 1,048,576,000 bytes.

Important
When a transmission error occurs, the remaining memory on my server is actually less than 1GB. However, if I free up the memory to make the available amount 3GB, the above - mentioned transmission can succeed!

after I reduce the target transmission data volume, for example, setting -up:524288000 -iosize:1048576, the transmission will also succeed even when the memory is less than 1GB. The final printed value on the server side is the same as that on the client side.

Therefore, I preliminarily infer that the reason is as follows:

Insufficient server memory causes an exception in the frame - based sending at the underlying level of msquic.

Is my inference correct?

If the server is in a low - memory situation, can this problem be solved by adjusting configuration items? If so, is there a recommended configuration modification list for low - memory scenarios?

[nibanks]

 totalRecv += (int)Event->RECEIVE.Buffers[0].Length; //caculate read size!

You have a bug that you are only looking at [0] in this code. There can be more than 1 item in this array sometimes.

[baiyesong]

I noticed that the struct Buffer sent in Client is as follows:

Init part:

RequestBuffer.Init(IoSize, Timed ? UINT64_MAX : Download);

Struct part:

struct PerfIoBuffer {
        QUIC_BUFFER* Buffer {nullptr};
        operator QUIC_BUFFER* () noexcept { return Buffer; }
        ~PerfIoBuffer() noexcept { if (Buffer) { CXPLAT_FREE(Buffer, QUIC_POOL_PERF); } }
        void Init(uint32_t IoSize, uint64_t Initial) noexcept {
            Buffer = (QUIC_BUFFER*)CXPLAT_ALLOC_NONPAGED(sizeof(QUIC_BUFFER) + IoSize, QUIC_POOL_PERF);
            Buffer->Length = IoSize;
            Buffer->Buffer = (uint8_t*)(Buffer + 1);
            *(uint64_t*)(Buffer->Buffer) = CxPlatByteSwapUint64(Initial);
            for (uint32_t i = sizeof(uint64_t); i < IoSize; ++i) {
                Buffer->Buffer[i] = (uint8_t)i;
            }
        }
    } RequestBuffer;

Send part:

QUIC_BUFFER* Buffer = Client.RequestBuffer;
....
MsQuic->StreamSend(Handle, Buffer, 1, Flags, Buffer);

So,It seems that in the client, only one buffer is sent every time it is sent？

My other understanding is that although the Send function in the client only contains one Buffer each time it sends, the Buffer array received in one instance on the server may contain multiple Buffer arrays?

[nibanks]

The number and size of buffers on the send side is completely independent from the receive side.

[baiyesong]

get it! thanks!