Concurrent request very slow

[flexwang]

I am testing trtllm backend v0.6.0 for llama2-7b with below setup. The code snippets is as below. If I set one request with length being about 1000, it took about 2-3 seconds to finish. And if I send 16 requests at the same time, it took about 30 seconds, so it feels like there is no inflight batching at all.

        with multiprocessing.Manager() as manager:
            def send(client, input_id, input_length, i, shared_list):
                inputs = [
                    _input("input_ids", np.array(input_id, dtype=np.int32).reshape(1, -1)),
                    _input("input_lengths", np.array([input_length], dtype=np.int32).reshape(1, -1)),
                    _input("request_output_len", np.array([max_new_tokens], dtype=np.uint32).reshape(1, -1)),
                    _input("end_id", np.array([2], dtype=np.uint32).reshape(1, -1)),
                ]
                with grpcclient.InferenceServerClient("localhost:8001", verbose=False) as client:
                    print('send')
                    shared_list[i] = client.infer('tensorrt_llm', inputs).as_numpy('sequence_length').reshape(-1)[0]
            processes = []
            shared_list = manager.list([""] * batch_size)
            start = time.time()
            for i in range(batch_size):
                process = multiprocessing.Process(target=send, args=(None, input_id[i], input_lengths[i], i, shared_list))
                processes.append(process)
                process.start()

            for process in processes:
                process.join()
            latency = time.time() - start

Trtllm setup as below

name: "tensorrt_llm"
backend: "tensorrtllm"
max_batch_size: 1024

model_transaction_policy {
  decoupled: false
}

input [
  {
    name: "input_ids"
    data_type: TYPE_INT32
    dims: [ -1 ]
    allow_ragged_batch: true
  },
  {
    name: "input_lengths"
    data_type: TYPE_INT32
    dims: [ 1 ]
    reshape: { shape: [ ] }
  },
  {
    name: "request_output_len"
    data_type: TYPE_UINT32
    dims: [ 1 ]
  },
  {
    name: "end_id"
    data_type: TYPE_UINT32
    dims: [ 1 ]
    reshape: { shape: [ ] }
    optional: true
  },
  {
    name: "pad_id"
    data_type: TYPE_UINT32
    dims: [ 1 ]
    reshape: { shape: [ ] }
    optional: true
  }
]
output [
  {
    name: "output_ids"
    data_type: TYPE_INT32
    dims: [ -1, -1 ]
  },
  {
    name: "sequence_length"
    data_type: TYPE_INT32
    dims: [ -1 ]
  }
]
instance_group [
  {
    count: 1
    kind : KIND_CPU
  }
]
parameters: {
  key: "max_beam_width"
  value: {
    string_value: "1"
  }
}
parameters: {
  key: "FORCE_CPU_ONLY_INPUT_TENSORS"
  value: {
    string_value: "no"
  }
}
parameters: {
  key: "gpt_model_type"
  value: {
    string_value: "inflight_batching"
  }
}
parameters: {
  key: "gpt_model_path"
  value: {
    string_value: "/models/trt_engines/llama-2-7b-chat-hf/1-gpu/"
  }
}
parameters: {
  key: "max_tokens_in_paged_kv_cache"
  value: {
    string_value: "8192"
  }
}
parameters: {
  key: "max_kv_cache_length"
  value: {
    string_value: "4096"
  }
}
parameters: {
  key: "batch_scheduler_policy"
  value: {
    string_value: "max_utilization"
  }
}
parameters: {
  key: "max_num_sequences"
  value: {
    string_value: "96"
  }
}
parameters: {
  key: "exclude_input_in_output"
  value: {
    string_value: "true"
  }
}

Below is how we generate the engine

python build.py --model_dir /models/llama-2-7b-chat-hf/ \
                --dtype float16 \
                --remove_input_padding \
                --use_gpt_attention_plugin float16 \
	 --use_inflight_batching \
                --enable_context_fmha \
        --paged_kv_cache \
        --max_input_len 3072 \
        --max_output_len 1024 \
                --use_gemm_plugin float16 \
                 --output_dir /models/trt_engines/llama-2-7b-chat-hf/1-gpu/ \
        --world_size 1 \
        --max_batch_size 96 \
 --tp_size 1

[flexwang]

A more broad question, what is the recommended way of running batched inference through triton+trtllm - asking since the server only accepts one prompt at a time.

[sfc-gh-zhwang]

I think the issue is from max_tokens_in_paged_kv_cache, but as long as i set it for larger than 16k, it will OOM. It seems weird on A100(40GiB) machine for a llama2-7b fp16 model.

[shannonphu]

Same question, it feels like there is no inflight batching or dynamic batching. Were you able to resolve this?

[fursovia]

Same issue. It seams that inflight batching does not work

[kaiyux]

Could you please set a proper value to --max_num_tokens when building the engine?

Besides, inflight_fused_batching is recommended instead of inflight_batching for the argument gpt_model_type.

Could you please try that and let us know if that helps? Thanks.

[fursovia]

@kaiyux should I set --max_num_tokens to be equal to --max_input_len?

I've tried both inflight_fused_batching and inflight_batching -- the results were similar

[kaiyux]

should I set --max_num_tokens to be equal to --max_input_len?

@fursovia --max_num_tokens could be roughly estimated by max_batch_size * ratio * max_input_len + max_batch_size * (1 - ratio), ratio stands for the ratio of the context phase requests on each forward calculation in the inference, it's usually around 5~20%, and depends on the actual scenario.

Please also note that you may also want to increase --max_batch_size, so that larger batches are allowed to feed into the engine.

[sfc-gh-zhwang]

@kaiyux you mean max_batch_size * ratio * max_input_len + max_batch_size * (1 - ratio) * max_output_len?

[kaiyux]

@sfc-gh-zhwang No, I meant max_batch_size * ratio * max_input_len + max_batch_size * (1 - ratio).

Let me explain more on this: --max_num_tokens means the max number of input tokens, which equals to max_batch_size * max_input_len by default. When remove input padding is enabled, since the tokens from different sequences are packed together, a max_num_tokens lower than default is allowed, and engines can be built with higher max_batch_size, which helps getting higher throughput because larger batch size could be reached.

Let's say if the percentage of context phase requests among all requests is ratio, since the max input length of context phase is max_input_len, and that of generation phase is 1, the rough estimation would be max_batch_size * ratio * max_input_len + max_batch_size * (1 - ratio) * 1.

[PannenetsF]

Hi, I can reproduce the problem stably.

python benchmark_core_model.py  --num-requests 200 --max-input-len 1024 -i grpc token-norm-dist --input-mean 256 --input-stdev 0 --output-mean 256 --output-stdev 0 
python benchmark_core_model.py  --num-requests 200 --max-input-len 1024 -i grpc token-norm-dist --input-mean 512 --input-stdev 0 --output-mean 512 --output-stdev 0 
python benchmark_core_model.py  --num-requests 200 --max-input-len 1024 -i grpc token-norm-dist --input-mean 256 --input-stdev 0 --output-mean 256 --output-stdev 0 

You will find the command in L3 gets a 5x slow-down, even if the workload is the same as in L1.

[flexwang]

using v0.6.1 fixed the issue.