[k8s] Force deletion of misbehaving pods

[romilbhardwaj]

User reported terminating 70B models served with vLLM would stall for 20 min.

service:
  readiness_probe: /v1/models

resources:
  # Can change to use more via `--gpus A100:N`.  N can be 1 to 8.
  accelerators: A100:2
  cpus: 22
  memory: 500
  # Note: Big models need LOTS of disk space, especially if saved in float32.
  # So specify a lot of disk.
  disk_size: 400
  # Keep fixed.
  cloud: kubernetes
  ports: 8000
  image_id: docker:vllm/vllm-openai:latest

envs:
  # Specify the training config via `--env MODEL=<>`
  MODEL: ""

setup: |
  conda deactivate
  python3 -c "import huggingface_hub; huggingface_hub.login('${HUGGINGFACE_TOKEN}')"

run: |
  conda deactivate
  python3 -u -m vllm.entrypoints.openai.api_server \
    --host 0.0.0.0 \
    --port 8000 \
    --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
    --trust-remote-code \
    --model $MODEL

From the kubelet:

error killing pod: [failed to “KillContainer” for “ray-node” with KillContainerError: “rpc error: code = DeadlineExceeded desc = context deadline exceeded”, failed to “KillPodSandbox” for “57e4f054-de56-4a2e-ad68-bd1d786fb02a” with KillPodSandboxError: “rpc error: code = Unknown desc = failed to stop container \“b0da7a28782608dd40df232ac4cb8d75b11a5fe64eeb45745a6dfa6bceee87b7\“: failed to kill container \“b0da7a28782608dd40df232ac4cb8d75b11a5fe64eeb45745a6dfa6bceee87b7\“: context deadline exceeded: unknown”]

kubectl delete pod NAME --grace-period=0 --force fixes it. I've seen this issue before when running training on Kubernetes outside of SkyPilot, and IIRC it is related to erring processes leaving file descriptors open that the kubelet keeps waiting for to be closed.

We should probably have this --grace-period=0 --force logic in our pod termination:

skypilot/sky/provision/kubernetes/instance.py

Lines 616 to 617 in 465d36c

	kubernetes.core_api().delete_namespaced_pod(
	pod_name, namespace, _request_timeout=config_lib.DELETION_TIMEOUT)

[github-actions]

This issue is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This issue was closed because it has been stalled for 10 days with no activity.

[romilbhardwaj]

Reopening - ran into this today when a fio benchmark got stalled, and sky down wouldn't work. Had to manually kubectl delete pod --force

[romilbhardwaj]

Ran into this again yesterday with a misbehaving mingpt run.