Skip to content

Jobs that error don't have the sidecar killed by pepr #687

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
corang opened this issue Aug 21, 2024 · 2 comments · Fixed by #813
Closed

Jobs that error don't have the sidecar killed by pepr #687

corang opened this issue Aug 21, 2024 · 2 comments · Fixed by #813
Assignees
@rjferguson21
Labels
bug Something isn't working
Milestone
0.28.0

Comments

@corang
Copy link
Contributor

corang commented Aug 21, 2024 •
edited
Loading

Steps to reproduce

  1. Have a job pod that isn't istio aware error via a non-zero exit code on main container

Expected result

Pepr kills istio sidecar so that the job pod can restart/retry

Actual Result

Job pod sidecar runs forever so job never retries/eventually completes

Severity/Priority

Medium, can break automated deployments
@corang corang added the possible-bug Something may not be working label Aug 21, 2024
@mjnagel
Copy link
Contributor

mjnagel commented Aug 21, 2024 •
edited by rjferguson21
Loading

Lines in question - https://github.com/defenseunicorns/uds-core/blob/main/src/pepr/istio/index.ts#L50-L51

@mjnagel mjnagel added bug Something isn't working and removed possible-bug Something may not be working labels Sep 6, 2024
@mjnagel
Copy link
Contributor

mjnagel commented Sep 6, 2024

Should validate the intent here and make sure that jobs are able to retry on failure as expected. Likely will need to update the lines in question to not consider the exit code.

@mjnagel mjnagel added this to the 0.28.0 milestone Sep 16, 2024
@rjferguson21 rjferguson21 mentioned this issue Sep 24, 2024
Merged
5 tasks
noahpb pushed a commit that referenced this issue Sep 25, 2024
@rjferguson21 @mjnagel @noahpb
fix: ensure istio sidecar is killed if job fails (#813)
97cba4b 
## Description
Ensure Istio sidecar is killed if Job pod exits with non-zero exit
status.

Tested with Jobs (exit code zero and non-zero)
```yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: failing-job
  namespace: keycloak
spec:
  template:
    spec:
      containers:
      - name: fail-container
        image: quay.io/keycloak/keycloak:25.0.6 
        command: ["sh", "-c", "echo 'This will fail'; exit 1"]
      restartPolicy: Never
  backoffLimit: 3
---
apiVersion: batch/v1
kind: Job
metadata:
  name: success-job
  namespace: keycloak
spec:
  template:
    spec:
      containers:
      - name: succeed-container
        image: quay.io/keycloak/keycloak:25.0.6 
        command: ["sh", "-c", "echo 'This will succeed'; exit 0"]
      restartPolicy: Never
  backoffLimit: 3
  ```
## Related Issue

Fixes #687

## Type of change

- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Other (security config, docs update, etc)

## Checklist before merging

- [ ] Test, docs, adr added or updated as needed
- [ ] [Contributor Guide](https://github.com/defenseunicorns/uds-template-capability/blob/main/CONTRIBUTING.md) followed

Co-authored-by: Micah Nagel <[email protected]>
docandrew pushed a commit that referenced this issue Sep 25, 2024
@rjferguson21 @mjnagel @docandrew
fix: ensure istio sidecar is killed if job fails (#813)
29dcb93 
## Description
Ensure Istio sidecar is killed if Job pod exits with non-zero exit
status.

Tested with Jobs (exit code zero and non-zero)
```yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: failing-job
  namespace: keycloak
spec:
  template:
    spec:
      containers:
      - name: fail-container
        image: quay.io/keycloak/keycloak:25.0.6 
        command: ["sh", "-c", "echo 'This will fail'; exit 1"]
      restartPolicy: Never
  backoffLimit: 3
---
apiVersion: batch/v1
kind: Job
metadata:
  name: success-job
  namespace: keycloak
spec:
  template:
    spec:
      containers:
      - name: succeed-container
        image: quay.io/keycloak/keycloak:25.0.6 
        command: ["sh", "-c", "echo 'This will succeed'; exit 0"]
      restartPolicy: Never
  backoffLimit: 3
  ```
## Related Issue

Fixes #687

## Type of change

- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Other (security config, docs update, etc)

## Checklist before merging

- [ ] Test, docs, adr added or updated as needed
- [ ] [Contributor Guide](https://github.com/defenseunicorns/uds-template-capability/blob/main/CONTRIBUTING.md) followed

Co-authored-by: Micah Nagel <[email protected]>
UnicornChance pushed a commit that referenced this issue Sep 26, 2024
@rjferguson21 @mjnagel @UnicornChance
fix: ensure istio sidecar is killed if job fails (#813)
eaa629c 
## Description
Ensure Istio sidecar is killed if Job pod exits with non-zero exit
status.

Tested with Jobs (exit code zero and non-zero)
```yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: failing-job
  namespace: keycloak
spec:
  template:
    spec:
      containers:
      - name: fail-container
        image: quay.io/keycloak/keycloak:25.0.6 
        command: ["sh", "-c", "echo 'This will fail'; exit 1"]
      restartPolicy: Never
  backoffLimit: 3
---
apiVersion: batch/v1
kind: Job
metadata:
  name: success-job
  namespace: keycloak
spec:
  template:
    spec:
      containers:
      - name: succeed-container
        image: quay.io/keycloak/keycloak:25.0.6 
        command: ["sh", "-c", "echo 'This will succeed'; exit 0"]
      restartPolicy: Never
  backoffLimit: 3
  ```
## Related Issue

Fixes #687

## Type of change

- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Other (security config, docs update, etc)

## Checklist before merging

- [ ] Test, docs, adr added or updated as needed
- [ ] [Contributor Guide](https://github.com/defenseunicorns/uds-template-capability/blob/main/CONTRIBUTING.md) followed

Co-authored-by: Micah Nagel <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rjferguson21 rjferguson21

Labels
bug Something isn't working
Projects
None yet
Milestone
0.28.0
Development

Successfully merging a pull request may close this issue.

fix: ensure istio sidecar is killed if job fails defenseunicorns/uds-core
3 participants
@corang @rjferguson21 @mjnagel