Skip to content

kops jobs do not dump kube-scheduler, kube-ontroller-manager, kube-proxy, etc. logs #2397

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wongma7 opened this issue Apr 3, 2017 · 3 comments
Assignees

Comments

@wongma7
Copy link
Contributor

wongma7 commented Apr 3, 2017

For instance, in this random recent PR job https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/41929/pull-kubernetes-e2e-kops-aws/19348/build-log.txt we run cluster/log-dump.sh and get

I0403 16:01:41.664] Dumping master and node logs to /workspace/_artifacts
I0403 16:01:41.665] Copying 'kube-apiserver kube-scheduler rescheduler kube-controller-manager etcd etcd-events glbc cluster-autoscaler kube-addon-manager fluentd cloud-init-output' from ec2-54-67-111-203.us-west-1.compute.amazonaws.com
I0403 16:01:41.665] scp: /var/log/kube-scheduler.log: Permission denied
...
I0403 16:01:41.665] scp: /var/log/kube-controller-manager.log: Permission denied
...
I0403 16:01:41.667] scp: /var/log/kube-proxy.log: Permission denied

Here's what /var/log looks like in a kops cluster:

-r--------  1 root root              31705 Apr  3 22:46 kube-controller-manager.log

LOG_DUMP_SSH_USER is admin, who doesn't have permissions to read & copy the file.

This could be a kops issue, I don't know what the standard is for setting permissions on these log files.

justinsb added a commit to justinsb/kubernetes that referenced this issue Apr 4, 2017
We make the files world-readable, so that installation techniques that
lock down the logfiles can still be dumped.

Issue kubernetes/test-infra#2397
@justinsb
Copy link
Member

justinsb commented Apr 4, 2017

Trying out a chmod -R a+r /var/log in kubernetes/kubernetes#44017

k8s-github-robot pushed a commit to kubernetes/kubernetes that referenced this issue Apr 4, 2017
Automatic merge from submit-queue

cluster/log-dump - chmod files before dumping

We make the files world-readable, so that installation techniques that
lock down the logfiles can still be dumped.

Issue kubernetes/test-infra#2397

```release-note
NONE
```
@fejta
Copy link
Contributor

fejta commented Apr 6, 2017

/assign @justinsb

@wongma7
Copy link
Contributor Author

wongma7 commented Apr 6, 2017

change to cluster/log-dump fixed it, no more permission denied errors in recent jobs :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants