master: opal/stacktrace improvements #2772
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Adding @hjelmn since they were seeing sometime similar to the problem I attempted a fix for in this PR. I found the discussion here helpful: |
jjhursey
force-pushed
the
topic/stacktrace-improv
branch
from
e54ecf3 to
332d633
Compare
- This prevents us for accidentally masking a signal that was meant to terminate the application. Signed-off-by: Joshua Hursey <[email protected]>
- New MCA option: opal_stacktrace_output
- Specifies where the stack trace output stream goes.
- Accepts: none, stdout, stderr, file[:filename]
- Default filename 'stacktrace'
- Filename will be `stacktrace.PID`, or if VPID is available,
then the filename will be `stacktrace.VPID.PID`
- Update util/stacktrace to allow for different output avenues
including files. Previously this was hardcoded to 'stderr'.
- Since opal_backtrace_print needs to be signal safe, passing it a
FILE object that actually represents a file stream is difficult. This
is because we cannot open the file in the signal handler using
`fopen` (not safe), but have to use `open` (safe). Additionally, we
cannot use `fdopen` to convert the `int fd` to a `FILE *fh` since it
is also not signal safe.
- I did not want to break the backtrace.h API so I introduced a new
rule (documented in `backtrace.c`) that if the `FILE *file`
argument is `NULL` then look for the `opal_stacktrace_output_fileno`
variable to tell you which file descriptor to use for output.
Signed-off-by: Joshua Hursey <[email protected]>
jjhursey
force-pushed
the
topic/stacktrace-improv
branch
from
332d633 to
6d98559
Compare
jsquyres
changed the title
|
Refs #2801 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The first commit fixes a bug that I saw with this test program:
If I run the program with Open MPI, it catches the first
SIGFPEand prints a stack through the opal/stacktrace mechanism, but let's the program continue executing 😞 . When it hits the secondSIGFPEthen the system's default signal handler is invoked and kills the program.I would expect Open MPI to print the stack and kill the program on the first instance of this signal. And if I run Open MPI with
-mca opal_signal 6,7,11(skipping8 = SIGFPE) then it does exactly that (exits the program at the first encounter).So the first commit makes sure that we re-raise the signal after processing the stacktrace.
The second commit adds some more flexibility to the stacktrace mechanism in particular where the stacktrace output is directed. Default is still
stderr, but now we have options fornone,stdout, and a set of per-process files.See the commit messages for some more detail.