Skip to content

nthread limit when running in taskset/cpuset #35787

Closed
@sgaure

Description

@sgaure

The number of threads available for multiprocessing is capped by the number of cpus. This, I believe, is done to avoid running more threads than cpus. However, when the process has access to fewer cpus because of bindings in the OS (via taskset/cpuset/numactl on linux. I suppose similar mechanisms are available in other OSs), the nthread-cap is still the number of cpus in the computer.

For consistency, the actual limit should be honored:
Here is an example on a 16-cpu linux machine:

$ julia -t 24 -E 'Threads.nthreads()'
16
$ taskset -c 0-3 julia -t 24 -E 'Threads.nthreads()'
16

To be consistent the last command should output 4.

Certain batch-systems will only give the user the number of cpus asked for, and if the user script tells julia to use more there will be contention in julia.

This could also be combined with allowing all as a thread-specification, meaning all cpus you can get hold of, to yield something like this:

$ taskset -c 0-3 julia -t all -E 'Threads.nthreads()'
4

Even if I can't come up with a good example, I might possibly imagine cases where you actually want more threads than cpus, so there should perhaps be a way to override the automatic cap (-T in place of -t?).

Metadata

Metadata

Assignees

No one assigned

    Labels

    multithreadingBase.Threads and related functionality

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions