EL thread labelling probably wrong

[weissi]

I just received a top -H -n 1 from a container which only runs one SwiftNIO program which has too many ELGs. What I saw there is

    342 root      20   0 27.311g 9.857g  48992 S  0.0  2.6   0:00.00 NIO-ELT-1-#28
    343 root      20   0 27.311g 9.857g  48992 S  0.0  2.6   0:00.00 NIO-ELT-1-#28
    344 root      20   0 27.311g 9.857g  48992 S  0.0  2.6   0:00.00 NIO-ELT-1-#28
    345 root      20   0 27.311g 9.857g  48992 S  0.0  2.6   0:00.00 NIO-ELT-1-#28
    346 root      20   0 27.311g 9.857g  48992 S  0.0  2.6   0:00.00 NIO-ELT-1-#28

see that the thread name NIO-ELT-1-#28 is repeated multiple times? That can't be right, the myGroupID (1 in this case) should be increasing for every spawned MTELG.

[Lukasa]

Hmm. A quick local test of the atomic add does reveal it's working well.

[weissi]

I agree, the code

        let myGroupID = nextEventLoopGroupID.add(1)
        self.myGroupID = myGroupID
        var idx = 0
        self.eventLoops = threadInitializers.map { initializer in
            // Maximum name length on linux is 16 by default.
            let ev = MultiThreadedEventLoopGroup.setupThreadAndEventLoop(name: "NIO-ELT-\(myGroupID)-#\(idx)",
                                                                         selectorFactory: selectorFactory,
                                                                         initializer: initializer)
            idx += 1
            return ev
        }

looks easy enough and isn't obviously wrong... Really can't see how we could get those clashes.

[weissi]

hmmmm, maybe is there something that could propagate the thread name to some other, new thread? Let's say we spawn a posix thread from a thread with a certain label, maybe the new thread inherits that label?

[weissi]

running a quick

$ pbpaste | while read j j j j j j j j j j j line; do echo "$line"; done | sort | uniq -c | sort -n
   1 NIO-ELT-0-#0
   1 NIO-ELT-0-#1
   1 NIO-ELT-0-#2
   1 NIO-ELT-1-#13
   1 NIO-ELT-1-#20
   1 NIO-ELT-1-#23
   1 NIO-ELT-1-#27
   1 NIO-ELT-1-#4
   1 NIO-ELT-2-#0
   1 NIO-ELT-3-#0
   1 NIO-ELT-4-#0
   1 NIO-ELT-5-#0
   1 NIO-ELT-6-#0
   1 NIO-ELT-7-#0
   2 NIO-ELT-1-#24
   2 NIO-ELT-1-#7
   3 NIO-ELT-1-#1
   3 NIO-ELT-1-#12
   3 NIO-ELT-1-#18
   3 NIO-ELT-1-#2
   3 NIO-ELT-1-#25
   3 NIO-ELT-1-#5
   4 NIO-ELT-1-#14
   4 NIO-ELT-1-#22
   5 NIO-ELT-1-#8
   8 NIO-ELT-1-#0
   8 NIO-ELT-1-#10
   8 NIO-ELT-1-#9
   9 NIO-ELT-1-#21
  10 NIO-ELT-1-#19
  10 NIO-ELT-1-#6
  17 NIO-ELT-1-#11
  21 NIO-ELT-1-#16
  21 NIO-ELT-1-#26
  22 NIO-ELT-1-#15
  26 NIO-ELT-1-#3
  27 NIO-ELT-1-#28
  43 NIO-ELT-1-#17
 315 program-name

it sure looks like something odd is going on! We only have 1 thread called NIO-ELT-1-#13 but 43 threads called NIO-ELT-1-#17

[weissi]

man page suggests that new threads inherit the program name. hmm

       By default, all the threads created using pthread_create()
       inherit the program name.  The pthread_setname_np() function can
       be used to set a unique name for a thread, which can be useful
       for debugging multithreaded applications.  The thread name is a
       meaningful C language string, whose length is restricted to 16
       characters, including the terminating null byte ('\0').  The
       thread argument specifies the thread whose name is to be changed;
       name specifies the new name.

[weissi]

Okay, I could repro this. By running

group.next().execute {
    for _ in 0..<200 {
        DispatchQueue(label: "q").async { pause() }
    }

    for _ in 0..<200 {
        DispatchQueue.global().async { pause() }
    }
}

I got the following picture

root@cd68f07999a7:/Users/johannes/devel/swift-nio# cat /proc/1/task/*/comm
NIOEchoServer
NIO-ELT-0-#1
NIO-ELT-0-#1
NIO-ELT-0-#1
NIO-ELT-0-#1
NIO-ELT-0-#1
NIO-ELT-0-#1
NIO-ELT-0-#1
NIO-ELT-0-#1
NIO-ELT-0-#1
NIO-ELT-0-#1
NIO-ELT-0-#1
NIO-ELT-0-#1
NIO-ELT-0-#1
NIO-ELT-0-#1
NIO-ELT-0-#1
NIO-ELT-0-#1
NIO-ELT-0-#1
NIO-ELT-0-#1
NIO-ELT-0-#1
NIO-ELT-0-#1
NIO-ELT-0-#1
NIO-ELT-0-#1
[...]

Unclear where the issue is:

• is NIO setting the thread name in the wrong way?
• is Dispatch doing the wrong thing?
• is it a kernel/pthread bug?

[Lukasa]

Do you need both loops, or does only one trigger it?

[weissi]

I think every Dispatch thread that is created from an .async call from a NIO thread (and probably other labelled threads) will propagate that label.

[weissi]

This is not a NIO bug, this is a Dispatch bug: https://bugs.swift.org/browse/SR-15686