Sidecar is not working on certain docker hosts

[ap-wtioit]

Graylog sidecar crashes when /etc/mtab contains lines that are "too long".

● graylog-sidecar.service - Wrapper service for Graylog controlled collector
     Loaded: loaded (/etc/systemd/system/graylog-sidecar.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Mon 2025-01-27 08:58:01 CET; 3s ago
    Process: 325428 ExecStart=/usr/bin/graylog-sidecar (code=exited, status=2)
   Main PID: 325428 (code=exited, status=2)
        CPU: 21ms

Jan 23 14:58:58 host.fqdn systemd[1]: Started Wrapper service for Graylog controlled collector.
Jan 23 14:58:58 host.fqdn graylog-sidecar[2816708]: time="2025-01-23T14:58:58+01:00" level=info msg="Using node-id: cc1dcbf4-a2c2-44e6-81a5-23da826fc852"
Jan 23 14:58:58 host.fqdn graylog-sidecar[2816708]: time="2025-01-23T14:58:58+01:00" level=info msg="No node name was configured, falling back to hostname"
Jan 23 14:58:58 host.fqdn graylog-sidecar[2816708]: time="2025-01-23T14:58:58+01:00" level=info msg="Starting signal distributor"
Jan 23 14:58:58 host.fqdn graylog-sidecar[2816708]: panic: runtime error: index out of range [3] with length 3
Jan 23 14:58:58 host.fqdn graylog-sidecar[2816708]: goroutine 19 [running]:
Jan 23 14:58:58 host.fqdn graylog-sidecar[2816708]: github.com/elastic/gosigar.(*FileSystemList).Get.func1({0xc000130600?, 0xc00053c000?})
Jan 23 14:58:58 host.fqdn graylog-sidecar[2816708]:         /home/jenkins/workspace/napshots_collector-sidecar_1.5.0/vendor/github.com/elastic/gosigar/sigar_linux_common.go:113>
Jan 23 14:58:58 host.fqdn graylog-sidecar[2816708]: github.com/elastic/gosigar.readFile({0x81f16f?, 0x2f000000008271fb?}, 0xc0001e7448)
Jan 23 14:58:58 host.fqdn graylog-sidecar[2816708]:         /home/jenkins/workspace/napshots_collector-sidecar_1.5.0/vendor/github.com/elastic/gosigar/sigar_linux_common.go:386>
Jan 23 14:58:58 host.fqdn graylog-sidecar[2816708]: github.com/elastic/gosigar.(*FileSystemList).Get(0xc0001e7550)
Jan 23 14:58:58 host.fqdn graylog-sidecar[2816708]:         /home/jenkins/workspace/napshots_collector-sidecar_1.5.0/vendor/github.com/elastic/gosigar/sigar_linux_common.go:106>
Jan 23 14:58:58 host.fqdn graylog-sidecar[2816708]: github.com/Graylog2/collector-sidecar/common.GetFileSystemList75({0xc0001e7680?, 0xc0001e7680?})
Jan 23 14:58:58 host.fqdn graylog-sidecar[2816708]:         /home/jenkins/workspace/napshots_collector-sidecar_1.5.0/common/sigar.go:111 +0x3f
Jan 23 14:58:58 host.fqdn graylog-sidecar[2816708]: github.com/Graylog2/collector-sidecar/api.UpdateRegistration(0x8087a0?, {0x0, 0x0}, 0xc0001de640, 0xc000190200?, 0xc00044e2d>
Jan 23 14:58:58 host.fqdn graylog-sidecar[2816708]:         /home/jenkins/workspace/napshots_collector-sidecar_1.5.0/api/graylog.go:170 +0x185
Jan 23 14:58:58 host.fqdn graylog-sidecar[2816708]: github.com/Graylog2/collector-sidecar/services.updateCollectorRegistration(0x0?, {0x0, 0x0}, 0x0?, 0x0?)
Jan 23 14:58:58 host.fqdn graylog-sidecar[2816708]:         /home/jenkins/workspace/napshots_collector-sidecar_1.5.0/services/periodicals.go:124 +0x165
Jan 23 14:58:58 host.fqdn graylog-sidecar[2816708]: github.com/Graylog2/collector-sidecar/services.StartPeriodicals.func1()
Jan 23 14:58:58 host.fqdn graylog-sidecar[2816708]:         /home/jenkins/workspace/napshots_collector-sidecar_1.5.0/services/periodicals.go:63 +0x385
Jan 23 14:58:58 host.fqdn graylog-sidecar[2816708]: created by github.com/Graylog2/collector-sidecar/services.StartPeriodicals in goroutine 1
Jan 23 14:58:58 host.fqdn graylog-sidecar[2816708]:         /home/jenkins/workspace/napshots_collector-sidecar_1.5.0/services/periodicals.go:38 +0x4f
Jan 23 14:58:58 host.fqdn systemd[1]: graylog-sidecar.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

On our server the long mtab lines are caused by docker, the longest being 4178 bytes:

overlay /var/lib/docker/overlay2/229ea374b94e0a60368c0759cc104aa3ef44e7bd46e8e389603307673aab34e7/merged overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/25NOBV3WHXG5P3GRO2FRUCWAB3:/var/lib/docker/overlay2/l/NORNMK2W42ELHAA4GJWQX5ZOAW:/var/lib/docker/overlay2/l/RDVR4KEE5JQPI5SDRG333UL7RR:/var/lib/docker/overlay2/l/NDA6EGGA5TVHNZYRLSNACMCQWF:/var/lib/docker/overlay2/l/PSHEG744SP6T5QTAYIBKOHQRTF:/var/lib/docker/overlay2/l/CVJ6EWWFZZB4XNNCTTX2OWOKC7:/var/lib/docker/overlay2/l/OYEE2F2W4IA5RFZE6ZBTHJIUSH:/var/lib/docker/overlay2/l/ZRFGD5OZAXDXEACSMEYWGGSBX7:/var/lib/docker/overlay2/l/CILYIYAPTFNWP7UM7S55KQSI4D:/var/lib/docker/overlay2/l/F4PHFL4L2NAF5GJSJIGQPERKPU:/var/lib/docker/overlay2/l/5R62UHZSGCI2AH7SYFUHVMH56E:/var/lib/docker/overlay2/l/2SENKE2UI73NBAGTGIVH6K3O2A:/var/lib/docker/overlay2/l/DWRXBEGY7U3IIIBVEA6V2AKRBJ:/var/lib/docker/overlay2/l/3GUSTJRGFDDYXLT7V44I6YB3XN:/var/lib/docker/overlay2/l/DEC4B5I67LMALKVPQJAFP5HX46:/var/lib/docker/overlay2/l/ZI5YOAKJUTPBKKOMMTWKITYIXS:/var/lib/docker/overlay2/l/RKWKKH46TJP7S6ZERYR5SPS73N:/var/lib/docker/overlay2/l/LQTPCWBTTXHTHKNCQZF2E2D6AP:/var/lib/docker/overlay2/l/HUNN3R34OVJSNKILBJSMTJTIFO:/var/lib/docker/overlay2/l/S6DGOW3IPS2PT463PG7SYRMUJX:/var/lib/docker/overlay2/l/GKZU26J5SSSQH2RBIMKYUDUNWQ:/var/lib/docker/overlay2/l/SPS2VYCUOWEKCA47Q2FL3T3ZVO:/var/lib/docker/overlay2/l/DJ7CFAJ64XB7AFGAM5UZ3MEOHT:/var/lib/docker/overlay2/l/MJOQ7IFA5JLZ4APQHPRA2GFIKF:/var/lib/docker/overlay2/l/DVAUELMLR5VFOLXLCG4RHVJJH4:/var/lib/docker/overlay2/l/DDV4OD36D4CACTBTVXQ7CQVJTK:/var/lib/docker/overlay2/l/TQSEOKAR5Q4QJI52HJH6TWSENX:/var/lib/docker/overlay2/l/ZJPLLJWA2EDO2QFO6JL5YU5NXC:/var/lib/docker/overlay2/l/NW7MD4MUYBUC27L6XYJTKELXZU:/var/lib/docker/overlay2/l/HF3FA5NGW27JT43D44L4UFWSUS:/var/lib/docker/overlay2/l/MG27ZBUPW3NNRRM4TPE672DPKZ:/var/lib/docker/overlay2/l/IBM7RSWRDIZJPZNAOOW6H3SQUW:/var/lib/docker/overlay2/l/GJI7YYX4T3YCSS35JXQVOLMUPU:/var/lib/docker/overlay2/l/POIXRSHOGGH5SRY6IKBSO7WM6E:/var/lib/docker/overlay2/l/C3HJJ3QW33WNHC33X5VFLUT3DR:/var/lib/docker/overlay2/l/CZNUABCWBZ6T6Z636WZUKBJY3L:/var/lib/docker/overlay2/l/Z2KSHZNRNQ4HUHLXOA4ZBE2P3E:/var/lib/docker/overlay2/l/SJTVBG2UJSGT23AVCSDVR4PNVL:/var/lib/docker/overlay2/l/M7T7PG36WVHPGA5J5QFK6E3Z3T:/var/lib/docker/overlay2/l/3K52OVGIRBUIUOQ7A6XKE3CI7Q:/var/lib/docker/overlay2/l/NPAO3V2FJSNSOIR5IJYRXSJUN2:/var/lib/docker/overlay2/l/MJEUBSKJ3EIEXKI63O23MVJP6L:/var/lib/docker/overlay2/l/KI4KHMEL5YIOHU6M5HJOOP7SOA:/var/lib/docker/overlay2/l/OJ2AAUKW7A27XRU3A6JLFSXPED:/var/lib/docker/overlay2/l/JMSLHEDCQMFBEPIPZGKKPUL2GZ:/var/lib/docker/overlay2/l/63VRX6JDKMNOBZTXFFXGTWLHH4:/var/lib/docker/overlay2/l/WATKHFOLY7ATZ6OSPN63M4SMTN:/var/lib/docker/overlay2/l/2OGULZVZAUODJXF3ZKMTMIRYWC:/var/lib/docker/overlay2/l/2NZZ4KSCVJKL2GCIHHXWPYY3VC:/var/lib/docker/overlay2/l/XOMQEH3JRC4GCBK4ZKEKTISBZV:/var/lib/docker/overlay2/l/SJHHUSXZQ4ZPEZGOFYJ2B7FZHK:/var/lib/docker/overlay2/l/RFETRSPAIPUYYLBY4UWAUEI4WD:/var/lib/docker/overlay2/l/FBF7BNUTJTB4QXRBS4IMF3XTA5:/var/lib/docker/overlay2/l/FEPH4I477I3WOOLYHEFUDJYNYC:/var/lib/docker/overlay2/l/JNZJLFLK6EBJOXBVRY423B5WCW:/var/lib/docker/overlay2/l/QXPFXSXPE3OFAVKJMKEU5OBLGF:/var/lib/docker/overlay2/l/X4LAMESX5ANCAMDB3H5R7EEAGH:/var/lib/docker/overlay2/l/6EM7RPQGAT2QUCPZTOYQL2YA7W:/var/lib/docker/overlay2/l/4YCKFX7RIZSE3X7NR33L2AMXTF:/var/lib/docker/overlay2/l/J4IAXBV7AJD4UXI6E2DKBI5USH:/var/lib/docker/overlay2/l/U4LYG45JPNMLITFZPXRZ4YORNH:/var/lib/docker/overlay2/l/U7YWJNILQ4S3E2GMNKVBDU3HE2:/var/lib/docker/overlay2/l/4Q7NTRFPH6PISJPCGL6MSNKXAB:/var/lib/docker/overlay2/l/S4Q5LW6NPUGGV4PNH7RGIZULDB:/var/lib/docker/overlay2/l/JH4VWREORYUTBEG5OWJGRUTSWZ:/var/lib/docker/overlay2/l/DOSO5DTS2LYOPZNKNNDA6UXJVN:/var/lib/docker/overlay2/l/GI43U22RIMZQU2WCKLILS7VIWU:/var/lib/docker/overlay2/l/ELJJ5B5BDXE2XLUU3SXA7NCFV6:/var/lib/docker/overlay2/l/WSEEZO7VSPHRUZV6ZFJZFI2JOJ:/var/lib/docker/overlay2/l/Q7CHINR3TSEIMNYBXBQEM7GV4X:/var/lib/docker/overlay2/l/OJQEGWPTEYASKMEKOJ2APQ47XI,upperdir=/var/lib/docker/overlay2/229ea374b94e0a60368c0759cc104aa3ef44e7bd46e8e389603307673aab34e7/diff,workdir=/var/lib/docker/overlay2/229ea374b94e0a60368c0759cc104aa3ef44e7bd46e8e389603307673aab34e7/work 0 0

This seems to be caused by elastic/gosigar#166