Random segfaults with req_raw_header

[ghost]

Hello

We are seeing occasional segfaults using req_raw_header.

Here's how to reproduce:

local h = ngx.req.raw_header()

backtrace:

Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0817073f in ngx_http_lua_ngx_req_raw_header ()
(gdb) bt full
#0 0x0817073f in ngx_http_lua_ngx_req_raw_header ()
No symbol table info available.
#1 0xf76b23c5 in lj_BC_FUNCC () from /usr/local/openresty/luajit/lib/libluajit-5.1.so.2
No symbol table info available.
#2 0xf76c9f25 in lua_pcall (L=0xf69f41c0, nargs=0, nresults=1, errfunc=1) at lj_api.c:1129
g = 0xf69f41f0
oldh = 0 '\000'
ef =
status =
PRETTY_FUNCTION = "lua_pcall"
#3 0x0818c00c in ngx_http_lua_header_filter_by_chunk ()
No symbol table info available.
#4 0x0818c2b2 in ngx_http_lua_header_filter_file ()
No symbol table info available.
#5 0x0818bc83 in ngx_http_lua_header_filter ()
No symbol table info available.
#6 0x081b3da8 in ngx_http_headers_more_filter ()
No symbol table info available.
#7 0x082323b9 in ngx_http_subs_header_filter ()
No symbol table info available.
#8 0x081091ad in ngx_http_not_modified_header_filter ()
No symbol table info available.
#9 0x080cbbe3 in ngx_http_send_header ()
No symbol table info available.
#10 0x080cdc45 in ngx_http_send_special_response.isra ()
---Type to continue, or q to quit---
No symbol table info available.
#11 0x080ce013 in ngx_http_special_response_handler ()
No symbol table info available.
#12 0x080d1f93 in ngx_http_finalize_request ()
No symbol table info available.
#13 0x080d3aa7 in ngx_http_process_request_line ()
No symbol table info available.
#14 0x080a993b in ngx_event_process_posted ()
No symbol table info available.
#15 0x080a922b in ngx_process_events_and_timers ()
No symbol table info available.
#16 0x080b3468 in ngx_worker_process_cycle ()
No symbol table info available.
#17 0x080b1a81 in ngx_spawn_process ()
No symbol table info available.
#18 0x080b4f74 in ngx_master_process_cycle ()
No symbol table info available.
#19 0x0807f450 in main ()
No symbol table info available.

nginx version: openresty/1.13.6.1
built with OpenSSL 1.0.1t 3 May 2016

on debian jessie

I'm not sure which requests are causing this and if it's possible to find

[ghost]

Note that i'm using header_filter_by_lua_file in the http context to call the code

[ghost]

My full configure line

configure arguments: --prefix=/usr/local/openresty/nginx --with-debug --with-cc-opt='-DNGX_LUA_USE_ASSERT -DNGX_LUA_ABORT_AT_PANIC -O2' --add-module=../ngx_devel_kit-0.3.0 --add-module=../echo-nginx-module-0.61 --add-module=../xss-nginx-module-0.05 --add-module=../ngx_coolkit-0.2rc3 --add-module=../set-misc-nginx-module-0.31 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.07 --add-module=../srcache-nginx-module-0.31 --add-module=../ngx_lua-0.10.11 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.18 --add-module=../redis2-nginx-module-0.14 --add-module=../redis-nginx-module-0.3.7 --add-module=../rds-json-nginx-module-0.15 --add-module=../rds-csv-nginx-module-0.08 --add-module=../ngx_stream_lua-0.0.3 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -Wl,-rpath,/usr/local/lib' --with-http_sub_module --with-threads --with-file-aio --with-http_gunzip_module --with-http_ssl_module --with-ipv6 --with-http_stub_status_module --with-http_geoip_module --with-http_realip_module --error-log-path=/usr/local/nginx/logs/error.log --http-log-path=/usr/local/nginx/logs/access.log --conf-path=/usr/local/nginx/conf/nginx.conf --with-http_perl_module --with-stream --with-stream_ssl_module

I also tried the latest lua-nginx-module with the same result

[p0pr0ck5]

This sounds vaguely similar to an issue reported several years ago involving the ngx.req.raw_header API, also specifically on Debian Jessie (and using the system's PCRE): p0pr0ck5/lua-resty-waf#24

@Vladimir000 might you consider building OpenResty with PCRE statically, as a test?

[ghost]

@p0pr0ck5 I'm doing tests and i found out it happens with requests with an invalid request line (first line), eg A AAA AA instead of GET / ...
Even if i pass "false" to ngx.req.raw_header.

I will try what you proposed

[ghost]

Of course i move the code to the server context, no problem, as the request line is valid.

But in the doc it says

syntax: str = ngx.req.raw_header(no_request_line?)
context: set_by_lua*, rewrite_by_lua*, access_by_lua*, content_by_lua*, header_filter_by_lua*

and

header_filter_by_lua_file
syntax: header_filter_by_lua_file
context: http, server, location, location if

Possibly broken in the http context?

[p0pr0ck5]

Do you have an example request/packet capture that can reliably cause a segfault, as well as a minimal Nginx config?

[agentzh]

@Vladimir000 It seems like you are trying to call this Lua API function in an error_page handler for 400 responses for invalid requests. DO NOT DO THAT! A segfault would almost always be guaranteed, even when using standard nginx modules in such contexts. This is because nginx does not even fully initialize the request data structure for invalid client requests and when you are trying to do something complex with the request, you'll surely run into troubles and memory issues (actually segfault is the best thing you can get).

[ghost]

@agentzh can you be more precise, does it mean all ngx.req.* shouldn't be used in the http context?

[agentzh]

@Vladimir000 You should not do anything fancy, not even the standard proxy_pass or fastcgi_pass directives, inside the location serving the 400 error pages (not to mention any of those *_by_lua* directives).