[Grid] Feature Request: Request Access Logs

[dylanlive]

🚀 Feature Proposal

For all Grid Components (Hub, Node, Router, etc.) it would be beneficial to allow storing request access logs to assist in investigating unauthorized access. Per recommendation of security engineers, these access logs should include (apache combined log format & X-Forwarded-For):

• Client IP
• Timestamp
• HTTP Method
• Path
• HTTP Response Code
• User agent
• Number of byes returned to the client
• Referer
• X-Forwarded-For

These can either be:

1. Stored within a file
2. Embedded within the OpenTelemetry Data (though I'd need to research how to export it out and into its own dedicated file)

I noticed OpenTelemetry already has some of the data (method, path, status code, IP although unsure if that's the client or the host machine's ip).

If Grid utilized Reactor Netty, access logs likely could be easily enabled.

My search for "netty access logs" only show Reactor Netty, which is why I concluded it's not possible to do with Netty out of the box.

Motivation

Security Engineers may ask for services to log requests. While it's possible to put Nginx (with logs) in front of Grid Hub or Router, the requests to Grid Node are not logged. It's important to see what kinds of requests were made across all nodes, in the event there are unauthorized requests.

In Se3 Grid, it was possible to put Nginx in front of a Node, and launch the node with -remoteHost "http://NODE_IP:80" (port 80 so that the hub would pass requests through nginx, and nginx forward to 5555). The -remoteHost flag appears to have been removed in Se4 Alpha.

Example

1. Launch java -jar selenium-server-4.0.0-alpha-7.jar hub --access-log-file access.log or within the config something like:

[logging]
# Configure logging
# Type: boolean
enable = true
# Store Access Logs within a file
# Type: string
access_log_file = access.log

2. Make a request to http://localhost:4444/status
3. Within access.log a line like:
   127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /status HTTP/1.1" 200 2326 "http://localhost:4444/status" "selenium/3.141.59 (java unix)"

[dylanlive]

I am empathetic that Selenium isn't really aimed to be an http server. I was hoping its http server backend, Netty, would achieve my goals of access logging.

Perhaps an alternative solution would be to reintroduce the remoteHost functionality.

I spent some time looking into if my Se3 solution via -remoteHost & Nginx was possible in Se4. This was achievable by:

1. Running Node on Port 5555
2. Running Nginx (with required logging) on Port 80, with it proxy forwarding requests to port 5555
3. Setting remoteHost to <ip>:80, so that the node would register itself to the hub as port 80

I'm finding no evidence in the codebase that you can run Selenium on a particular Port, but report to the Hub a different port.

• selenium/java/server/src/org/openqa/selenium/grid/node/httpd/NodeServer.java

  Line 119 in 337e431

  LOG.info("Reporting self as: " + serverOptions.getExternalUri());

• selenium/java/server/src/org/openqa/selenium/grid/server/BaseServerOptions.java

  Line 88 in 337e431

  int port = getPort();

• selenium/java/server/src/org/openqa/selenium/grid/server/BaseServerOptions.java

  Lines 51 to 52 in 337e431

  	int newPort = config.getInt(SERVER_SECTION, "port")
  	.orElseGet(PortProber::findFreePort);

[pujagani]

Thank you for providing the feature details in such great depth. I understand the need for the requested feature. We are leveraging Open-Telemetry to provide event-logs. More details are described here . The first iteration of adding event-logs added the mandatory fields for HTTP request as per the OpenTelemetry specification. The fields that are missing (and needed is request logs) are also a part of OpenTelemetry specification. As a result, adding those fields seemed like a viable solution. It would address the request logs feature and strengthen the HTTP event-logs context.

The changes are made as part of #8902 . It includes all the fields except Referer (which might not be the valid field in case of Selenium Grid request) and response content-length (the way to do this currently does not seem efficient, since it involves writing all the bytes in memory and counting the length twice).

The logs can be written to a file using "--logs " flag while starting the Grid in Standalone mode or in a fully-distributed mode.

Example:
java -jar /Users/Puja/Projects/repos/selenium/bazel-bin/java/server/src/org/openqa/selenium/grid/selenium_server_deploy.jar standalone --log /Users/Puja/Desktop/LOG.log

[dylanlive]

@pujagani Thank you! Much appreciated! I'll have a look

[dylanlive]

It looks like this is still missing on some requests, such as GET /status

10:09:16.746 INFO [LoggingOptions$1.lambda$export$0] - {"traceId": "939b55d7ff75a295488e96b0664d4f5a","spanId": "5b485f0b25a8cf55","spanKind": "INTERNAL","eventTime": 1606327756746300046,"eventName": "Computed grid status","attributes": {"grid.status": true,"http.status_code": 200,"logger": "org.openqa.selenium.grid.router.GridStatusHandler"}}

I'll look to see the effort to add that in. Still curious if moving Grid to Reactor Netty is a viable alternative.

[pujagani]

Thank you for pointing that out. I am fixing that and re-checking to be sure in-case some other requests were missed out. The main aim is to log any incoming requests from the client.

However, I am afraid that request access logs would be the driving force to move to the Grid Reactor Netty for the next release. I have raised this to the maintainers to get a definitive answer about the future plans to move to Reactory Netty server.