Spark: Avoid closing deserialized copies of shared resources like FileIO

[xiaoxuandev]

This change prevents calling close() on FileIO during the cleanup of Spark's broadcast variable when memoryStore.remove(blockId) is called. Closing FileIO can unintentionally shut down shared resources—such as a shared connection pool—when S3FileIO is backed by the Apache HTTPClient. Calling close() will trigger the shutdown of the HttpClientConnectionManager, leading to request failures if other instances are still in use.
Fixes: #12858, #12046

[bk-mz]

👍

[singhpk234]

is there a way to ensure that the io and hence the pool is eventually closed ?

[xiaoxuandev]

Thanks for the review! Based on code inspection and some local debugging, Iceberg doesn’t explicitly call close() on most FileIO instances (i.e., the regular, non-deserialized ones). The exception is S3FileIO, which overrides finalize() to invoke close() during garbage collection—and this applies to deserialized copies as well.
And given that the underlying connection pool maybe shared, we might want to consider removing finalize() from S3FileIO to avoid unintended side effects during garbage collection. cc: @rdblue @aokolnychyi

Side note: finalize() was deprecated in Java 9 due to potential performance issues, deadlocks, and unpredictable behavior during GC.

[mgmarino]

Thanks, @xiaoxuandev this looks to work similarly to the PR that I opened to fix this issue here: #12129, but I was unable to come up with tests there. Would love to get this in.