[SPARK-52459][SQL] Fix a subtle thread-safety issue with SQLAppStatusListener

[huangxiaopingRD]

What changes were proposed in this pull request?

Fix a subtle thread-safety issue with SQLAppStatusListener.
In our scenario, a deadlock occurred in the HashMap in SQLAppStatusListener. The thread stack is as follows:

scala.runtime.BoxesRunTime.equalsNumObject(BoxesRunTime.java:142)
scala.runtime.BoxesRunTime.equals2(BoxesRunTime.java:131)
scala.runtime.BoxesRunTime.equals(BoxesRunTime.java:123)
scala.collection.mutable.HashTable.elemEquals(HashTable.scala:365)
scala.collection.mutable.HashTable.elemEquals$(HashTable.scala:365)
scala.collection.mutable.HashMap.elemEquals(HashMap.scala:44)
scala.collection.mutable.HashTable.findEntry0(HashTable.scala:140)
scala.collection.mutable.HashTable.findOrAddEntry(HashTable.scala:169)
scala.collection.mutable.HashTable.findOrAddEntry$(HashTable.scala:167)
scala.collection.mutable.HashMap.findOrAddEntry(HashMap.scala:44)
scala.collection.mutable.HashMap.put(HashMap.scala:126)
scala.collection.mutable.HashMap.update(HashMap.scala:131)
org.apache.spark.sql.execution.ui.SQLAppStatusListener.$anonfun$aggregateMetrics$17(SQLAppStatusListener.scala:289)
org.apache.spark.sql.execution.ui.SQLAppStatusListener.$anonfun$aggregateMetrics$17$adapted(SQLAppStatusListener.scala:273)
org.apache.spark.sql.execution.ui.SQLAppStatusListener$$Lambda$2571/551075293.apply(Unknown Source)
scala.collection.immutable.List.foreach(List.scala:431)
org.apache.spark.sql.execution.ui.SQLAppStatusListener.aggregateMetrics(SQLAppStatusListener.scala:273)
org.apache.spark.sql.execution.ui.SQLAppStatusListener.$anonfun$new$3(SQLAppStatusListener.scala:73)
org.apache.spark.sql.execution.ui.SQLAppStatusListener.$anonfun$new$3$adapted(SQLAppStatusListener.scala:69)
org.apache.spark.sql.execution.ui.SQLAppStatusListener$$Lambda$2643/1155315027.apply(Unknown Source)
scala.collection.Iterator.foreach(Iterator.scala:943)
scala.collection.Iterator.foreach$(Iterator.scala:943)
scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
scala.collection.IterableLike.foreach(IterableLike.scala:74)
scala.collection.IterableLike.foreach$(IterableLike.scala:73)
scala.collection.AbstractIterable.foreach(Iterable.scala:56)
org.apache.spark.sql.execution.ui.SQLAppStatusListener.$anonfun$new$2(SQLAppStatusListener.scala:69)
org.apache.spark.sql.execution.ui.SQLAppStatusListener$$Lambda$2138/1048529349.apply$mcV$sp(Unknown Source)
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
org.apache.spark.util.Utils$.tryLog(Utils.scala:2102)
org.apache.spark.status.ElementTrackingStore.$anonfun$close$1(ElementTrackingStore.scala:200)
org.apache.spark.status.ElementTrackingStore$$Lambda$2641/427051673.apply(Unknown Source)
scala.collection.immutable.List.foreach(List.scala:431)
scala.collection.generic.TraversableForwarder.foreach(TraversableForwarder.scala:38)
scala.collection.generic.TraversableForwarder.foreach$(TraversableForwarder.scala:38)
scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:47)
org.apache.spark.status.ElementTrackingStore.close(ElementTrackingStore.scala:199) => holding Monitor(org.apache.spark.status.ElementTrackingStore@2073989974})

The flame graph is as follows：

Why are the changes needed?

Fixed the problem that SQLAppStatusListener aggregation metrics always got stuck

Does this PR introduce any user-facing change?

No

Was this patch authored or co-authored using generative AI tooling?

No