Client-side performance overhead

[jgronfur]

We are trying to add the aws-advanced-jdbc-wrapper to our client database connection configuration so that we get the benefits of IAM authentication and fast failover.
However our tests at production scale show that the aws-advanced-jdbc-wrapper adds 50+% additional latency and client-side cpu utilization under the same workload. Even when disabling all plugins, the performance impact is still there (although slightly less).

In particular the performance overhead seems to be when iterating through a ResultSet to get the values of the columns for each row. It seems that it is doing the wrapper logic for each plugin on every call for every column of every row.

Our tech stack is:

• Java 21
• Aurora MySQL Compatible
• HikariCP
• Maria Connector/J (used to avoid GPL)
• aws-advanced-jdbc-wrapper

Our database config is:

app.datasource.username=REDACTED
app.datasource.hostname=REDACTED
app.datasource.url=jdbc:aws-wrapper:mariadb://${app.datasource.hostname}:3306/REDACTED
app.datasource.driverClassName=software.amazon.jdbc.Driver
app.datasource.configuration.poolName=REDACTED
app.datasource.configuration.minimumIdle=2
app.datasource.configuration.maximumPoolSize=32
app.datasource.configuration.exception-override-class-name=software.amazon.jdbc.util.HikariCPSQLException
app.datasource.configuration.data-source-properties.connectionCollation=utf8mb4
app.datasource.configuration.data-source-properties.sslMode=trust
app.datasource.configuration.data-source-properties.wrapperPlugins=auroraConnectionTracker,failover,efm2,iam
app.datasource.configuration.data-source-properties.wrapperDialect=aurora-mysql

[davecramer]

So with no plugins you are seeing latency? What are the actual measurements?

[jgronfur]

I wrote a simple application that uses the same tech stack as above that does 50,000 queries to a table that has 10 columns and 1000 rows. It iterates through every row and field and gets the value of each and does a wasNull check.

Tests were done on a 2 cpu instance in AWS in the same AZ as the Aurora-MySQL instance.

Standard MariaDB Driver

• process cpu utilization 35%
• 1 min load average .34
• 5 min load average .27
• Run time: 53900
• Run time: 51994
• Average 1.05894 ms per query

Wrapper, no plugins

• process cpu utilization 86%
• 1 min load average .82
• 5 min load average .86
• Run time: 264536
• Run time: 261229
• Average 5.25765 ms per query

Wrapper, default plugins, no iam authentication

• process cpu utilization 90%
• 1 min load average .75
• 5 min load average .78
• Run time: 356232
• Run time: 352008
• Average 7.0824 ms per query

Wrapper driver, default plugins, with iam authentication

• process cpu utilization 90%
• 1 min load average 1.00
• 5 min load average 0.97
• Run time: 355865
• Run time: 352444
• Average 7.08309 ms per query

The wrapper, without any plugins, increases client cpu utilization by the process by almost 150% and the per query time by almost 400%. Adding plugins, including IAM authentication, into the mix only makes the performance worse. I didn't do any multi-threaded performance testing here, but previous attempts have shown that performance is even worse due to cpu resource contention.

Everything that we are doing is relatively standard, except that we have a proprietary ResultSet wrapper that converts them to a ByteBuffer backed implementation to allow easy caching/serialization/etc... of ResultSets and the ResultSet/Statement/Connection resources can be freed as soon as possible. The standard MariaDB driver test included this same wrapper so the performance overhead is not related to that. I will work on a second test application that tries again without the proprietary code.

Stack trace samples taken during the tests reveal that most of them were in the process of iterating through the raw JDBC ResultSet (25 out of 30 traces, 3 were SSL/networking related, 2 were query preparation). Samples of the iterating traces resemble the following, even when no plugins are specified:

"main" #1 [27875] prio=5 os_prio=0 cpu=128627.50ms elapsed=151.02s tid=0x00007f003402c140 nid=27875 runnable [0x00007f003bb09000]
java.lang.Thread.State: RUNNABLE
at software.amazon.jdbc.plugin.DefaultConnectionPlugin.execute(DefaultConnectionPlugin.java:128)
at software.amazon.jdbc.ConnectionPluginManager.lambda$execute$5(ConnectionPluginManager.java:348)
at software.amazon.jdbc.ConnectionPluginManager$$Lambda/0x00007effc03523e0.call(Unknown Source)
at software.amazon.jdbc.ConnectionPluginManager.lambda$null$0(ConnectionPluginManager.java:270)
at software.amazon.jdbc.ConnectionPluginManager$$Lambda/0x00007effc028b178.call(Unknown Source)
at software.amazon.jdbc.ConnectionPluginManager.executeWithTelemetry(ConnectionPluginManager.java:246)
at software.amazon.jdbc.ConnectionPluginManager.lambda$makePluginChainFunc$1(ConnectionPluginManager.java:270)
at software.amazon.jdbc.ConnectionPluginManager$$Lambda/0x00007effc028af60.call(Unknown Source)
at software.amazon.jdbc.ConnectionPluginManager.executeWithSubscribedPlugins(ConnectionPluginManager.java:236)
at software.amazon.jdbc.ConnectionPluginManager.execute(ConnectionPluginManager.java:345)
at software.amazon.jdbc.util.WrapperUtils.executeWithPlugins(WrapperUtils.java:244)
at software.amazon.jdbc.wrapper.ResultSetWrapper.wasNull(ResultSetWrapper.java:2389)
at com.zaxxer.hikari.pool.HikariProxyResultSet.wasNull(HikariProxyResultSet.java)

[jgronfur]

Ran the test without the proprietary resultset implementation:

Standard driver, no proprietary resultset

• cpu utilization 30%
• 1 min load average .24
• 5 min load average .22
• Run time: 49784
• Run time: 49353
• Average 0.98137 ms per query

Wrapper driver, default plugins, with iam authentication, No proprietary resultset

• cpu utilization 88%
• 1 min load average .85
• 5 min load average .78
• Run time: 312872
• Run time: 311065
• Average 6.23937 ms per query

So our proprietary resultset implementation does have a performance impact, but not nearly as much as the aws wrapper.

[jgronfur]

Any guidance or update on this performance issue? @davecramer

[sergiyvamz]

Hi @jgronfur

Any chance you can share your sample application? We'd like to run it on our side and find a root cause of the issue. Thank you!

[jgronfur]

jdbc-wrapper-test.zip
Attached is a zip file of the stripped down test. The README.md file should cover everything that is needed for the test

[jgronfur]

Any update on this?

[sergiyvamz]

@jgronfur I expect some results next week.

[sergiyvamz]

Hi @jgronfur

We've successfully merged a significant optimization into our codebase. Would you please test our latest snapshot build and provide feedback on whether you observe performance improvements? Your assessment would be greatly appreciated.

https://github.com/aws/aws-advanced-jdbc-wrapper/blob/main/docs/using-the-jdbc-driver/UsingTheJdbcDriver.md#using-a-snapshot-of-the-driver
https://aws.oss.sonatype.org/content/repositories/snapshots/software/amazon/jdbc/aws-advanced-jdbc-wrapper/2.6.1-SNAPSHOT/aws-advanced-jdbc-wrapper-2.6.1-20250617.215704-9.jar (or any later builds)

Thank you!

[jgronfur]

That snapshot has amazing performance improvements!
I didn't have the exact same Aurora cluster and EC2 instance to test from so I reran both the direct jdbc and the wrapper test:

Standard driver, no proprietary resultset

• cpu utilization 27%
• 1 min load average .15
• 5 min load average .15
• Run time: 62246
• Run time: 62420
• Average 1.25 ms per query

Wrapper driver, default plugins, with iam authentication, No proprietary resultset

• cpu utilization 32%
• 1 min load average .18
• 5 min load average .18
• Run time: 66936
• Run time: 66312
• Average 1.33 ms per query

So the observed performance impact has been reduced from 500+% to around 6%! Thank you so much for your work on this!