Performance Regression for BigFloat and BigInt due to #51362

[Zentrik]

There seem to have been quite a few regressions recently on the scalar BaseBenchmarks, the real regressions seem to be due to #51362. The regressions seem to indicate #51362 had a bigger performance impact than intended, at least from what I gather reading the comments to that pr.

Full regression list is here, I've listed a couple of the biggest ones below

Benchmark	% Change
scalar.arithmetic.(mul, Complex{BigFloat}, Complex{UInt64})	50.08
scalar.arithmetic.(mul, Complex{BigInt}, Complex{BigFloat})	49.82
scalar.arithmetic.(mul, UInt64, Complex{BigFloat})	49.76
scalar.arithmetic.(mul, Complex{BigFloat}, Complex{BigInt})	49.61
scalar.arithmetic.(mul, Complex{BigFloat}, Complex{Int64})	49.47
scalar.arithmetic.(mul, UInt64, BigFloat)	49.33
scalar.arithmetic.(mul, BigFloat, Complex{Int64})	48.49
scalar.arithmetic.(sub, Complex{BigFloat}, Complex{BigInt})	47.95

[Zentrik]

There also seem to be a few regressions on the union array benchmarks involving BigFloats and BigInts, see here.

[nsajko]

I guess this should get better once #53584 is fixed.

[KristofferC]

Since the cause was a correctness fix I will drop the milestone here but ideally something like #53607 could come in and fix it.