TPC-DS Q44 regression for the " Mixed size objects" dataset

[oleg-lvovitch-aws]

Analytics Accelerator Library for Amazon S3 Version

0.0.1

AWS Region

us-east-1, though it applies broadly.

Describe the running environment

TPC DS benchmarking

What happened?

Q44 shows 8% regression by comparison with the baseline.

Relevant log output

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[oleg-lvovitch-aws]

The impact is particularly visible for large objects (>1GB) or large datasets, for highly selective queries that rely on column dictionary data instead of scanning column values.

For example, for queries that have a specific predicate match (eg: ss_customer_sk=12345), it is required to only read Parquet column dictionaries, and the actual column data is not required. The column data will only be required if the predicate matches, which for very specific predicates as above, does not happen that often.

Currently, we are making an assumption that if a column is getting read, then it's likely that most of it will be read as well and pre-fetch the entire column. For queries that rely almost exclusively on dictionaries, this results in potentially large over-read.

[ahmarsuhail]

This has now been resolved with #189, and released with v0.0.2 https://github.com/awslabs/analytics-accelerator-s3/releases/tag/v0.0.2