Add ability to read double-quoted csvs to wr.catalog.create_csv_table()

[gballardin]

I am creating a table off some csv files in S3 that I generated in a previous step in Athena with aws wrangler. Athena by default double quotes its csv output.

When I create a table in the Glue catalog with wr.catalog.create_csv_table() on the double-quoted output data I mentioned above there is no way to pass that function the WITH SERDEPROPERTIES ('quoteChar' = '\"') parameter. As a results, my table exists (I can see it listed in my Athena tables), has the correct partitions, but returns nothing when I query it with Athena. This seems to be a conceptually similar problem to this other aws wrnagler issue where {'skip.header.line.count': "1"} was missing and was later (v. 1.7.0) added on to the wr.catalog.create_csv_table() signature as a one off change.

Can you please add the ability to pass some {key: value} pairs for the SERDEPROPERTIES to this function? That way this function would replicate what

ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
WITH SERDEPROPERTIES ('quoteChar' = '\\"')

does when creating a table with a CREATE TABLE statement in Athena.

[maxispeicher]

Hey @gballardin,

I've created a commit which allows to specify serde_library and serde_parameters for wr.catalog.create_csv_table. Please note that in the current implementation serde_parameters overrides the default parameters which means that you always need to speficy all desired serde properties.
Example:

import awswrangler as wr

wr.catalog.create_csv_table(
    database=glue_database,
    table=glue_table,
    path=path,
    columns_types={"c0": "string", "c1": "string", "c2": "string"},
    serde_library="org.apache.hadoop.hive.serde2.OpenCSVSerde",
    serde_parameters={"separatorChar": ",", "quoteChar": '"', "escapeChar": "\\"},
)

If you want to test it you can install it with the following commands 🙂

pip uninstall awswrangler -y
pip install git+https://github.com/maxispeicher/aws-data-wrangler.git@add_possibility_to_change_serde_info

[gballardin]

That is great! Thank you so much for implementing my feature request so quickly.

[jaidisido]

Now covered in 2.8.0.