`to_parquet` with `dtype` mutates input DataFrame

[eferm]

Describe the bug

Hi! When writing with to_parquet my input DataFrame seems to mutate in-place when passing the dtype parameter. Perhaps more specifically, the DataFrame changes in-place to the Pandas-equivalent nullable dtypes described in dtype.

My expectation is that the dtype casts would only be applied towards serialized data, and not that it would have side effects on my input DataFrame.

However, if this is expected behavior perhaps docs could be clarified?

To Reproduce

My environment:

$ python --version
Python 3.8.9
$ pip freeze | grep awswrangler
awswrangler==2.6.0

Simple repro:

>>> import awswrangler as wr
>>> import pandas as pd
>>> df = pd.DataFrame([1, 2, 3], columns=["a"])
>>> df.dtypes
a    int64
dtype: object
>>> wr.s3.to_parquet(df, "s3://bucket/path/to/file.parquet", dtype={"a": "bigint"})
{'paths': ['s3://bucket/path/to/file.parquet'], 'partitions_values': {}}
>>> df.dtypes
a    Int64
dtype: object

My work-around is to pass in a copy of the DataFrame:

>>> wr.s3.to_parquet(df.copy(), "s3://bucket/path/to/file.parquet", dtype={"a": "bigint"})
{'paths': ['s3://bucket/path/to/file.parquet'], 'partitions_values': {}}
>>> df.dtypes
a    int64
dtype: object

[jaidisido]

Thanks for raising this. I agree that the input dataframe should not be mutated in place. The culprit is this method which works directly on the dataframe instead of a copy. Passing a copy seems to resolve the issue:

df = _apply_dtype(df=df.copy(), dtype=dtype, catalog_table_input=catalog_table_input, mode=mode)

Will provide a fix but first need to check that there would not be an impact on performance from the deep copy operation.

[jaidisido]

After review, I am afraid the negative impact on memory of making a deep data frame copy (i.e doubling memory size) outweighs the benefit of having an immutable data frame returned. So will clarify the docs instead to make it clearer to the user as you suggested

[eferm]

Got it, thanks for investigating @jaidisido ! Clarifying in docs and letting the user do df.copy() when needed is fine.

[jaidisido]

Docs updated in latest release 2.8.0. Thanks again