GH-44188: [Python] Fix pandas roundtrip with bytes column names

[Piong1997]

Rationale for this change

There is a bug that when column dtype is np.bytes，it will goto the final branch and run level=level.astype(dtype)

Are these changes tested?

Yes

• GitHub Issue: [Python] Pandas conversion roundtrip fails with bytes column names #44188

[jorisvandenbossche]

@Piong1997 do you have a reproducible example that shows the bug? (it would be good to add a test for it)

[Piong1997]

@Piong1997 do you have a reproducible example that shows the bug? (it would be good to add a test for it)
@jorisvandenbossche
Sure, the following code reproduce this bug:

import pandas as pd # pd.__version__ is 2.0.3
import pyarrow # pyarrow.__version__ is 17.0.0

df = pd.DataFrame([[0.1, 0.2], [0.3, 0.4]], columns=[b'col1', b'col2'])
table = pyarrow.Table.from_pandas(df)
table.to_pandas() # error occurred

Note that this bug can only be reproduced in pandas 2.0 or above.
This bug was introduced in a certain version of pyarrow, pyarrow version 6.0 can work normally.

[jorisvandenbossche]

Thanks, that indeed errors right now. Could you add that as a test case in test_pandas.py ?

[Piong1997]

Thanks, that indeed errors right now. Could you add that as a test case in test_pandas.py ?

I have already added the case

[github-actions]

⚠️ GitHub issue #44188 has been automatically assigned in GitHub to PR creator.

[github-actions]

⚠️ GitHub issue #44188 has been automatically assigned in GitHub to PR creator.

[pitrou]

LGTM, thank you @Piong1997 ! I rebased and will merge if CI is green.

[pitrou]

@github-actions crossbow submit pandas

[github-actions]

Revision: e619630

Submitted crossbow builds: ursacomputing/crossbow @ actions-690c1f2da2

Task	Status
test-conda-python-3.10-pandas-latest-numpy-latest
test-conda-python-3.11-pandas-latest-numpy-1.26
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-pandas-nightly-numpy-nightly
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit e2ac52d.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.

[kou]

@github-actions crossbow submit wheel-windows-cp310-cp310-amd64

[github-actions]

Revision: e619630

Submitted crossbow builds: ursacomputing/crossbow @ actions-a92b6e4575

Task	Status
wheel-windows-cp310-cp310-amd64