Skip to content

Quit early on error for tbl/csv #84

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 29, 2025

Conversation

alamb
Copy link
Collaborator

@alamb alamb commented Mar 29, 2025

If there is an error writing (like out of disk space) when writing to csv, the program generates much more data and discards it rather than stopping early

Let's stop early so tpchgen-cli can report the error quick and concisely

[2025-03-29T19:32:49Z DEBUG tpchgen_cli] Logging configured from environment variables
[2025-03-29T19:32:49Z DEBUG tpchgen_cli] Creating distributions and text pool
[2025-03-29T19:32:50Z INFO  tpchgen_cli] Created static distributions and text pools in 1.030596141s
[2025-03-29T19:32:50Z INFO  tpchgen_cli] Writing table lineitem (SF=100) to lineitem.tbl
[2025-03-29T19:32:50Z DEBUG tpchgen_cli] Generating 4883 parts in total
[2025-03-29T19:32:51Z DEBUG tpchgen_cli::generate] Using 22 threads
[2025-03-29T19:33:13Z DEBUG tpchgen_cli::generate] Error sending buffer to writer: channel closed
[2025-03-29T19:33:13Z DEBUG tpchgen_cli::generate] Error sending buffer to writer: channel closed
[2025-03-29T19:33:13Z INFO  tpchgen_cli::statistics] Created 3.32 GB in 22.024473445s (0.15 GB/sec)
[2025-03-29T19:33:13Z DEBUG tpchgen_cli::statistics] Wrote 3567903086 bytes in 221 buffers  15.40 MB/buffers
[2025-03-29T19:33:13Z DEBUG tpchgen_cli::generate] Error sending buffer to writer: channel closed
[2025-03-29T19:33:13Z DEBUG tpchgen_cli::generate] Error sending buffer to writer: channel closed
.... (many more) ....
[2025-03-29T19:35:59Z DEBUG tpchgen_cli::generate] Error sending buffer to writer: channel closed
[2025-03-29T19:35:59Z DEBUG tpchgen_cli::generate] Error sending buffer to writer: channel closed
[2025-03-29T19:35:59Z DEBUG tpchgen_cli::generate] Error sending buffer to writer: channel closed
[2025-03-29T19:35:59Z DEBUG tpchgen_cli::generate] Error sending buffer to writer: channel closed
[2025-03-29T19:35:59Z DEBUG tpchgen_cli::generate] Error sending buffer to writer: channel closed
[2025-03-29T19:35:59Z DEBUG tpchgen_cli::generate] waiting for writer task to complete
Error: Os { code: 28, kind: StorageFull, message: "No space left on device" }

With this PR:

alamb@aal-perf:~$ RUST_LOG=debug tpchgen-cli --tables=line-item -s 100   --tables=line-item
[2025-03-29T19:41:56Z DEBUG tpchgen_cli] Logging configured from environment variables
[2025-03-29T19:41:56Z DEBUG tpchgen_cli] Creating distributions and text pool
[2025-03-29T19:41:57Z INFO  tpchgen_cli] Created static distributions and text pools in 1.03640647s
[2025-03-29T19:41:57Z INFO  tpchgen_cli] Writing table lineitem (SF=100) to lineitem.tbl
[2025-03-29T19:41:57Z DEBUG tpchgen_cli] Generating 4883 parts in total
[2025-03-29T19:41:57Z DEBUG tpchgen_cli::generate] Using 22 threads
[2025-03-29T19:42:20Z DEBUG tpchgen_cli::generate] Error sending buffer to writer: channel closed
[2025-03-29T19:42:20Z DEBUG tpchgen_cli::generate] Error sending buffer to writer: channel closed
[2025-03-29T19:42:20Z INFO  tpchgen_cli::statistics] Created 3.32 GB in 22.478921973s (0.15 GB/sec)
[2025-03-29T19:42:20Z DEBUG tpchgen_cli::statistics] Wrote 3567903086 bytes in 221 buffers  15.40 MB/buffers
[2025-03-29T19:42:20Z DEBUG tpchgen_cli::generate] writer task is done early, stopping writer
[2025-03-29T19:42:20Z DEBUG tpchgen_cli::generate] waiting for writer task to complete
Error: Os { code: 28, kind: StorageFull, message: "No space left on device" }

@alamb alamb marked this pull request as ready for review March 29, 2025 19:42
@clflushopt clflushopt merged commit cb2108c into clflushopt:main Mar 29, 2025
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants