feat: Enable `rustflags = "-C force-frame-pointers=yes"` default for opendal release process

[Zheaoli]

For now, if we want to use the perf or any performance analysis tool to analysis the performance of the opendal process. We will be in trouble because of the symbol issue

For example, I try to use perf to analysis the following code

import opendal
import uuid

op = opendal.Operator("fs", root="/tmp")

uuid_str=str(uuid.uuid4)

for _ in range(10000000):
    op.write(f"/{uuid_str}", [0]*64*1024*1024)
    op.read(f"/{uuid_str}")
    op.delete(f"/{uuid_str}")

The perf record is here

+   63.46%     0.00%  python   [unknown]                                 [.] 0x8b48fb894828ec83
+   43.07%    13.11%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] <&pyo3::types::iterator::PyIterator a
+   14.66%     6.43%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] pyo3::conversions::std::num::<impl py
+   13.72%    13.72%  python   libpython3.12.so.1.0                      [.] listiter_next
+   12.49%     0.00%  python   [unknown]                                 [.] 0000000000000000
+   11.24%     0.00%  python   [kernel.kallsyms]                         [k] asm_exc_page_fault
+   11.23%     0.00%  python   [kernel.kallsyms]                         [k] exc_page_fault
+   11.22%     4.01%  python   [kernel.kallsyms]                         [k] do_user_addr_fault
+   11.06%    11.06%  python   ld-linux-x86-64.so.2                      [.] _dl_update_slotinfo
+   10.81%     0.00%  python   libc.so.6                                 [.] __libc_start_call_main
+   10.81%     0.00%  python   libpython3.12.so.1.0                      [.] Py_BytesMain
+   10.81%     0.00%  python   libpython3.12.so.1.0                      [.] Py_RunMain
+   10.79%     0.00%  python   libpython3.12.so.1.0                      [.] pymain_run_python.constprop.0
+   10.79%     0.00%  python   libpython3.12.so.1.0                      [.] _PyRun_AnyFileObject
+   10.79%     0.00%  python   libpython3.12.so.1.0                      [.] _PyRun_SimpleFileObject
+   10.79%     0.00%  python   libpython3.12.so.1.0                      [.] run_mod
+   10.79%     0.00%  python   libpython3.12.so.1.0                      [.] run_eval_code_obj
+   10.79%     0.00%  python   libpython3.12.so.1.0                      [.] PyEval_EvalCode
+   10.79%     0.00%  python   [JIT] tid 210049                          [.] py::<module>:/root/opendal-demo/demo.
+   10.79%     0.00%  python   libpython3.12.so.1.0                      [.] _PyEval_EvalFrameDefault
+    8.14%     8.14%  python   ld-linux-x86-64.so.2                      [.] __tls_get_addr
+    6.88%     1.97%  python   libc.so.6                                 [.] __memmove_avx_unaligned_erms
+    6.81%     0.11%  python   [kernel.kallsyms]                         [k] handle_mm_fault
+    6.64%     6.64%  python   ld-linux-x86-64.so.2                      [.] update_get_addr
+    6.53%     0.23%  python   [kernel.kallsyms]                         [k] __handle_mm_fault
+    6.27%     0.03%  python   [kernel.kallsyms]                         [k] handle_pte_fault
+    5.17%     0.00%  python   [unknown]                                 [.] 0x0000000000000098

We can find that we can not trace the rust code.

Since I enable the rustflag, here's new perf result

+   92.23%     0.00%  python   libc.so.6                                 [.] __libc_start_call_main              ◆
+   92.23%     0.00%  python   libpython3.12.so.1.0                      [.] Py_BytesMain                        ▒
+   92.23%     0.00%  python   libpython3.12.so.1.0                      [.] Py_RunMain                          ▒
+   92.22%     0.00%  python   libpython3.12.so.1.0                      [.] pymain_run_python.constprop.0       ▒
+   92.22%     0.00%  python   libpython3.12.so.1.0                      [.] _PyRun_AnyFileObject                ▒
+   92.22%     0.00%  python   libpython3.12.so.1.0                      [.] _PyRun_SimpleFileObject             ▒
+   92.22%     0.00%  python   libpython3.12.so.1.0                      [.] run_mod                             ▒
+   92.22%     0.00%  python   libpython3.12.so.1.0                      [.] run_eval_code_obj                   ▒
+   92.22%     0.00%  python   libpython3.12.so.1.0                      [.] PyEval_EvalCode                     ▒
+   92.22%     0.00%  python   [JIT] tid 211989                          [.] py::<module>:/root/opendal-demo/demo▒
+   92.22%     0.00%  python   libpython3.12.so.1.0                      [.] _PyEval_EvalFrameDefault            ▒
+   88.98%     0.00%  python   libpython3.12.so.1.0                      [.] PyObject_Vectorcall                 ▒
+   88.98%     0.00%  python   libpython3.12.so.1.0                      [.] method_vectorcall_VARARGS_KEYWORDS  ▒
+   88.98%     0.00%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] opendal_python::operator::_::<impl p▒
+   88.98%     0.00%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] pyo3::impl_::trampoline::cfunction_w▒
+   88.98%     0.00%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] pyo3::impl_::trampoline::trampoline ▒
+   83.73%     0.00%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] std::panic::catch_unwind            ▒
+   83.73%     0.00%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] std::panicking::try                 ▒
+   83.73%     0.00%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] __rust_try                          ▒
+   83.73%     0.00%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] std::panicking::try::do_call        ▒
+   83.73%     0.00%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] pyo3::impl_::trampoline::trampoline:▒
+   83.73%     0.00%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] pyo3::impl_::trampoline::cfunction_w▒
+   83.73%     0.00%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] opendal_python::operator::_::<impl o▒
+   83.22%     0.00%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] pyo3::impl_::extract_argument::extra▒
+   83.22%     0.00%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] <T as pyo3::impl_::extract_argument:▒
+   83.22%     0.00%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] pyo3::types::any::PyAny::extract    ▒
+   83.22%     0.00%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] pyo3::types::sequence::<impl pyo3::c▒
+   81.47%     7.67%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] pyo3::types::sequence::extract_seque▒
+   50.67%     2.93%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] <&pyo3::types::iterator::PyIterator ▒
+   41.45%     0.43%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] pyo3::marker::Python::from_owned_ptr▒
+   40.37%     2.06%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] <T as pyo3::conversion::FromPyPointe▒
+   36.33%     1.75%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] pyo3::gil::register_owned           ▒
+   24.69%     3.91%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] std::thread::local::LocalKey<T>::try▒
+   16.31%     0.34%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] pyo3::types::any::PyAny::extract    ▒
+   13.54%     3.58%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] pyo3::conversions::std::num::<impl p▒
+   10.77%     1.33%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] pyo3::gil::register_owned::{{closure▒
+    8.00%     4.11%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] pyo3::gil::OWNED_OBJECTS::__getit   ▒
+    7.77%     3.46%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] std::thread::local::LocalKey<T>::try▒
+    5.24%     0.00%  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] core::ptr::drop_in_place<pyo3::gil::▒
+    4.94%     4.94%  python   ld-linux-x86-64.so.2                      [.] _dl_update_slotinfo        
....
     0.05%     0.00%             0  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] opendal_python::operator::_::<impl opendal_python::operator::Operator>::__pymethod_delete__
     0.05%     0.00%             0  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] opendal_python::operator::Operator::delete
     0.05%     0.00%             0  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] opendal::types::operator::blocking_operator::BlockingOperator::delete
     0.05%     0.00%             0  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] opendal::types::operator::operator_functions::FunctionDelete::call
     0.05%     0.00%             0  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] opendal::types::operator::operator_functions::OperatorFunction<T,R>::call
     0.05%     0.00%             0  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] core::ops::function::FnOnce::call_once
     0.05%     0.00%             0  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] opendal::types::operator::blocking_operator::BlockingOperator::delete_with::{{closure}}
     0.05%     0.00%             0  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] <alloc::sync::Arc<T> as opendal::raw::accessor::Accessor>::blocking_delete
     0.05%     0.00%             0  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] opendal::raw::layer::<impl opendal::raw::accessor::Accessor for L>::blocking_delete
     0.05%     0.00%             0  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] opendal::raw::layer::LayeredAccessor::blocking_delete
     0.05%     0.00%             6  python   [kernel.kallsyms]                         [k] ext4_block_write_begin
     0.05%     0.00%             0  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] opendal::raw::layer::<impl opendal::raw::accessor::Accessor for L>::blocking_delete
     0.05%     0.00%             0  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] <opendal::layers::complete::CompleteAccessor<A> as opendal::raw::layer::LayeredAccessor>::blocking_delete
     0.05%     0.00%             0  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] opendal::raw::layer::<impl opendal::raw::accessor::Accessor for L>::blocking_delete
     0.05%     0.00%             0  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] <opendal::layers::error_context::ErrorContextAccessor<A> as opendal::raw::layer::LayeredAccessor>::blocking_delete
     0.05%     0.00%             0  python   _opendal.cpython-312-x86_64-linux-gnu.so  [.] <opendal::services::fs::backend::FsBackend as opendal::raw::accessor::Accessor>::blocking_delete

And

[Xuanwo]

Users interested in benchmarking can compile the OpenDAL Python bindings themselves. Is there any benefit to enabling this by default?

[Zheaoli]

Users interested in benchmarking can compile the OpenDAL Python bindings themselves. Is there any benefit to enabling this by default?

I can use the perf directly when it is enabled by default. This would be great when we find the process issue. We perf it and don't need to recompile the whole package

[Xuanwo]

I can use the perf directly when it is enabled by default. This would be great when we find the process issue. We perf it and don't need to recompile the whole package

perf is not a normal use cases. I believe the workflow should be:

• perf during developement.
• release the package with the best result.

[messense]

I agree with @Zheaoli that enabling frame pointers is very useful, I'm open to enable it by default.

FYI, Ubuntu is also going to enable frame pointers by default soon: https://ubuntu.com/blog/ubuntu-performance-engineering-with-frame-pointers-by-default

[Xuanwo]

Will this change affect the performance or binary size? And why rustc disabled it by default?

[Zheaoli]

Will this change affect the performance or binary size? And why rustc disabled it by default?

1. binary size will not be changed
2. the performance will lose from 1% to 2% (reported by the Ubuntu community)

From the Ubuntu report

Lower Overhead: Unwinding with frame pointers is significantly cheaper than using DWARF or DWARF-derived information.
Debugging Accessibility: Even those new to profiling can access high-quality data, democratising the process of performance optimisation. It will allow bcc-tools, bpftrace, perf and other such tooling to work out of the box.

I think those feature will help our user to diagnose the process problem more easy

[Xuanwo]

the performance will lose from 1% to 2%

Seems fine. Is there any other reason for rustc not enabling it by default?

[suyanhanx]

https://doc.rust-lang.org/rustc/codegen-options/index.html#force-frame-pointers
https://stackoverflow.com/questions/75332808/will-rustc-enable-frame-pointers-on-linux-by-default

Seems that it is not disabled by default. It depends on the target.

[Xuanwo]

I'm not very familiar with the differences between DWARF and frame-pointers. Does this mean I don't need to add debug=1 when building OpenDAL to generate symbols?

[Zheaoli]

I'm not very familiar with the differences between DWARF and frame-pointers.

FYI

1. https://www.manjusaka.blog/posts/2023/08/22/a-little-bit-idea-about-unwind
2. https://maskray.me/blog/2020-11-08-stack-unwinding#%E4%B8%AD%E6%96%87%E7%89%88

[Xuanwo]

If we add force-frame-pointers=yes, can we remove #3759?

[Zheaoli]

If we add force-frame-pointers=yes, can we remove #3759?

No! Without symbol, we can't do anything!

[xxchan]

I'm also doubtful about adding too many random rustflags. However, searching GitHub, I found tikv also enables this by default, so it seems a reasonable choice. https://github.com/tikv/tikv/blob/f9727af132109754e63fbb4910b73563d0b1da45/Makefile#L37-L51

ref: tikv/tikv#12480

---

Edit: But it seems that they used this to solve another niche issue

This PR introduces a new version of pprof-rs, which provides stack backtracking based on frame pointers,

[xxchan]

I'm not very familiar with the differences between DWARF and frame-pointers.

FYI

1. manjusaka.blog/posts/2023/08/22/a-little-bit-idea-about-unwind
2. maskray.me/blog/2020-11-08-stack-unwinding#%E4%B8%AD%E6%96%87%E7%89%88

These went to deep. 🤪

I think basically:

• frame pointer is part of debuginfo.
• DWARF is the format of debuginfo.
• debuginfo contains mainly mapping between source code and machine code.
  • debug=full: can use debugger to inspect variables and stuff
  • debug=1: can get line number, function name. so often needed for backtrace (e.g. panic) and profiling

perf is not a normal use cases

Agree. But I think profiling performance issues in production is also needed sometimes? Although in my experience, debug=1 is enough for generating flamegraphs (I havn't used perf though). Don't know why frame pointer is needed for opendal (or perf?).

[Xuanwo]

I don't have any major objections to this. @Zheaoli, would you be willing to submit a PR for this change?