docstring code formatter: add support for Markdown Python code snippets

[BurntSushi]

The initial implementation of the docstring code snippet formatter in #8811 only reformats doctest code snippets. This issue tracks the work for making it also reformat Markdown Python code snippets. Work should probably proceed by adding a new CodeExampleKind:

ruff/crates/ruff_python_formatter/src/expression/string.rs

Lines 1373 to 1385 in d9845a2

	/// The kind of code example observed in a docstring.
	#[derive(Clone, Debug, Eq, PartialEq)]
	enum CodeExampleKind {
	/// Code found in Python "doctests."
	///
	/// Documentation describing doctests and how they're recognized can be
	/// found as part of the Python standard library:
	/// https://docs.python.org/3/library/doctest.html.
	///
	/// (You'll likely need to read the regex matching used internally by the
	/// doctest module to determine more precisely how it works.)
	Doctest(CodeExampleDoctest),
	}

[ofek]

I am especially looking forward to this as my code snippets render as documentation and are almost exclusively written as Markdown Python code blocks.

[BurntSushi]

@ofek Do you have any example code I could take a look at that uses Markdown blocks?

[ofek]

• https://github.com/pypa/hatch/blob/52a81d80cd03bfb90d99a5837c778971ea0c8c22/src/hatch/env/plugin/interface.py#L23-L41 (rendered)
• https://github.com/pypa/hatch/blob/52a81d80cd03bfb90d99a5837c778971ea0c8c22/src/hatch/env/plugin/interface.py#L694-L697 (rendered)
• https://github.com/DataDog/integrations-core/blob/master/datadog_checks_base/datadog_checks/base/checks/base.py#L990-L995 (rendered)

[konstin]

Another example, with mixed non-python and python blocks: https://github.com/sacdallago/bio_embeddings/blob/efb9801f0de9b9d51d19b741088763a7d2d0c3a2/bio_embeddings/embed/bepler_embedder.py#L1-L38. i think it makes sense to assume blocks are python just like rustdoc defaults to code blocks being python, but we should skip blocks with an explicitly different language

[ofek]

I would prefer if we don't treat Markdown blocks differently compared to other parsers and assume unspecified blocks are generic, similar to the first block in your example where they show shell commands to run.

[BurntSushi]

I think it makes sense to assume unmarked blocks are Python code. As @konstin said, this is what rustdoc does and it works great. The mitigation here is that if the blocks are not valid Python code, then we can leave them as-is untouched.

[ofek]

I understand what you both are saying but I'm saying that model does not work for Python because historically we do not use Markdown and have no such concept of embedded documentation that is rendered in a standard way. If you do what you say here you're going to immediately fail on the vast majority of code blocks, because most are not Python, because we do not do that.

[zanieb]

Here are some additional examples from a real project (kind of randomly pulled, there are more for sure)

• https://github.com/PrefectHQ/prefect/blob/bbaa0208d476240da1ac370d2735c4af41c304f0/src/prefect/runner/runner.py#L6-L29 (python annotation)
• https://github.com/PrefectHQ/prefect/blob/bbaa0208d476240da1ac370d2735c4af41c304f0/src/prefect/utilities/annotations.py#L81-L89 (no annotation, python)
• https://github.com/PrefectHQ/prefect/blob/bbaa0208d476240da1ac370d2735c4af41c304f0/src/prefect/blocks/kubernetes.py#L38-L42 (python annotation)
• https://github.com/PrefectHQ/prefect/blob/bbaa0208d476240da1ac370d2735c4af41c304f0/src/prefect/settings.py#L160-L163 (python annotation)
• https://github.com/PrefectHQ/prefect/blob/bbaa0208d476240da1ac370d2735c4af41c304f0/src/prefect/utilities/templating.py#L180-L186 (no annotation, json)
• https://github.com/PrefectHQ/prefect/blob/bbaa0208d476240da1ac370d2735c4af41c304f0/src/prefect/workers/process.py#L10-L12 (bash annotation)