DCMIPP support (different from DCMI) in Zephyr

[josuah]

@Sambsc19 previous message on #76798:

Happy to see so much progess on this. Im trying to get the equivilant of the stm32cube ide sample app continuous_DBM working on the stm32h7s78-dk (ov5640) on zephyr rtos. Theres just 1 thing that has me stuck, the mcu has DCMIPP not DCMI. As i understand DCMIPP does not use DMA. I havent come across this type of device tree config in Zephyr before (pipe0) and am unsure what the correct way to implement it is. I would like to stream camera to display but i also want to stream camera to host pc over usb. I can share pinctrl, dtsi, etc etc if thats helpful? Again, the pipe0 implementation is whats got me here. Any help massively appreciated.

Simon.

[josuah]

The missing piece here is the DCMIPP driver implementation for the STM32H7[R/S] SoC. Once that is done, you would be able to use the capture_to_lvgl sample as well as the upcoming uvc sample along with a supported image sensor.

P.S.: the path to discover what's missing

 

You mentioned you had this devkit:

https://www.st.com/en/evaluation-tools/stm32h7s78-dk.html

Here is where to check for board support:

https://docs.zephyrproject.org/latest/boards/index.html

It seems like the STM32H7S78-DK is supported:

https://docs.zephyrproject.org/latest/boards/st/stm32h7s78_dk/doc/index.html

Here is its root devicetree config, which includes the definition for the SoC stm32h7s7X8 that is inside:

zephyr/boards/st/stm32h7s78_dk/stm32h7s78_dk.dts

Line 8 in 67e07b7

#include <st/h7rs/stm32h7s7X8.dtsi>

And then it is a treasure hunt to find the full chip definition with all the peripherals:

zephyr/dts/arm/st/h7rs/stm32h7s7X8.dtsi

Line 8 in 67e07b7

#include <st/h7rs/stm32h7s7.dtsi>

zephyr/dts/arm/st/h7rs/stm32h7s7.dtsi

Line 8 in 67e07b7

#include <st/h7rs/stm32h7r7.dtsi>

zephyr/dts/arm/st/h7rs/stm32h7r3.dtsi

Line 8 in 67e07b7

#include <st/h7rs/stm32h7rs.dtsi>

And finally there... This seems to be the file where to insert the DCMIPP driver definition.

https://github.com/zephyrproject-rtos/zephyr/blob/main/dts/arm/st/h7rs/stm32h7rs.dtsi

STM32 boards and chips rely on the hal_stm32 layer to access CubeMX implementation, which are also used in Zephyr.

There is definitely a DCMIPP present here:

https://github.com/zephyrproject-rtos/hal_stm32/blob/main/stm32cube/stm32h7rsxx/drivers/src/stm32h7rsxx_hal_dcmipp.c

However, there was not yet any integration of it into Zephyr. There is the DCMI one, but not the DCMIPP one:

https://github.com/zephyrproject-rtos/zephyr/blob/main/drivers/video/video_stm32_dcmi.c

[Sambsc19]

Hi @josuah thanks for opening the issue. I have made some progress and thought I'd reach out with some concerns while i'm developing.

I've updated to zephyr v4.1.0-rc1

I've created st.stm32-dcmipp.yaml, Kconfig.stm32_dcmipp and other required config files.

I've created video_stm32_dcmipp.c and included the CUBEMX driver files.

My dts:

/ {

    chosen {
		zephyr,camera = &dcmipp;
	};

};

&i2c1 {
	status = "okay";
	pinctrl-0 = <&i2c1_scl_pb6 &i2c1_sda_pb9>;
	pinctrl-names = "default";
	clock-frequency = <I2C_BITRATE_FAST>;

	ov5640: ov5640@3c {
		compatible = "ovti,ov5640";
		reg = <0x3c>;
		status = "okay";
		reset-gpios = <&gpion 7 GPIO_OPEN_DRAIN>;
		powerdown-gpios = <&gpiof 5 GPIO_ACTIVE_HIGH>;

		port {
			ov5640_ep_out: endpoint {
				// remote-endpoint = <&dcmi_ep_in>;
				remote-endpoint-label = "dcmi_ep_in"; /* Add this line */
			};
		};

	};
};

&dcmipp {
	pinctrl-0 = <&dcmi_hsync_pg3 &dcmi_pixclk_pd5 &dcmi_vsync_pb7
				 &dcmi_d0_pc6 &dcmi_d1_pc7 &dcmi_d2_pe0 &dcmi_d3_pe1
				 &dcmi_d4_pe4 &dcmi_d5_pd3 &dcmi_d6_pb8 &dcmi_d7_pd14>;

	pinctrl-names = "default";
	status = "okay";

	sensor = <&ov5640>;
	bus-width = <8>;
	hsync-active = <1>; //high correct
	vsync-active = <1>; //high correct
	pixelclk-active = <1>; //rising, correct
	capture-rate = <1>;
	// dmas = <&hpdma1 0 75 (STM32_DMA_PERIPH_TO_MEMORY | STM32_DMA_PERIPH_NO_INC | STM32_DMA_MEM_INC | STM32_DMA_PERIPH_8BITS | STM32_DMA_MEM_32BITS | STM32_DMA_PRIORITY_HIGH)>;

	port {
		dcmi_ep_in: endpoint {
			// remote-endpoint = <&ov5640_ep_out>;
			remote-endpoint-label = "ov5640_ep_out"; /* Add this line */
		};
	};
};

dtsi:

dcmipp: dcmipp@50002000 {
			compatible = "st,stm32-dcmipp";
			reg = <0x50002000 0x400>;
			interrupts = <95 0>;
			interrupt-names = "dcmipp";
		
			clocks = <&rcc STM32_CLOCK(APB5, 2U)>,  /* APB5 peripheral clock for control */
					<&rcc STM32_SRC_HSI_KER PPRE5(0)>;
			clock-names = "dcmipp_pclk",            /* APB5 clock for control */
						  "dcmipp_aclk";             /* AXI clock for data transfer */
		
			// resets = <&rcc STM32_RESET(APB5, 2U)>,  /* Reset for APB5 (control interface) */
			// 		 <&rcc STM32_SRC_HSI_KER PPRE5(0)>;   /* Reset for AXI interface */
			// reset-names = "dcmipp_pclk",            /* Reset for APB5 clock domain */
			// 			  "dcmipp_aclk";             /* Reset for AXI clock domain */
		
			status = "disabled";                     /* Set to "okay" in board-specific DTS */
		};
		

Logs:

Description of logs:

I have implemented the bare minimum to observe the 2 drivers initialize. It seems both drivers are included in build however the ov5640 is initialized before video_stm32_dcmipp. Tying to change priority causes error. In the stm32cube dcmipp sample i am trying to reproduce the dcmipp is initialized first. For simplicity I am currently calling the ov5640 init from the dcmipp driver init.

When the ov5640 initializes I can see reset and powerdown pins pulled low on logic analyser but it seems there is no i2c communication. I dont see any clock signal or other gpio action either.

[00:00:00.128,000] video_ov5640: Unable to write to reset all registers

Questions:

• I have verified the gpios and alternate function values with datasheet and STM32CUBEMX .ioc. Does the dts look correct to you? I am tryng to determine the i2c comms issue.

• How do i configure AXI master clock? <&rcc STM32_SRC_HSI_KER PPRE5(0)>; is wrong. ppre5 is already defined in dts &rcc. DCMIPP is AXI mater INI8 as seen in attached snippet. Ive attached a snippet of the DCMIPP clocks and resets from the ref manual. the pxclk is an input from the dcmipp device and defined with gpios so the pclk and aclk are required.

• no "resets" defined in st,stm32-dcmipp. should they be added here?

• I have not seen this sample demonstrated. is it supposed to init the ov5640 first?

Any answers are most appreciated.

the stm32cube sample I'm using as a guide is at STM32CubeIDE\workspace_e\DCMIPP_ContinuousDBM\Appli\Src

The init functions in order executed in log:

static int ov5640_init(const struct device *dev)
{
	const struct ov5640_config *cfg = dev->config;
	struct video_format fmt;
	uint16_t chip_id;
	int ret;

	LOG_WRN("ov5640_init");

	return 0;
}

static int video_stm32_dcmipp_init(const struct device *dev)
{
	const struct video_stm32_dcmipp_config *config = dev->config;
	struct video_stm32_dcmipp_data *data = dev->data;
	int err;

	LOG_WRN("DCMIPP INIT");

	// k_msleep(5000);

	/* Configure DT provided pins */
	err = pinctrl_apply_state(config->pctrl, PINCTRL_STATE_DEFAULT);
	if (err < 0) {
		LOG_ERR("pinctrl setup failed. Error %d.", err);
		return err;
	}
	else{
		LOG_WRN("pinctrl setup success");
	}

	// /* Initialize DMA peripheral */
	// err = stm32_dma_init(dev);
	// if (err < 0) {
	// 	LOG_ERR("DMA initialization failed.");
	// 	return err;
	// }

	/* Enable DCMI clock */
	err = stm32_dcmipp_enable_clock(dev);
	if (err < 0) {
		LOG_ERR("Clock enabling failed.");
		return -EIO;
	}

	data->dev = dev;
	k_fifo_init(&data->fifo_in);
	k_fifo_init(&data->fifo_out);

	// /* Run IRQ init */
	config->irq_config(dev);

	DCMIPP_ParallelConfTypeDef ParallelConfig = {0};
	DCMIPP_PipeConfTypeDef PipeConfig = {0};

	/** Pipe 0 Config */
	data->hdcmipp.Instance = DCMIPP;
	if (HAL_DCMIPP_Init(&data->hdcmipp) != HAL_OK)
	{
		LOG_ERR("HAL_DCMIPP_Init error");
		Error_Handler();
	}
	ParallelConfig.PCKPolarity = DCMIPP_PCKPOLARITY_RISING ;
	ParallelConfig.HSPolarity = DCMIPP_HSPOLARITY_HIGH ;
	ParallelConfig.VSPolarity = DCMIPP_VSPOLARITY_HIGH ;
	ParallelConfig.ExtendedDataMode = DCMIPP_INTERFACE_8BITS;
	ParallelConfig.Format = DCMIPP_FORMAT_RGB565;
	ParallelConfig.SwapBits = DCMIPP_SWAPBITS_DISABLE;
	ParallelConfig.SwapCycles = DCMIPP_SWAPCYCLES_ENABLE;
	ParallelConfig.SynchroMode = DCMIPP_SYNCHRO_HARDWARE;
	HAL_DCMIPP_PARALLEL_SetConfig(&data->hdcmipp, &ParallelConfig);
	PipeConfig.FrameRate = DCMIPP_FRAME_RATE_ALL;
	if (HAL_DCMIPP_PIPE_SetConfig(&data->hdcmipp, DCMIPP_PIPE0, &PipeConfig) != HAL_OK)
	{
		LOG_ERR("HAL_DCMIPP_PIPE_SetConfig error");
		Error_Handler();
	}

	/* Initialize DCMI peripheral */
	err = HAL_DCMIPP_Init(&data->hdcmipp);
	if (err != HAL_OK) {
		LOG_ERR("DCMI initialization failed.");
		return -EIO;
	}

	k_sleep(K_MSEC(100));
	LOG_WRN("%s inited", dev->name);

	init_cam_ov5640();

	return 0;
}

int init_cam_ov5640(void)
{
	const struct device *dev = DEVICE_DT_GET(DT_NODELABEL(ov5640));
	const struct ov5640_config *cfg = dev->config;
	struct video_format fmt;
	uint16_t chip_id;
	int ret;
	LOG_WRN("init_cam_ov5640");

	if (!device_is_ready(cfg->i2c.bus)) {
		LOG_ERR("Bus device is not ready");
		return -ENODEV;
	}
	else{
		LOG_WRN("ov5640 ready");
	}

	if (!gpio_is_ready_dt(&cfg->reset_gpio)) {
		LOG_ERR("%s: device %s is not ready", dev->name, cfg->reset_gpio.port->name);
		return -ENODEV;
	}
	else{
		LOG_WRN("%s: device %s is ready", dev->name, cfg->reset_gpio.port->name);
	}

	if (!gpio_is_ready_dt(&cfg->powerdown_gpio)) {
		LOG_ERR("%s: device %s is not ready", dev->name, cfg->powerdown_gpio.port->name);
		return -ENODEV;
	}
	else{
		LOG_WRN("%s: device %s is ready", dev->name, cfg->powerdown_gpio.port->name);
	}

	/* Power up sequence */
	if (cfg->powerdown_gpio.port != NULL) {
		ret = gpio_pin_configure_dt(&cfg->powerdown_gpio, GPIO_OUTPUT_ACTIVE);
		if (ret) {
			return ret;
		}
	}

	if (cfg->reset_gpio.port != NULL) {
		ret = gpio_pin_configure_dt(&cfg->reset_gpio, GPIO_OUTPUT_ACTIVE);
		if (ret) {
			return ret;
		}
	}

	LOG_WRN("setting powerdown_gpio and reset_gpio low in 2 seconds");

	k_sleep(K_SECONDS(2));

	if (cfg->powerdown_gpio.port != NULL) {
		gpio_pin_set_dt(&cfg->powerdown_gpio, 0);
	}

	k_sleep(K_MSEC(1));

	if (cfg->reset_gpio.port != NULL) {
		gpio_pin_set_dt(&cfg->reset_gpio, 0);
	}

	k_sleep(K_MSEC(20));

	/* Reset all registers */
	ret = ov5640_write_reg(&cfg->i2c, SCCB_SYS_CTRL1_REG, 0x11);
	if (ret) {
		LOG_ERR("Unable to write to reset all registers");
		return -EIO;
	}

	/* Software reset */
	ret = ov5640_write_reg(&cfg->i2c, SYS_CTRL0_REG, SYS_CTRL0_SW_RST);
	if (ret) {
		LOG_ERR("Unable to perform software reset");
		return -EIO;
	}

	k_sleep(K_MSEC(5));

	/* Initialize register values */
	ret = ov5640_write_multi_regs(&cfg->i2c, init_params_common,
				      ARRAY_SIZE(init_params_common));
	if (ret) {
		LOG_ERR("Unable to initialize the sensor");
		return -EIO;
	}

	if (ov5640_is_dvp(dev)) {
		ret = ov5640_write_multi_regs(&cfg->i2c, init_params_dvp,
					      ARRAY_SIZE(init_params_dvp));

		if (ret) {
			LOG_ERR("Unable to initialize the sensor with DVP parameters");
			return -EIO;
		}
	} else {
		/* Set virtual channel */
		ret = ov5640_modify_reg(&cfg->i2c, 0x4814, 3U << 6,
					(uint8_t)(DEFAULT_MIPI_CHANNEL) << 6);
		if (ret) {
			LOG_ERR("Unable to set virtual channel");
			return -EIO;
		}
	}

	/* Check sensor chip id */
	ret = ov5640_read_reg(&cfg->i2c, CHIP_ID_REG, &chip_id, sizeof(chip_id));
	if (ret) {
		LOG_ERR("Unable to read sensor chip ID, ret = %d", ret);
		return -ENODEV;
	}

	if (chip_id != CHIP_ID_VAL) {
		LOG_ERR("Wrong chip ID: %04x (expected %04x)", chip_id, CHIP_ID_VAL);
		return -ENODEV;
	}

	/* Set default format */
	fmt.pixelformat = VIDEO_PIX_FMT_RGB565;
	if (ov5640_is_dvp(dev)) {
		/* Set default resolution to QQVGA (160x120) */
		fmt.width = 160;
		fmt.height = 120;
	} else {
		/* Set default resolution to 720p */
		fmt.width = 1280;
		fmt.height = 720;
	}
	fmt.pitch = fmt.width * 2;
	ret = ov5640_set_fmt(dev, VIDEO_EP_OUT, &fmt);
	if (ret) {
		LOG_ERR("Unable to configure default format");
		return -EIO;
	}

	return 0;
};

[Sambsc19]

Device is found at 3C with driver initializations omitted. I can see comms on logic analyser.

Device cannot be found after running current driver initializations.

Now investing other devices on i2c1 and driver initializations.

[josuah]

Hello and thank you for trying!

I will have more time to look into detail later, but a frequent cause of an image sensor not responding over I2C is the "system" clock not being provided, which indeed can come from a driver priority issue: the DCMIPP might need to be initialized first to provide the clock to the image sensor.

A similar situation faced by @ngphibang in this issue: #80304

[JarmouniA]

How do i configure AXI master clock?

@Sambsc19 I think it's configured by default, and enabled for the peripheral on bus transaction request, unless clock gating is disabled (see below). And the frequency is given by:

So I think you only need 1 cell in clocks property.

[josuah]

Adding an I2C scan at every step of the initialization could eventually help with telling at which step it "disappears" from the bus.

It might also be possible to use the OV5640 driver or import parts of its init sequence.

[josuah]

@Sambsc19: There seems to be some effort to get DCMIPP supported on Zephyr: #87070 (comment)

[josuah]

Forwarded message from @avolmat-st from here:

Hi @Sambsc19, actually I have most of the driver ready. I am developping it in the STM32N6 context so it has support for all pipe0/1 and 2 but since I used this in the STM32N6-DK context this is done in CSi mode exclusively for the moment. However adding parallel mode as well shouldn't be much of an issue on top of it. Buffer managements / stream pause (whenever there is no more buffers) is all done and working. Obviously same as for you, I did the pipe0 originally but rapidly moved to the pipe1 which has demosaicing / decimation / downscale / color conversion etc available and allow to work with the IMX335 bayer sensor available on the STM32N6-DK (for which I have written the driver as well). As we discussed above in this thread, the issue I have currently is simply to be able to address each pipe separately since I was relying on the endpoint arguments. Now I am moving to implementing 3 devices so that it is possible to address all pipe. In term of feature, here's what is available already:

• CSI
• Pipe0 (basically direct capture of what is coming from the sensor)
• Pipe1 (demosaicing / decimation / color conversion (RGB <=> YUV) / downscaling and various output formats that can be generated.
• Pipe2 (same as above Pipe1 except color conversion)
  I am currently working on adding the ISP processing on top of it, hence could you can wait very few days before I finalize this so I can share it. I'd also like to rebase my IMX335 sensor branch on top of the CSS helper functions and also probably on top of the control framework series since this will help cleanup the code as well.
  As far as testing is concerned, I haven't done it via the usb uvc sample as you did however I confirm that it is well working using the capture test app to done a preview on the LTDC.

[josuah]

I'd also like to rebase my IMX335 sensor branch on top of the CSS helper functions and also probably on top of the control framework series since this will help cleanup the code as well.

We have an IMX219 driver in the work here:

• drivers: video: add imx219 sensor (RPi Cam v2) #88011

If the IMX335 has a module publicly available let us know which: I'd be glad to reproduce the results on our MIPI/CSI hardware and help migrate to CCI and other proposed APIs.