Not caching in multi-stage build

[gurbaaz27]

I have a Dockerfile utlising cache advantage of multi-stage build, and if I build it using docker cli it works perfectly. But if use docker-py to build the image using the same dockerfile, it re-installs the requirements using pip every time, even if there is no change in code directory.

Dockerfile

## Multi Stage build to take advantage of the Docker cache layers to speed up the builds

## Stage I

# Start from the official Python base image.
FROM python:3.8.10

# Set the current working directory to /code.
# This is where we'll put the requirements.txt file and the app directory.
WORKDIR /code

COPY install_functions.txt .
COPY copy_requirements.sh .
COPY functions functions
RUN ./copy_requirements.sh

## Stage II

FROM python:3.8.10

## Update and Install Packages
RUN apt-get update
RUN apt-get install ffmpeg libsm6 libxext6 -y

## Update pip
ENV PIP_ROOT_USER_ACTION=ignore
ENV PIP_NO_CACHE_DIR=1
RUN pip install --upgrade pip

# Set the current working directory to /code.
# This is where we'll put the requirements.txt file and the app directory.
WORKDIR /code

# Install the package dependencies in the requirements file.
COPY --from=0 /code .
COPY install_functions.txt .
COPY install_requirements.sh .
RUN ./install_requirements.sh

# Copy the remaining functions to the /code directory.
COPY functions functions

Docker-cli command

docker build -t $tagname .

Output

Step 1/18 : FROM python:3.8.10
 ---> a369814a9797
Step 2/18 : WORKDIR /code
 ---> Using cache
 ---> 6050321ebdb3
Step 3/18 : COPY install_functions.txt .
 ---> Using cache
 ---> 9ec3c9862a9a
Step 4/18 : COPY copy_requirements.sh .
 ---> Using cache
 ---> 82b6e6bc1665
Step 5/18 : COPY functions functions
 ---> Using cache
 ---> 8eeae60ff70f
Step 6/18 : RUN ./copy_requirements.sh
 ---> Using cache
 ---> bd26459355ba
Step 7/18 : FROM python:3.8.10
 ---> a369814a9797
Step 8/18 : RUN apt-get update
 ---> Using cache
 ---> 3cf92e617eb0
Step 9/18 : RUN apt-get install ffmpeg libsm6 libxext6 -y
 ---> Using cache
 ---> 10cb11108f34
Step 10/18 : ENV PIP_ROOT_USER_ACTION=ignore
 ---> Using cache
 ---> 22aab820db77
Step 11/18 : ENV PIP_NO_CACHE_DIR=1
 ---> Using cache
 ---> 9b57f4673e40
Step 12/18 : RUN pip install --upgrade pip
 ---> Using cache
 ---> 874b82f5fd5f
Step 13/18 : WORKDIR /code
 ---> Using cache
 ---> 6256061d5bae
Step 14/18 : COPY --from=0 /code .
 ---> Using cache
 ---> 4f91640f93f8
Step 15/18 : COPY install_functions.txt .
 ---> Using cache
 ---> fea29b2f56da
Step 16/18 : COPY install_requirements.sh .
 ---> Using cache
 ---> 11db1813a9b2
Step 17/18 : RUN ./install_requirements.sh
 ---> Using cache
 ---> 7786c8b7aaf3
Step 18/18 : COPY functions functions
 ---> Using cache
 ---> 80a1da876432
Successfully built 80a1da876432

Python script

docker_client = docker.APIClient(base_url="unix://var/run/docker.sock")
docker_client.build(path=path, tag=tag, decode=True)

Output

Step 1/18 : FROM python:3.8.10

 ---> a369814a9797
Step 2/18 : WORKDIR /code

 ---> Using cache
 ---> 6050321ebdb3
Step 3/18 : COPY install_functions.txt .

 ---> 6ae19224e0e3
Step 4/18 : COPY copy_requirements.sh .

 ---> d35632789017
Step 5/18 : COPY functions functions

 ---> 5a4118f5bd03
Step 6/18 : RUN ./copy_requirements.sh

 ---> Running in 2ae618c78cc1
 Requirements corresponding to needed functions copied successfully 
 ---> fe49af04deb0
Step 7/18 : FROM python:3.8.10

 ---> a369814a9797
Step 8/18 : RUN apt-get update

 ---> Using cache
 ---> 3cf92e617eb0
Step 9/18 : RUN apt-get install ffmpeg libsm6 libxext6 -y

 ---> Using cache
 ---> 10cb11108f34
Step 10/18 : ENV PIP_ROOT_USER_ACTION=ignore

 ---> Using cache
 ---> 22aab820db77
Step 11/18 : ENV PIP_NO_CACHE_DIR=1

 ---> Using cache
 ---> 9b57f4673e40
Step 12/18 : RUN pip install --upgrade pip

 ---> Using cache
 ---> 874b82f5fd5f
Step 13/18 : WORKDIR /code

 ---> Using cache
 ---> 6256061d5bae
Step 14/18 : COPY --from=0 /code .

 ---> Using cache
 ---> 4f91640f93f8
Step 15/18 : COPY install_functions.txt .

 ---> f12aa2b9d99d
Step 16/18 : COPY install_requirements.sh .

 ---> afc723add0be
Step 17/18 : RUN ./install_requirements.sh

 ---> Running in d747e4fc7a10
Collecting numpy==1.23.2
  Downloading numpy-1.23.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.1/17.1 MB 240.7 MB/s eta 0:00:00

Collecting pandas==1.4.4
  Downloading pandas-1.4.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.7/11.7 MB 210.0 MB/s eta 0:00:00

Collecting python-dateutil==2.8.2
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
...... and so on

Any help would be appreciated.