Add support for docker-py / using VCR.py with docker-py makes tests hang indefinitely with 100% CPU load

[aklajnert]

Here's my example test file:

import docker
import vcr


#@vcr.use_cassette()
def test_docker_images():
    env = docker.from_env()
    images = env.images.list()

    assert images

The test works fine, but if I'll uncomment the @vcr.use_cassette(), it will hang forever.

Environment (Ubuntu, python 3.7):

attrs==19.3.0
certifi==2020.4.5.1
chardet==3.0.4
docker==4.2.0
idna==2.9
importlib-metadata==1.6.0
more-itertools==8.2.0
multidict==4.7.5
packaging==20.3
pluggy==0.13.1
py==1.8.1
pyparsing==2.4.7
pytest==5.4.1
PyYAML==5.3.1
requests==2.23.0
six==1.14.0
urllib3==1.25.9
vcrpy==4.0.2
wcwidth==0.1.9
websocket-client==0.57.0
wrapt==1.12.1
yarl==1.4.2
zipp==3.1.0

The examples from the documentation are working fine.

Edit: tested on Windows, and the same example works as expected, the test runs and cassette is created.

[waynehamadi]

exactly same issue here

[hartwork]

I'm confirming the issue, Python 3.8, docker-py 1.10.6, vcrpy 4.2.1.
I needed to adapt the example code a but, this produces the issue:

import docker
import vcr

@vcr.use_cassette()
def test_docker_images():
    env = docker.from_env()
    images = env.images()
    assert images

Running Docker daemon is needed of course.

In a fresh venv I did:

# pip install pytest docker-py vcrpy 'urllib3<2'

"Hanging" is actually one core spinning at 100% CPU, apparently forever.

[hartwork]

As indicated by @gravelBridge through #692 the issue is an infinite loop at…

vcrpy/vcr/patch.py

Lines 237 to 238 in 14cef83

	while not isinstance(connection, connection_class):
	connection = get_conn(pool, timeout)

…where the instance is of type docker.transport.unixconn.UnixHTTPConnection in practice while an instance of VCRRequestsHTTPConnection is expected. I think that may mean that:

• vcrpy would need to start monkey patching in docker-py modules
• that docker-py is not supported as of today
• that cases like this will all produce 100% CPU usage with no end, which is not ideal.

[lucasgomide]

hey @insuyun

I'm facing a similar issue using docker-py + vcrpy. In my case I got the following error (using official vcr package)

docker.errors.DockerException: Error while fetching server API version: UnixHTTPConnectionPool(host='localhost', port=None): Max retries exceeded with url: /version (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x15748f590>: Failed to establish a new connection: [Errno 61] Connection refused'))

Then after switching to your branch the error changed to

    def _get_raw_response_socket(self, response):
        self._raise_for_status(response)
        if self.base_url == "http+docker://localnpipe":
            sock = response.raw._fp.fp.raw.sock
        elif self.base_url.startswith("http+docker://ssh"):
            sock = response.raw._fp.fp.channel
        else:
>           sock = response.raw._fp.fp.raw
E           AttributeError: 'VCRHTTPResponse' object has no attribute '_fp'. Did you mean: 'fp'?

        .venv/lib/python3.12/site-packages/docker/api/client.py:353: AttributeError <---- ERROR

do you have any idea how to address that?

[hartwork]

@lucasgomide docker-py is not at all supported, please see #519 (comment) above.

[insuyun]

hey @insuyun

I'm facing a similar issue using docker-py + vcrpy. In my case I got the following error (using official vcr package)

docker.errors.DockerException: Error while fetching server API version: UnixHTTPConnectionPool(host='localhost', port=None): Max retries exceeded with url: /version (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x15748f590>: Failed to establish a new connection: [Errno 61] Connection refused'))

Then after switching to your branch the error changed to

    def _get_raw_response_socket(self, response):
        self._raise_for_status(response)
        if self.base_url == "http+docker://localnpipe":
            sock = response.raw._fp.fp.raw.sock
        elif self.base_url.startswith("http+docker://ssh"):
            sock = response.raw._fp.fp.channel
        else:
>           sock = response.raw._fp.fp.raw
E           AttributeError: 'VCRHTTPResponse' object has no attribute '_fp'. Did you mean: 'fp'?

        .venv/lib/python3.12/site-packages/docker/api/client.py:353: AttributeError <---- ERROR

do you have any idea how to address that?

HI @lucasgomide. This patch works for me. One thing you may try is to setup ignore_localhost to True. I think this would be helpful if you don't want to record/replay docker's requests.