De-serialized response from custom serializer does not have expected CachedResponse attributes

[jseabold]

The problem

My CachedResponse instance does not appear to be getting default attributes that aren't on the original response like 'created_at' and 'expires' properly. It has cache_key which gets explicitly assigned.

Repro:

In [1]: import requests_cache

In [2]: import pickle, gzip

In [3]: from requests_cache.serializers import SerializerPipeline, Stage

In [4]: compressed_serializer = SerializerPipeline([pickle, Stage(gzip, dumps='compress', loads='decompress')])

In [5]: session = requests_cache.CachedSession(serializer=compressed_serializer)

In [6]: session.get('http://httpbin.org/get')
Out[6]: <Response [200]>

In [7]: session.get('http://httpbin.org/get')
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-7-f8b0eaba1a9f> in <module>
----> 1 session.get('http://httpbin.org/get')

~/.miniconda3/lib/python3.9/site-packages/requests/sessions.py in get(self, url, **kwargs)
    553 
    554         kwargs.setdefault('allow_redirects', True)
--> 555         return self.request('GET', url, **kwargs)
    556 
    557     def options(self, url, **kwargs):

~/.miniconda3/lib/python3.9/site-packages/requests_cache/session.py in request(self, method, url, params, data, json, expire_after, **kwargs)
    114         """
    115         with self.request_expire_after(expire_after), patch_form_boundary(**kwargs):
--> 116             return super().request(
    117                 method,
    118                 url,

~/.miniconda3/lib/python3.9/site-packages/requests/sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    540         }
    541         send_kwargs.update(settings)
--> 542         resp = self.send(prep, **send_kwargs)
    543 
    544         return resp

~/.miniconda3/lib/python3.9/site-packages/requests_cache/session.py in send(self, request, **kwargs)
    141         if not (self._disabled or actions.skip_read):
    142             cached_response = self.cache.get_response(cache_key)
--> 143             actions.update_from_cached_response(cached_response)
    144         is_expired = getattr(cached_response, 'is_expired', False)
    145 

~/.miniconda3/lib/python3.9/site-packages/requests_cache/cache_control.py in update_from_cached_response(self, response)
    126         Check for relevant cache headers on a cached response, and set corresponding request headers.
    127         """
--> 128         if not response or not response.is_expired:
    129             return
    130 

~/.miniconda3/lib/python3.9/site-packages/requests_cache/models/response.py in is_expired(self)
     89     def is_expired(self) -> bool:
     90         """Determine if this cached response is expired"""
---> 91         return self.expires is not None and datetime.utcnow() >= self.expires
     92 
     93     @property

AttributeError: 'CachedResponse' object has no attribute 'expires'

Environment

• requests-cache version: 0.8.1 installed via pip
• Python version: 3.9
• Platform: Ubuntu

I've been unable to get the package to successfully build on 3.9 due to a puzzling reported setuptools import error, so I don't know if this persists in master or is somehow a local issue. Can provide anymore information as needed.

[JWCook]

Thanks for reporting that. I'm not sure off the top of my head why that AttributeError would be happening, but I'll look into it.

As for the ImportError with setuptools, are you seeing that when installing for local development with poetry? If so, it might be this issue: python-poetry/poetry#3153

Try installing the poetry 1.2 pre-release:

curl -o install-poetry.py https://install.python-poetry.org
python install-poetry.py --version 1.2.0a2

[JWCook]

@jseabold Here, try this:

import gzip
from requests_cache import CachedSession, SerializerPipeline, Stage, pickle_serializer

compressed_serializer = SerializerPipeline([
    pickle_serializer,
    Stage(dumps=gzip.compress, loads=gzip.decompress),
])
session = CachedSession(serializer=compressed_serializer)

session.cache.clear()
session.get('http://httpbin.org/get')
session.get('http://httpbin.org/get')

The difference with requests_cache.serializers.pickle_serializer is that it's doing a bit more pre-/post-processing work so it pickles builtin python types, and then reassembles the CachedResponse after deserializing, rather than directly pickling/unpickling CachedResponse objects. It makes the serialized data more portable across requests-cache versions.

I was able to reproduce the error with your example, but still not quite sure why that's happening. In the mean time, I updated the pre-release docs with the working example above.

[JWCook]

Okay, now I remember what's going on. So requests.Response makes some modifications to pickling behavior via __getstate__ and __setstate__. Notably, it only pickles a subset of attributes, which excludes our custom attributes including expires. I had previously added a workaround for this, but then removed it after making the modified pickle_serializer (mentioned in my previous comment) the default.

I think using plain pickle should be supported, though, so I'll add this workaround back.

[JWCook]

Fixed in pre-release build 0.9.0.dev1. Your original example now works as expected. Let me know if you run into any more problems.