Getting too many surrogate keys

[daniel-ifrim]

I'm getting this in the logs:

[2017-09-17 18:05:25] report.CRITICAL: Return status 400 {"method":"GET","url":"http:/","invalidateInfo":"https://api.fastly.com/service/ [service_id] /purge"} []

I added a debug message on $responseBody in _purge() function and I see this:

[2017-09-17 18:05:25] report.CRITICAL: HTTP/1.1 400 Bad Request {"status": "fail", "reason": "Batch surrogate key request only supports up to 256 surrogate keys"} {"method":"GET","url":"http:/","invalidateInfo":"https://api.fastly.com/service/ [service_id] /purge"} []

Maybe split the cache tag keys and send consecutive purges with keys sets, each having 250-256 keys ? Or something better ?

[udovicic]

@daniel-ifrim On which Magento version is this issue observed?

[daniel-ifrim]

@udovicic On Magento EE 2.1.4.
The store has products with many options.
One way to reproduce is to use Target Rule (Related Products Rules in admin) and have a rule active that covers many products. It will try to send many cache tags to purge.
But I'm not in this case. The 256 surrogate keys limit could be handled by the extension.

Is there already a check in the code on 256 max surrogate keys ? I just don't know if I'm missing it.

[udovicic]

@daniel-ifrim Currently, there are no restrictions on length of the keys. There is a piece of code that shortens the keys so more can fit, but that is it. I will see what can be done about this and will keep you updated

[daniel-ifrim]

@udovicic I don't know if I made myself clear.
It's about validation of the number of surrogate keys that Fastly accepts.
It's not about the length of a single surrogate key.
What I saw in logs was Fastly saying that the purge request sent too many surrogate keys. And Fastly returns HTTP 400 Bad Request.

In another case, there are configurable products with many options that can have over 256 simple products attached. Splitting 400 surrogate keys into 2 purge requests (vendor/fastly/magento2/Model/Api.php/Api.php::cleanBySurrogateKey()) with the first one having 256 keys and the second one 134 keys looks ok.

The issue with Target Rule is extreme.
In the case of Target Rule where indexer wants to clean cache by tags for a lot of products, there can be a hard limit in the extension like 3-10 purge requests (10 x 256 = 2560 surrogate keys to clean). So Magento 2 won't spam Fastly.

[udovicic]

@daniel-ifrim Yes, I understand the issue, no worries.

Current state is that all cache tags are bundled into single purge request. So in case you have a 1000 tags to clear, it would be required to send them in smaller batches as they do not fit in the single request. And yes, there needs to be a limit on how many batches as well or someone will spam the service.

I will keep you updated.

[vvuksan]

Fixed in #133 and part of 1.2.39 release.