[ty] Recursive protocols

[sharkdp]

Summary

Use a self-reference "marker" and fixpoint iteration to solve the stack overflow problems with recursive protocols. This is not pretty and somewhat tedious, but seems to work fine. Much better than all my fixpoint-iteration attempts anyway.

closes astral-sh/ty#93

Test Plan

New Markdown tests.

[github-actions]

mypy_primer results

No ecosystem changes detected ✅

[sharkdp]

Moving this out of draft-mode to get a general review before I spend more time resolving all the minor TODOs.

[AlexWaygood]

This looks like an excellent solution to me!

[dcreager]

Was just about to ask if we could re-use this technique for generics 😄

👍 for it being a TODO

[sharkdp]

The doc-comment of this function is probably more relevant? This is just the place where we would also have to look for references to the protocol class.

Instead of ClassLiteral, I would imagine that this function would maybe have to take Type as an argument if we want to handle self-referential type aliases like type Tree = dict[str, Tree]. Or maybe an enum (different things that we would want to replace).

[carljm]

Thank you!!

This clearly addresses a lot of the practical cases we see, but it does still have a problem with mutual recursion of multiple different classes.

I think we could consider landing this anyway as an initial step, but I think we will need to figure out how to generalize it beyond simple self-reference.

[carljm]

It seems like this approach has a significant limitation, in that it can handle direct self-reference, but it can't handle mutual recursion (because we can only represent a self-reference, not a "recursive reference to arbitrary something", which means we can't break a recursive loop involving more than one protocol.) And indeed, this still stack overflows on this branch:

from __future__ import annotations

from typing import Protocol
from ty_extensions import is_fully_static, is_subtype_of, static_assert

class Foo(Protocol):
    x: Bar

class Bar(Protocol):
    x: Foo

static_assert(is_fully_static(Foo))
static_assert(is_fully_static(Bar))

[sharkdp]

After talking to @carljm we decided to merge this as an immediate solution to an observed real-world problem. I don't think it introduces any wrong behavior either. And as Carl pointed out, the tests are hopefully useful in any case.

That said, we also think that there are severe limitations here (see Carl's comment regarding mutually-recursive protocols). And it's also not clear if this is a generalizable solution that would also work for self-referential type aliases or generic classes. @dcreager: it's probably not useful to spend time right now trying to make this specific approach here work for generics.

What I will do instead after merging this is to take one step back and think about the broader problem here, and do some research. Hopefully that leads to a general solution. I might not be able to spend the necessary amount of time on this before Tuesday. So if anyone else wants to work on this before then, feel free to do that (let me know).