WIP: 476 - 20200602 type hints (Being broken apart into subsets) #473

hjmjohnson · 2020-06-03T00:28:05Z

Related to #476

Description

The next round of adding typehints

Status

Ready

Types of changes

Docstrings/Documentation updated

.pyre_configuration

.pyre/pyre.stderr

wyli

thanks @hjmjohnson and the team, this PR is quite large, it'll take some time to finish the review. we'll make comments as we go through the changes, instead of collecting all the comments and release in one go...

Nic-Ma

Thanks for your great work on this topic.
I haven't finished review as this PR is very big.
Put several comments inline for discussion first.
Thanks.

examples/classification_3d/densenet_training_array.py

monai/data/csv_saver.py

monai/data/dataset.py

examples/segmentation_3d/unet_training_dict.py

hjmjohnson · 2020-06-03T12:53:30Z

@wyli @Nic-Ma I am concerned that if a PR like this is evaluated for every place where type-hinting could be better, that it will consume a huge amount of time resolving merge conflicts in the future.

We have 3 developers working on making typehint changes for the next week, and are attempting to have a typehint related PR every day with incremental improvements.

My fear is that if we try to do an entire weeks worth of changes before evolutionary improvements towards the final solution that we will need to spend extraordinary efforts resolving merge conflicts.

wyli · 2020-06-03T13:03:21Z

My fear is that if we try to do an entire weeks worth of changes before evolutionary improvements towards the final solution that we will need to spend extraordinary efforts resolving merge conflicts.

true, how about we split the commits of this PR into multiple PRs? I believe many of them are independent changes

hjmjohnson · 2020-06-03T13:43:41Z

@wyli I understand. I will work on that now. Let's remember that time taken to break these items apart is time taken away from the end goals. You will soon find many small PR's coming through.

monai/_version.py

docs/source/conf.py

Nic-Ma · 2020-06-03T14:48:04Z

Hi @hjmjohnson and @wyli ,

I agree with you guys to split the PR into smaller ones.
We should not merge a "big" but "draft" PR, it's very dangerous and unnecessary.
Maybe you can ask your 3 developers to submit 3 PRs for 3 different code folders, then let's review and refine them until merging, I think most of them are independent.
After merged, then try to submit another 3 PRs for another 3 folders in iterations.

What do you think?

Thanks.

hjmjohnson · 2020-06-03T14:58:04Z

These are not completely independent. Type hinting is good precisely because it can identify unforeseen relationships between different modules.

These are NOT independent. I've spent about 2 hours identifying the checks that need to be disabled in order to create a smaller patch set that can get us started.

I expect to spend all day today breaking this into smaller units to be reviewed.

NOTE: I do not think this is a "DRAFT", It is the first completed chapter.

Nic-Ma · 2020-06-03T15:04:32Z

Hi @hjmjohnson ,

Plan sounds good!
Thanks for your explanation.

hjmjohnson · 2020-06-03T15:13:22Z

Please see #477 for the critical first step necessary for building other less dependant PR's.

One often wants to run the very fast codeformatting tests without running any of the unit tests. Add a commands to allow this behavior with "./runtests.sh [--black|--pytype|--mypy|--flake8]". One often wants the `black` formatting tool to automatically fix formatting violations without requiring explicit manual call. Use "./runtests.sh --black-fix" Add mypy static checking tool to monai Run the complete mypy static checking tool during evaluations of code format style checks. Related to Project-MONAI#476 subtask for enable mypy Expand the documentation for runtest.sh script. Provide more expressive documentation for `-h` The dryrun request to print the commands used for static and format checking without actually running the commands.

Related to Project-MONAI#495

Related to Project-MONAI#494

One often wants to run the very fast codeformatting tests without running any of the unit tests. Add a sub-command to allow this behavior with `./runtests.sh --codeformat-only`. One often wants the `black` formatting tool to automatically fix formatting violations without requiring explicit manual call. These two subcommand features allow one to write scripts that allow walking through a series of refactorings addressing modifications one by one.

These types were injected by pyre with manual review.

Co-authored-by: Cameron Trentz <[email protected]>

https://www.python.org/dev/peps/pep-0484/#the-numeric-tower Co-authored-by: Cameron Trentz <[email protected]>

These types were injected by pyre with manual review.

Co-authored-by: Cameron Trentz <[email protected]>

(expression has type "float", variable has type "int") Co-authored-by: Cameron Trentz <[email protected]>

From compose.py Co-authored-by: Cameron Trentz <[email protected]>

Co-authored-by: Cameron Trentz <[email protected]>

… expected "float" Co-authored-by: Cameron Trentz <[email protected]>

(expression has type "List[Any]", variable has type "None") Co-authored-by: Cameron Trentz <[email protected]>

"Union[int, float, Tuple[Any, ...], List[Any]]" is not indexable Co-authored-by: Cameron Trentz <[email protected]>

(expression has type "Union[Tuple[Any, ...], List[Any]]", variable has type "Tuple[float, Any]") (expression has type "Tuple[Any, ...]", variable has type "List[Any]") (expression has type "Union[Tuple[Any, ...], List[Any]]", variable has type "Tuple[float, Any]") (expression has type "PNGSaver", variable has type "NiftiSaver") Co-authored-by: Cameron Trentz <[email protected]>

for + ("List[Any]" and "Tuple[Any]") Co-authored-by: Cameron Trentz <[email protected]>

Co-authored-by: Cameron Trentz <[email protected]>

Corner case warnings are suppressed until more beneficial changes are incorporated.

hjmjohnson · 2020-06-09T20:43:22Z

Abandoning. This work will now need to be redone it is too far diverged to justify working on it.

Nic-Ma reviewed Jun 3, 2020

View reviewed changes

.pyre_configuration Outdated Show resolved Hide resolved

Nic-Ma reviewed Jun 3, 2020

View reviewed changes

.pyre/pyre.stderr Outdated Show resolved Hide resolved

hjmjohnson force-pushed the 20200602-type-hints branch from 4c38c55 to 8f0efdd Compare June 3, 2020 12:14

wyli reviewed Jun 3, 2020

View reviewed changes

Nic-Ma reviewed Jun 3, 2020

View reviewed changes

examples/classification_3d/densenet_training_array.py Show resolved Hide resolved

monai/data/csv_saver.py Show resolved Hide resolved

monai/data/dataset.py Outdated Show resolved Hide resolved

examples/segmentation_3d/unet_training_dict.py Show resolved Hide resolved

hjmjohnson force-pushed the 20200602-type-hints branch from 8f0efdd to ccb60f4 Compare June 3, 2020 12:48

wyli reviewed Jun 3, 2020

View reviewed changes

monai/_version.py Outdated Show resolved Hide resolved

docs/source/conf.py Outdated Show resolved Hide resolved

hjmjohnson changed the title ~~20200602 type hints~~ WIP: 20200602 type hints (Being broken apart into subsets) Jun 3, 2020

hjmjohnson force-pushed the 20200602-type-hints branch from ccb60f4 to 8f414d2 Compare June 3, 2020 18:49

This was referenced Jun 3, 2020

476 - 20200602 type hints part3 #480

Closed

WIP - Enable pytype #483

Closed

hjmjohnson changed the title ~~WIP: 20200602 type hints (Being broken apart into subsets)~~ WIP: 476 - 20200602 type hints (Being broken apart into subsets) Jun 3, 2020

hjmjohnson force-pushed the 20200602-type-hints branch 2 times, most recently from 0db0311 to 71fa8d8 Compare June 5, 2020 16:13

hjmjohnson and others added 9 commits June 5, 2020 18:23

Temporarily add branch needed to trigger CI of this codebase.

207d570

Suppress mypy warnings about incompatible superclass signature

1a796f0

Related to Project-MONAI#495

Suppress mypy error: Cannot assign to a method

6902a57

Related to Project-MONAI#494

Mypy potential type mismatch resolved with assertion.

703daab

Mypy type ambiguity resolved.

92ef4a0

enable pytype ci

e172e4d

Indicate that the optional type returned must be an ndarray.

df47921

hjmjohnson and others added 26 commits June 5, 2020 22:02

Adding asserts to avoid None object attribute access failures.

136a5e6

Trivially determined type annotations added

9f03419

These types were injected by pyre with manual review.

remove (float) from docstring, add type hints

87f6a7f

Co-authored-by: Cameron Trentz <[email protected]>

remove (str) from docstring, add type hints

0e2caab

Co-authored-by: Cameron Trentz <[email protected]>

remove (int) from docstring, add type hints

625d6f7

Co-authored-by: Cameron Trentz <[email protected]>

remove (bool) from docstring, add type hints

53fd86f

Co-authored-by: Cameron Trentz <[email protected]>

remove (int or float) from docstring, add type hints

4274804

https://www.python.org/dev/peps/pep-0484/#the-numeric-tower Co-authored-by: Cameron Trentz <[email protected]>

Silence mypy warning false negative.

0f12ffd

Silence mypy warning about mismatch types

9855b00

Trivially determined type annotations added

54a7332

These types were injected by pyre with manual review.

add interp_order type hints

b9a4328

Co-authored-by: Cameron Trentz <[email protected]>

Fix inconsistent argument naming for order vs interp_order

4cb11e0

Fix error: Incompatible types in assignment

18becef

(expression has type "float", variable has type "int") Co-authored-by: Cameron Trentz <[email protected]>

Fix error: "get_seed" does not return a value

90ccef7

From compose.py Co-authored-by: Cameron Trentz <[email protected]>

Fix error: Item "None" of "Optional[Logger]" has no attribute "info"

fa76651

Co-authored-by: Cameron Trentz <[email protected]>

fix error: "List[Any]" has no attribute "shape"

395c9b4

Co-authored-by: Cameron Trentz <[email protected]>

Fix error: Argument "angle" to "Rotate" has incompatible type "None";…

0acb1ba

… expected "float" Co-authored-by: Cameron Trentz <[email protected]>

Fix error: Incompatible types in assignment

38b4a7e

(expression has type "List[Any]", variable has type "None") Co-authored-by: Cameron Trentz <[email protected]>

Fix error: Value of type is not indexable

bbdfce8

"Union[int, float, Tuple[Any, ...], List[Any]]" is not indexable Co-authored-by: Cameron Trentz <[email protected]>

Fix error: Unsupported operand types

e7b3955

for + ("List[Any]" and "Tuple[Any]") Co-authored-by: Cameron Trentz <[email protected]>

fix error: imported but unused

3738910

Co-authored-by: Cameron Trentz <[email protected]>

Add MONAINumpyDataType data TypeVar for consistency.

d5774d3

Force matching superclass function signature.

aeda297

Fix type inconsistencies for testing suite.

2eab7f2

Suppress low priority pytype warning

1644fe3

Corner case warnings are suppressed until more beneficial changes are incorporated.

hjmjohnson force-pushed the 20200602-type-hints branch from 71fa8d8 to 1644fe3 Compare June 6, 2020 12:34

hjmjohnson closed this Jun 9, 2020

hjmjohnson deleted the 20200602-type-hints branch June 15, 2020 15:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

WIP: 476 - 20200602 type hints (Being broken apart into subsets) #473

WIP: 476 - 20200602 type hints (Being broken apart into subsets) #473

Uh oh!

hjmjohnson commented Jun 3, 2020 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

wyli left a comment

Uh oh!

Nic-Ma left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hjmjohnson commented Jun 3, 2020

Uh oh!

wyli commented Jun 3, 2020

Uh oh!

hjmjohnson commented Jun 3, 2020

Uh oh!

Uh oh!

Uh oh!

Nic-Ma commented Jun 3, 2020 •

edited

Loading

Uh oh!

hjmjohnson commented Jun 3, 2020

Uh oh!

Nic-Ma commented Jun 3, 2020

Uh oh!

hjmjohnson commented Jun 3, 2020

Uh oh!

hjmjohnson commented Jun 9, 2020

Uh oh!

Uh oh!

WIP: 476 - 20200602 type hints (Being broken apart into subsets) #473

WIP: 476 - 20200602 type hints (Being broken apart into subsets) #473

Uh oh!

Conversation

hjmjohnson commented Jun 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Status

Types of changes

Uh oh!

Uh oh!

Uh oh!

wyli left a comment

Choose a reason for hiding this comment

Uh oh!

Nic-Ma left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hjmjohnson commented Jun 3, 2020

Uh oh!

wyli commented Jun 3, 2020

Uh oh!

hjmjohnson commented Jun 3, 2020

Uh oh!

Uh oh!

Uh oh!

Nic-Ma commented Jun 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hjmjohnson commented Jun 3, 2020

Uh oh!

Nic-Ma commented Jun 3, 2020

Uh oh!

hjmjohnson commented Jun 3, 2020

Uh oh!

hjmjohnson commented Jun 9, 2020

Uh oh!

Uh oh!

hjmjohnson commented Jun 3, 2020 •

edited

Loading

Nic-Ma commented Jun 3, 2020 •

edited

Loading