(monarch_hyperactor) Create python binding for a RemoteAllocator that takes a list of remote channel addresses #170
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D75928565 |
kiukchung
force-pushed
the
export-D75928565
branch
from
c215531 to
2422e42
Compare
kiukchung
added a commit
that referenced
this pull request
Jun 5, 2025
… takes a list of remote channel addresses (#170) Summary: To support multi-node actor meshes in OSS without having to write a custom allocator for each scheduler (e.g. `SlurmAllocator`, `KubernetesAllocator`) we take advantage of the infrastructure we already have in TorchX and TorchElastic. This Diff creates Python bindings for `RemoteAllocatorBase` that takes a list of server addresses (in channel_addr format - e.g. `metatls!devgpu032.nha1.facebook.com:26600` or `tcp!devgpu032.nha1.facebook.com:26601`) of remote-process-allocator server and connects to it. The internals reuse existing `RemoteProcessAlloc` with a custom `PyRemoteProcessAllocInitializer` that simply returns a `Vec<RemoteProcessAllocHost>` given the user provided list of server addresses. Recommended to start the review at `monarch/python/tests/test_allocator.py` to get a sense of what the API/Usage looks like. The next diff will provide a function that gets the list of server addresses given a job-id (more specifically a monarch server handle of the form `{scheduler}://{namespace}/{job_id}` e.g. `slurm://default/monarch-kiuk-123`) and returns an Allocator that can be used to create a `ProcMesh` as usual. NOTE: WIP fixing type-checking failures so ignore those... Differential Revision: D75928565
|
This pull request was exported from Phabricator. Differential Revision: D75928565 |
kiukchung
added a commit
that referenced
this pull request
Jun 5, 2025
… takes a list of remote channel addresses (#170) Summary: Pull Request resolved: #170 To support multi-node actor meshes in OSS without having to write a custom allocator for each scheduler (e.g. `SlurmAllocator`, `KubernetesAllocator`) we take advantage of the infrastructure we already have in TorchX and TorchElastic. This Diff creates Python bindings for `RemoteAllocatorBase` that takes a list of server addresses (in channel_addr format - e.g. `metatls!devgpu032.nha1.facebook.com:26600` or `tcp!devgpu032.nha1.facebook.com:26601`) of remote-process-allocator server and connects to it. The internals reuse existing `RemoteProcessAlloc` with a custom `PyRemoteProcessAllocInitializer` that simply returns a `Vec<RemoteProcessAllocHost>` given the user provided list of server addresses. Recommended to start the review at `monarch/python/tests/test_allocator.py` to get a sense of what the API/Usage looks like. The next diff will provide a function that gets the list of server addresses given a job-id (more specifically a monarch server handle of the form `{scheduler}://{namespace}/{job_id}` e.g. `slurm://default/monarch-kiuk-123`) and returns an Allocator that can be used to create a `ProcMesh` as usual. NOTE: WIP fixing type-checking failures so ignore those... Differential Revision: D75928565
kiukchung
force-pushed
the
export-D75928565
branch
from
2422e42 to
4e0e34a
Compare
|
This pull request was exported from Phabricator. Differential Revision: D75928565 |
kiukchung
added a commit
that referenced
this pull request
Jun 5, 2025
… takes a list of remote channel addresses (#170) Summary: Pull Request resolved: #170 To support multi-node actor meshes in OSS without having to write a custom allocator for each scheduler (e.g. `SlurmAllocator`, `KubernetesAllocator`) we take advantage of the infrastructure we already have in TorchX and TorchElastic. This Diff creates Python bindings for `RemoteAllocatorBase` that takes a list of server addresses (in channel_addr format - e.g. `metatls!devgpu032.nha1.facebook.com:26600` or `tcp!devgpu032.nha1.facebook.com:26601`) of remote-process-allocator server and connects to it. The internals reuse existing `RemoteProcessAlloc` with a custom `PyRemoteProcessAllocInitializer` that simply returns a `Vec<RemoteProcessAllocHost>` given the user provided list of server addresses. Recommended to start the review at `monarch/python/tests/test_allocator.py` to get a sense of what the API/Usage looks like. The next diff will provide a function that gets the list of server addresses given a job-id (more specifically a monarch server handle of the form `{scheduler}://{namespace}/{job_id}` e.g. `slurm://default/monarch-kiuk-123`) and returns an Allocator that can be used to create a `ProcMesh` as usual. NOTE: WIP fixing type-checking failures so ignore those... Differential Revision: D75928565
kiukchung
force-pushed
the
export-D75928565
branch
from
4e0e34a to
4be469b
Compare
facebook-github-bot
pushed a commit
that referenced
this pull request
Jun 5, 2025
… takes a list of remote channel addresses (#170) Summary: To support multi-node actor meshes in OSS without having to write a custom allocator for each scheduler (e.g. `SlurmAllocator`, `KubernetesAllocator`) we take advantage of the infrastructure we already have in TorchX and TorchElastic. This Diff creates Python bindings for `RemoteAllocatorBase` that takes a list of server addresses (in channel_addr format - e.g. `metatls!devgpu032.nha1.facebook.com:26600` or `tcp!devgpu032.nha1.facebook.com:26601`) of remote-process-allocator server and connects to it. The internals reuse existing `RemoteProcessAlloc` with a custom `PyRemoteProcessAllocInitializer` that simply returns a `Vec<RemoteProcessAllocHost>` given the user provided list of server addresses. Recommended to start the review at `monarch/python/tests/test_allocator.py` to get a sense of what the API/Usage looks like. The next diff will provide a function that gets the list of server addresses given a job-id (more specifically a monarch server handle of the form `{scheduler}://{namespace}/{job_id}` e.g. `slurm://default/monarch-kiuk-123`) and returns an Allocator that can be used to create a `ProcMesh` as usual. NOTE: WIP fixing type-checking failures so ignore those... Differential Revision: D75928565
facebook-github-bot
force-pushed
the
export-D75928565
branch
from
4be469b to
e1d1038
Compare
|
This pull request was exported from Phabricator. Differential Revision: D75928565 |
facebook-github-bot
pushed a commit
that referenced
this pull request
Jun 5, 2025
… takes a list of remote channel addresses (#170) Summary: See: P1833193535 for how the user-facing UX would look like. NOTE: Recommended to start the review at `monarch/python/tests/test_allocator.py` to get a sense of what the API/Usage looks like. NOTE: hyperactor's ChannelAddr can be represented in string format such as: `tcp!127.0.0.1:26600` or `metatls!devgpu001.pci.facebook.com:26600` which includes all the necessary information to create a `Channel`. Unfortunately, the current `RemoteAllocator` related interfaces take `transport` (`ChannelTransport`), `port`, and a list of `hostnames`, and applies the same transport and port to all. This isn't ideal (especially for flexibility in deployment and testing). So the python bindings take a list of channel address strings rather than a list of hostnames. To support multi-node actor meshes in OSS without having to write a custom allocator for each scheduler (e.g. `SlurmAllocator`, `KubernetesAllocator`) we take advantage of the infrastructure we already have in TorchX and TorchElastic. This Diff creates Python bindings for `RemoteAllocatorBase` that takes a list of server addresses (in channel_addr format - e.g. `metatls!devgpu032.nha1.facebook.com:26600` or `tcp!devgpu032.nha1.facebook.com:26601`) of remote-process-allocator server and connects to it. The internals reuse existing `RemoteProcessAlloc` with a custom `PyRemoteProcessAllocInitializer` that simply returns a `Vec<RemoteProcessAllocHost>` given the user provided list of server addresses. ## Next Steps: 1. [1/2] Add hostnames to `monarch.tools.mesh_spec.MeshSpec` struct + ability to query hostnames given a running job. 2. [2/2] Make it possible to run 2x remote process allocators (each on its own port) on MAST Differential Revision: D75928565
facebook-github-bot
force-pushed
the
export-D75928565
branch
from
e1d1038 to
a3ab415
Compare
|
This pull request was exported from Phabricator. Differential Revision: D75928565 |
kiukchung
added a commit
that referenced
this pull request
Jun 10, 2025
… takes a list of remote channel addresses (#170) Summary: See: P1833193535 for how the user-facing UX would look like. NOTE: Recommended to start the review at `monarch/python/tests/test_allocator.py` to get a sense of what the API/Usage looks like. NOTE: hyperactor's ChannelAddr can be represented in string format such as: `tcp!127.0.0.1:26600` or `metatls!devgpu001.pci.facebook.com:26600` which includes all the necessary information to create a `Channel`. Unfortunately, the current `RemoteAllocator` related interfaces take `transport` (`ChannelTransport`), `port`, and a list of `hostnames`, and applies the same transport and port to all. This isn't ideal (especially for flexibility in deployment and testing). So the python bindings take a list of channel address strings rather than a list of hostnames. To support multi-node actor meshes in OSS without having to write a custom allocator for each scheduler (e.g. `SlurmAllocator`, `KubernetesAllocator`) we take advantage of the infrastructure we already have in TorchX and TorchElastic. This Diff creates Python bindings for `RemoteAllocatorBase` that takes a list of server addresses (in channel_addr format - e.g. `metatls!devgpu032.nha1.facebook.com:26600` or `tcp!devgpu032.nha1.facebook.com:26601`) of remote-process-allocator server and connects to it. The internals reuse existing `RemoteProcessAlloc` with a custom `PyRemoteProcessAllocInitializer` that simply returns a `Vec<RemoteProcessAllocHost>` given the user provided list of server addresses. ## Next Steps: 1. [1/2] Add hostnames to `monarch.tools.mesh_spec.MeshSpec` struct + ability to query hostnames given a running job. 2. [2/2] Make it possible to run 2x remote process allocators (each on its own port) on MAST Differential Revision: D75928565
kiukchung
force-pushed
the
export-D75928565
branch
from
a3ab415 to
b2af73d
Compare
kiukchung
added a commit
that referenced
this pull request
Jun 10, 2025
… takes a list of remote channel addresses (#170) Summary: See: P1833193535 for how the user-facing UX would look like. NOTE: Recommended to start the review at `monarch/python/tests/test_allocator.py` to get a sense of what the API/Usage looks like. NOTE: hyperactor's ChannelAddr can be represented in string format such as: `tcp!127.0.0.1:26600` or `metatls!devgpu001.pci.facebook.com:26600` which includes all the necessary information to create a `Channel`. Unfortunately, the current `RemoteAllocator` related interfaces take `transport` (`ChannelTransport`), `port`, and a list of `hostnames`, and applies the same transport and port to all. This isn't ideal (especially for flexibility in deployment and testing). So the python bindings take a list of channel address strings rather than a list of hostnames. To support multi-node actor meshes in OSS without having to write a custom allocator for each scheduler (e.g. `SlurmAllocator`, `KubernetesAllocator`) we take advantage of the infrastructure we already have in TorchX and TorchElastic. This Diff creates Python bindings for `RemoteAllocatorBase` that takes a list of server addresses (in channel_addr format - e.g. `metatls!devgpu032.nha1.facebook.com:26600` or `tcp!devgpu032.nha1.facebook.com:26601`) of remote-process-allocator server and connects to it. The internals reuse existing `RemoteProcessAlloc` with a custom `PyRemoteProcessAllocInitializer` that simply returns a `Vec<RemoteProcessAllocHost>` given the user provided list of server addresses. ## Next Steps: 1. [1/2] Add hostnames to `monarch.tools.mesh_spec.MeshSpec` struct + ability to query hostnames given a running job. 2. [2/2] Make it possible to run 2x remote process allocators (each on its own port) on MAST Differential Revision: D75928565
kiukchung
force-pushed
the
export-D75928565
branch
from
b2af73d to
85b0961
Compare
|
This pull request was exported from Phabricator. Differential Revision: D75928565 |
kiukchung
added a commit
that referenced
this pull request
Jun 10, 2025
… takes a list of remote channel addresses (#170) Summary: Pull Request resolved: #170 See: P1833193535 for how the user-facing UX would look like. NOTE: Recommended to start the review at `monarch/python/tests/test_allocator.py` to get a sense of what the API/Usage looks like. NOTE: hyperactor's ChannelAddr can be represented in string format such as: `tcp!127.0.0.1:26600` or `metatls!devgpu001.pci.facebook.com:26600` which includes all the necessary information to create a `Channel`. Unfortunately, the current `RemoteAllocator` related interfaces take `transport` (`ChannelTransport`), `port`, and a list of `hostnames`, and applies the same transport and port to all. This isn't ideal (especially for flexibility in deployment and testing). So the python bindings take a list of channel address strings rather than a list of hostnames. To support multi-node actor meshes in OSS without having to write a custom allocator for each scheduler (e.g. `SlurmAllocator`, `KubernetesAllocator`) we take advantage of the infrastructure we already have in TorchX and TorchElastic. This Diff creates Python bindings for `RemoteAllocatorBase` that takes a list of server addresses (in channel_addr format - e.g. `metatls!devgpu032.nha1.facebook.com:26600` or `tcp!devgpu032.nha1.facebook.com:26601`) of remote-process-allocator server and connects to it. The internals reuse existing `RemoteProcessAlloc` with a custom `PyRemoteProcessAllocInitializer` that simply returns a `Vec<RemoteProcessAllocHost>` given the user provided list of server addresses. ## Next Steps: 1. [1/2] Add hostnames to `monarch.tools.mesh_spec.MeshSpec` struct + ability to query hostnames given a running job. 2. [2/2] Make it possible to run 2x remote process allocators (each on its own port) on MAST Reviewed By: technicianted Differential Revision: D75928565
kiukchung
force-pushed
the
export-D75928565
branch
from
85b0961 to
e90c354
Compare
kiukchung
added a commit
that referenced
this pull request
Jun 10, 2025
… takes a list of remote channel addresses (#170) Summary: See: P1833193535 for how the user-facing UX would look like. NOTE: Recommended to start the review at `monarch/python/tests/test_allocator.py` to get a sense of what the API/Usage looks like. NOTE: hyperactor's ChannelAddr can be represented in string format such as: `tcp!127.0.0.1:26600` or `metatls!devgpu001.pci.facebook.com:26600` which includes all the necessary information to create a `Channel`. Unfortunately, the current `RemoteAllocator` related interfaces take `transport` (`ChannelTransport`), `port`, and a list of `hostnames`, and applies the same transport and port to all. This isn't ideal (especially for flexibility in deployment and testing). So the python bindings take a list of channel address strings rather than a list of hostnames. To support multi-node actor meshes in OSS without having to write a custom allocator for each scheduler (e.g. `SlurmAllocator`, `KubernetesAllocator`) we take advantage of the infrastructure we already have in TorchX and TorchElastic. This Diff creates Python bindings for `RemoteAllocatorBase` that takes a list of server addresses (in channel_addr format - e.g. `metatls!devgpu032.nha1.facebook.com:26600` or `tcp!devgpu032.nha1.facebook.com:26601`) of remote-process-allocator server and connects to it. The internals reuse existing `RemoteProcessAlloc` with a custom `PyRemoteProcessAllocInitializer` that simply returns a `Vec<RemoteProcessAllocHost>` given the user provided list of server addresses. ## Next Steps: 1. [1/2] Add hostnames to `monarch.tools.mesh_spec.MeshSpec` struct + ability to query hostnames given a running job. 2. [2/2] Make it possible to run 2x remote process allocators (each on its own port) on MAST Reviewed By: technicianted Differential Revision: D75928565
kiukchung
force-pushed
the
export-D75928565
branch
from
e90c354 to
53d82a8
Compare
kiukchung
added a commit
that referenced
this pull request
Jun 10, 2025
… takes a list of remote channel addresses (#170) Summary: See: P1833193535 for how the user-facing UX would look like. NOTE: Recommended to start the review at `monarch/python/tests/test_allocator.py` to get a sense of what the API/Usage looks like. NOTE: hyperactor's ChannelAddr can be represented in string format such as: `tcp!127.0.0.1:26600` or `metatls!devgpu001.pci.facebook.com:26600` which includes all the necessary information to create a `Channel`. Unfortunately, the current `RemoteAllocator` related interfaces take `transport` (`ChannelTransport`), `port`, and a list of `hostnames`, and applies the same transport and port to all. This isn't ideal (especially for flexibility in deployment and testing). So the python bindings take a list of channel address strings rather than a list of hostnames. To support multi-node actor meshes in OSS without having to write a custom allocator for each scheduler (e.g. `SlurmAllocator`, `KubernetesAllocator`) we take advantage of the infrastructure we already have in TorchX and TorchElastic. This Diff creates Python bindings for `RemoteAllocatorBase` that takes a list of server addresses (in channel_addr format - e.g. `metatls!devgpu032.nha1.facebook.com:26600` or `tcp!devgpu032.nha1.facebook.com:26601`) of remote-process-allocator server and connects to it. The internals reuse existing `RemoteProcessAlloc` with a custom `PyRemoteProcessAllocInitializer` that simply returns a `Vec<RemoteProcessAllocHost>` given the user provided list of server addresses. ## Next Steps: 1. [1/2] Add hostnames to `monarch.tools.mesh_spec.MeshSpec` struct + ability to query hostnames given a running job. 2. [2/2] Make it possible to run 2x remote process allocators (each on its own port) on MAST Reviewed By: technicianted Differential Revision: D75928565
kiukchung
force-pushed
the
export-D75928565
branch
from
53d82a8 to
e5515fb
Compare
|
This pull request was exported from Phabricator. Differential Revision: D75928565 |
kiukchung
added a commit
that referenced
this pull request
Jun 10, 2025
… takes a list of remote channel addresses (#170) Summary: Pull Request resolved: #170 See: P1833193535 for how the user-facing UX would look like. NOTE: Recommended to start the review at `monarch/python/tests/test_allocator.py` to get a sense of what the API/Usage looks like. NOTE: hyperactor's ChannelAddr can be represented in string format such as: `tcp!127.0.0.1:26600` or `metatls!devgpu001.pci.facebook.com:26600` which includes all the necessary information to create a `Channel`. Unfortunately, the current `RemoteAllocator` related interfaces take `transport` (`ChannelTransport`), `port`, and a list of `hostnames`, and applies the same transport and port to all. This isn't ideal (especially for flexibility in deployment and testing). So the python bindings take a list of channel address strings rather than a list of hostnames. To support multi-node actor meshes in OSS without having to write a custom allocator for each scheduler (e.g. `SlurmAllocator`, `KubernetesAllocator`) we take advantage of the infrastructure we already have in TorchX and TorchElastic. This Diff creates Python bindings for `RemoteAllocatorBase` that takes a list of server addresses (in channel_addr format - e.g. `metatls!devgpu032.nha1.facebook.com:26600` or `tcp!devgpu032.nha1.facebook.com:26601`) of remote-process-allocator server and connects to it. The internals reuse existing `RemoteProcessAlloc` with a custom `PyRemoteProcessAllocInitializer` that simply returns a `Vec<RemoteProcessAllocHost>` given the user provided list of server addresses. ## Next Steps: 1. [1/2] Add hostnames to `monarch.tools.mesh_spec.MeshSpec` struct + ability to query hostnames given a running job. 2. [2/2] Make it possible to run 2x remote process allocators (each on its own port) on MAST Reviewed By: technicianted Differential Revision: D75928565
kiukchung
force-pushed
the
export-D75928565
branch
from
e5515fb to
cc962c4
Compare
|
This pull request was exported from Phabricator. Differential Revision: D75928565 |
kiukchung
added a commit
that referenced
this pull request
Jun 10, 2025
… takes a list of remote channel addresses (#170) Summary: Pull Request resolved: #170 See: P1833193535 for how the user-facing UX would look like. NOTE: Recommended to start the review at `monarch/python/tests/test_allocator.py` to get a sense of what the API/Usage looks like. NOTE: hyperactor's ChannelAddr can be represented in string format such as: `tcp!127.0.0.1:26600` or `metatls!devgpu001.pci.facebook.com:26600` which includes all the necessary information to create a `Channel`. Unfortunately, the current `RemoteAllocator` related interfaces take `transport` (`ChannelTransport`), `port`, and a list of `hostnames`, and applies the same transport and port to all. This isn't ideal (especially for flexibility in deployment and testing). So the python bindings take a list of channel address strings rather than a list of hostnames. To support multi-node actor meshes in OSS without having to write a custom allocator for each scheduler (e.g. `SlurmAllocator`, `KubernetesAllocator`) we take advantage of the infrastructure we already have in TorchX and TorchElastic. This Diff creates Python bindings for `RemoteAllocatorBase` that takes a list of server addresses (in channel_addr format - e.g. `metatls!devgpu032.nha1.facebook.com:26600` or `tcp!devgpu032.nha1.facebook.com:26601`) of remote-process-allocator server and connects to it. The internals reuse existing `RemoteProcessAlloc` with a custom `PyRemoteProcessAllocInitializer` that simply returns a `Vec<RemoteProcessAllocHost>` given the user provided list of server addresses. ## Next Steps: 1. [1/2] Add hostnames to `monarch.tools.mesh_spec.MeshSpec` struct + ability to query hostnames given a running job. 2. [2/2] Make it possible to run 2x remote process allocators (each on its own port) on MAST Reviewed By: technicianted Differential Revision: D75928565
kiukchung
force-pushed
the
export-D75928565
branch
from
cc962c4 to
e0a1354
Compare
facebook-github-bot
pushed a commit
that referenced
this pull request
Jun 10, 2025
… takes a list of remote channel addresses (#170) Summary: See: P1833193535 for how the user-facing UX would look like. NOTE: Recommended to start the review at `monarch/python/tests/test_allocator.py` to get a sense of what the API/Usage looks like. NOTE: hyperactor's ChannelAddr can be represented in string format such as: `tcp!127.0.0.1:26600` or `metatls!devgpu001.pci.facebook.com:26600` which includes all the necessary information to create a `Channel`. Unfortunately, the current `RemoteAllocator` related interfaces take `transport` (`ChannelTransport`), `port`, and a list of `hostnames`, and applies the same transport and port to all. This isn't ideal (especially for flexibility in deployment and testing). So the python bindings take a list of channel address strings rather than a list of hostnames. To support multi-node actor meshes in OSS without having to write a custom allocator for each scheduler (e.g. `SlurmAllocator`, `KubernetesAllocator`) we take advantage of the infrastructure we already have in TorchX and TorchElastic. This Diff creates Python bindings for `RemoteAllocatorBase` that takes a list of server addresses (in channel_addr format - e.g. `metatls!devgpu032.nha1.facebook.com:26600` or `tcp!devgpu032.nha1.facebook.com:26601`) of remote-process-allocator server and connects to it. The internals reuse existing `RemoteProcessAlloc` with a custom `PyRemoteProcessAllocInitializer` that simply returns a `Vec<RemoteProcessAllocHost>` given the user provided list of server addresses. ## Next Steps: 1. [1/2] Add hostnames to `monarch.tools.mesh_spec.MeshSpec` struct + ability to query hostnames given a running job. 2. [2/2] Make it possible to run 2x remote process allocators (each on its own port) on MAST Reviewed By: technicianted Differential Revision: D75928565
facebook-github-bot
force-pushed
the
export-D75928565
branch
from
e0a1354 to
a456f48
Compare
|
This pull request was exported from Phabricator. Differential Revision: D75928565 |
1 similar comment
|
This pull request was exported from Phabricator. Differential Revision: D75928565 |
kiukchung
added a commit
that referenced
this pull request
Jun 10, 2025
… takes a list of remote channel addresses (#170) Summary: Pull Request resolved: #170 See: P1833193535 for how the user-facing UX would look like. NOTE: Recommended to start the review at `monarch/python/tests/test_allocator.py` to get a sense of what the API/Usage looks like. NOTE: hyperactor's ChannelAddr can be represented in string format such as: `tcp!127.0.0.1:26600` or `metatls!devgpu001.pci.facebook.com:26600` which includes all the necessary information to create a `Channel`. Unfortunately, the current `RemoteAllocator` related interfaces take `transport` (`ChannelTransport`), `port`, and a list of `hostnames`, and applies the same transport and port to all. This isn't ideal (especially for flexibility in deployment and testing). So the python bindings take a list of channel address strings rather than a list of hostnames. To support multi-node actor meshes in OSS without having to write a custom allocator for each scheduler (e.g. `SlurmAllocator`, `KubernetesAllocator`) we take advantage of the infrastructure we already have in TorchX and TorchElastic. This Diff creates Python bindings for `RemoteAllocatorBase` that takes a list of server addresses (in channel_addr format - e.g. `metatls!devgpu032.nha1.facebook.com:26600` or `tcp!devgpu032.nha1.facebook.com:26601`) of remote-process-allocator server and connects to it. The internals reuse existing `RemoteProcessAlloc` with a custom `PyRemoteProcessAllocInitializer` that simply returns a `Vec<RemoteProcessAllocHost>` given the user provided list of server addresses. ## Next Steps: 1. [1/2] Add hostnames to `monarch.tools.mesh_spec.MeshSpec` struct + ability to query hostnames given a running job. 2. [2/2] Make it possible to run 2x remote process allocators (each on its own port) on MAST Reviewed By: technicianted Differential Revision: D75928565
kiukchung
force-pushed
the
export-D75928565
branch
from
a456f48 to
7052fa5
Compare
… takes a list of remote channel addresses (#170) Summary: See: P1833193535 for how the user-facing UX would look like. NOTE: Recommended to start the review at `monarch/python/tests/test_allocator.py` to get a sense of what the API/Usage looks like. NOTE: hyperactor's ChannelAddr can be represented in string format such as: `tcp!127.0.0.1:26600` or `metatls!devgpu001.pci.facebook.com:26600` which includes all the necessary information to create a `Channel`. Unfortunately, the current `RemoteAllocator` related interfaces take `transport` (`ChannelTransport`), `port`, and a list of `hostnames`, and applies the same transport and port to all. This isn't ideal (especially for flexibility in deployment and testing). So the python bindings take a list of channel address strings rather than a list of hostnames. To support multi-node actor meshes in OSS without having to write a custom allocator for each scheduler (e.g. `SlurmAllocator`, `KubernetesAllocator`) we take advantage of the infrastructure we already have in TorchX and TorchElastic. This Diff creates Python bindings for `RemoteAllocatorBase` that takes a list of server addresses (in channel_addr format - e.g. `metatls!devgpu032.nha1.facebook.com:26600` or `tcp!devgpu032.nha1.facebook.com:26601`) of remote-process-allocator server and connects to it. The internals reuse existing `RemoteProcessAlloc` with a custom `PyRemoteProcessAllocInitializer` that simply returns a `Vec<RemoteProcessAllocHost>` given the user provided list of server addresses. ## Next Steps: 1. [1/2] Add hostnames to `monarch.tools.mesh_spec.MeshSpec` struct + ability to query hostnames given a running job. 2. [2/2] Make it possible to run 2x remote process allocators (each on its own port) on MAST Reviewed By: technicianted Differential Revision: D75928565
facebook-github-bot
force-pushed
the
export-D75928565
branch
from
7052fa5 to
de51bc8
Compare
|
This pull request was exported from Phabricator. Differential Revision: D75928565 |
|
This pull request has been merged in b844326. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
To support multi-node actor meshes in OSS without having to write a custom allocator for each scheduler (e.g.
SlurmAllocator,KubernetesAllocator) we take advantage of the infrastructure we already have in TorchX and TorchElastic.This Diff creates Python bindings for
RemoteAllocatorBasethat takes a list of server addresses (in channel_addr format - e.g.metatls!devgpu032.nha1.facebook.com:26600ortcp!devgpu032.nha1.facebook.com:26601) of remote-process-allocator server and connects to it.The internals reuse existing
RemoteProcessAllocwith a customPyRemoteProcessAllocInitializerthat simply returns aVec<RemoteProcessAllocHost>given the user provided list of server addresses.Recommended to start the review at
monarch/python/tests/test_allocator.pyto get a sense of what the API/Usage looks like.The next diff will provide a function that gets the list of server addresses given a job-id (more specifically a monarch server handle of the form
{scheduler}://{namespace}/{job_id}e.g.slurm://default/monarch-kiuk-123) and returns an Allocator that can be used to create aProcMeshas usual.NOTE: WIP fixing type-checking failures so ignore those...
Differential Revision: D75928565