[3821] Remove deprecated Host Path, NFS, and EBS support for Kubernetes #3825

surahman · 2022-05-01T16:58:11Z

This PR removes support for the configuration of EBS, Host Path, and NFS via the topology configs. Closes #3821.

Tasks:

Volumes are no longer loaded from Configs. As such there is no longer a need to de-duplicate against volumes loaded from Pod Templates and the CLI.

Updated V1Controller to use VolumeFactory for Empty Directory generation.

Updated V1Controller to use VolumeFactory for Host Path generation.

Updated V1Controller to use VolumeFactory for Network File System generation.

Added interfaces: - Volume Mounts - Persistent Volume Claim

…to saadurrahman/3821-Remove-Deprecated-Volumes-K8s

Generating all volume mounts from within Volume Factory for maintainability.

…to saadurrahman/3821-Remove-Deprecated-Volumes-K8s

Test in <testCreatePersistentVolumeClaims> hardened for size.

…to saadurrahman/3821-Remove-Deprecated-Volumes-K8s

surahman · 2022-05-03T16:42:50Z

Good deployment test. Submit and Kill are successful and all required resources are added and removed.

Command

~/bin/heron submit kubernetes ~/.heron/examples/heron-api-examples.jar \
org.apache.heron.examples.api.AckingTopology acking \
--verbose \
--deploy-deactivated \
--config-property heron.kubernetes.executor.pod.template=pod-templ-executor.pod-template-executor.yaml \
--config-property heron.kubernetes.manager.pod.template=pod-templ-manager.pod-template-manager.yaml \
--config-property heron.kubernetes.manager.limits.cpu=2 \
--config-property heron.kubernetes.manager.limits.memory=3 \
--config-property heron.kubernetes.manager.requests.cpu=1 \
--config-property heron.kubernetes.manager.requests.memory=2 \
--config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-dynamic-volume.claimName=OnDemand \
--config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-dynamic-volume.storageClassName=storage-class-name-manager \
--config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-dynamic-volume.accessModes=ReadWriteOnce,ReadOnlyMany \
--config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-dynamic-volume.sizeLimit=256Gi \
--config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-dynamic-volume.volumeMode=Block \
--config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-dynamic-volume.path=path/to/mount/dynamic/volume \
--config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-dynamic-volume.subPath=sub/path/to/mount/dynamic/volume \
\
--config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-static-volume.claimName=OnDemand \
--config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-static-volume.accessModes=ReadWriteOnce,ReadOnlyMany \
--config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-static-volume.sizeLimit=512Gi \
--config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-static-volume.volumeMode=Block \
--config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-static-volume.path=path/to/mount/static/volume \
--config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-static-volume.subPath=sub/path/to/mount/static/volume \
\
--config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-shared-volume.claimName=requested-claim-by-user \
--config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-shared-volume.path=path/to/mount/shared/volume \
--config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-shared-volume.subPath=sub/path/to/mount/shared/volume \
\
--config-property heron.kubernetes.manager.volumes.emptyDir.manager-empty-dir.medium="Memory" \
--config-property heron.kubernetes.manager.volumes.emptyDir.manager-empty-dir.sizeLimit="50Mi" \
--config-property heron.kubernetes.manager.volumes.emptyDir.manager-empty-dir.path="empty/dir/path" \
--config-property heron.kubernetes.manager.volumes.emptyDir.manager-empty-dir.subPath="empty/dir/sub/path" \
--config-property heron.kubernetes.manager.volumes.emptyDir.manager-empty-dir.readOnly="true" \
\
--config-property heron.kubernetes.manager.volumes.hostPath.manager-host-path.type="File" \
--config-property heron.kubernetes.manager.volumes.hostPath.manager-host-path.pathOnHost="/dev/null" \
--config-property heron.kubernetes.manager.volumes.hostPath.manager-host-path.path="host/path/path" \
--config-property heron.kubernetes.manager.volumes.hostPath.manager-host-path.subPath="host/path/sub/path" \
--config-property heron.kubernetes.manager.volumes.hostPath.manager-host-path.readOnly="true" \
\
--config-property heron.kubernetes.manager.volumes.nfs.manager-nfs.server="nfs-server.address" \
--config-property heron.kubernetes.manager.volumes.nfs.manager-nfs.readOnly="true" \
--config-property heron.kubernetes.manager.volumes.nfs.manager-nfs.pathOnNFS="/dev/null" \
--config-property heron.kubernetes.manager.volumes.nfs.manager-nfs.path="nfs/path" \
--config-property heron.kubernetes.manager.volumes.nfs.manager-nfs.subPath="nfs/sub/path" \
--config-property heron.kubernetes.manager.volumes.nfs.manager-nfs.readOnly="true" \
\
--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-dynamic-volume.claimName=OnDemand \
--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-dynamic-volume.storageClassName=storage-class-name-executor \
--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-dynamic-volume.accessModes=ReadWriteOnce,ReadOnlyMany \
--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-dynamic-volume.sizeLimit=256Gi \
--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-dynamic-volume.volumeMode=Block \
--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-dynamic-volume.path=path/to/mount/dynamic/volume \
--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-dynamic-volume.subPath=sub/path/to/mount/dynamic/volume \
\
--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-static-volume.claimName=OnDemand \
--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-static-volume.accessModes=ReadWriteOnce,ReadOnlyMany \
--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-static-volume.sizeLimit=512Gi \
--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-static-volume.volumeMode=Block \
--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-static-volume.path=path/to/mount/static/volume \
--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-static-volume.subPath=sub/path/to/mount/static/volume \
\
--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-shared-volume.claimName=requested-claim-by-user \
--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-shared-volume.path=path/to/mount/shared/volume \
--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-shared-volume.subPath=sub/path/to/mount/shared/volume

Stateful Sets

Manager

Name:               acking-manager
Namespace:          default
CreationTimestamp:  Tue, 03 May 2022 11:33:31 -0400
Selector:           app=heron,topology=acking
Labels:             app=heron
                    topology=acking
Annotations:        <none>
Replicas:           1 desired | 1 total
Update Strategy:    RollingUpdate
  Partition:        0
Pods Status:        0 Running / 1 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:       app=heron
                topology=acking
  Annotations:  prometheus.io/port: 8080
                prometheus.io/scrape: true
  Containers:
   manager:
    Image:       apache/heron:testbuild
    Ports:       6001/TCP, 6002/TCP, 6003/TCP, 6004/TCP, 6005/TCP, 6006/TCP, 6007/TCP, 6008/TCP, 6009/TCP, 7775/TCP, 7776/UDP
    Host Ports:  0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/UDP
    Command:
      sh
      -c
      ./heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0--3944535742942496843.tar.gz . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor --topology-name=acking --topology-id=acking3ae6b6a0-bbb8-45a1-8015-37fd50fec7c6 --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181 --state-manager-root=/heron --state-manager-config-file=./heron-conf/statemgr.yaml --tmanager-binary=./heron-core/bin/heron-tmanager --stmgr-binary=./heron-core/bin/heron-stmgr --metrics-manager-classpath=./heron-core/lib/metricsmgr/* --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)" --classpath=heron-api-examples.jar --heron-internals-config-file=./heron-conf/heron_internals.yaml --override-config-file=./heron-conf/override.yaml --component-ram-map=exclaim1:1073741824,word:1073741824 --component-jvm-opts="" --pkg-type=jar --topology-binary-file=heron-api-examples.jar --heron-java-home=$JAVA_HOME --heron-shell-binary=./heron-core/bin/heron-shell --cluster=kubernetes --role=saad --environment=default --instance-classpath=./heron-core/lib/instance/* --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/* --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*: --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824 --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/* --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003 --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007 --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009
    Limits:
      cpu:     2
      memory:  3
    Requests:
      cpu:     1
      memory:  2
    Environment:
      HOST:                (v1:status.podIP)
      POD_NAME:            (v1:metadata.name)
      var_one_manager:    variable one on manager
      var_three_manager:  variable three on manager
      var_two_manager:    variable two on manager
    Mounts:
      /shared_volume/manager from shared-volume-manager (rw)
      empty/dir/path from manager-empty-dir (ro,path="empty/dir/sub/path")
      host/path/path from manager-host-path (ro,path="host/path/sub/path")
      nfs/path from manager-nfs (ro,path="nfs/sub/path")
      path/to/mount/dynamic/volume from manager-dynamic-volume (rw,path="sub/path/to/mount/dynamic/volume")
      path/to/mount/shared/volume from manager-shared-volume (rw,path="sub/path/to/mount/shared/volume")
      path/to/mount/static/volume from manager-static-volume (rw,path="sub/path/to/mount/static/volume")
   manager-sidecar-container:
    Image:        alpine
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:
      /shared_volume/manager from shared-volume-manager (rw)
  Volumes:
   manager-empty-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  50Mi
   manager-host-path:
    Type:          HostPath (bare host directory volume)
    Path:          /dev/null
    HostPathType:  File
   manager-nfs:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    nfs-server.address
    Path:      /dev/null
    ReadOnly:  true
   manager-shared-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  requested-claim-by-user
    ReadOnly:   false
   shared-volume-manager:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
Volume Claims:
  Name:          manager-static-volume
  StorageClass:  
  Labels:        onDemand=true
                 topology=acking
  Annotations:   <none>
  Capacity:      512Gi
  Access Modes:  [ReadWriteOnce ReadOnlyMany]
  Name:          manager-dynamic-volume
  StorageClass:  storage-class-name-manager
  Labels:        onDemand=true
                 topology=acking
  Annotations:   <none>
  Capacity:      256Gi
  Access Modes:  [ReadWriteOnce ReadOnlyMany]
Events:
  Type    Reason            Age    From                    Message
  ----    ------            ----   ----                    -------
  Normal  SuccessfulCreate  6m38s  statefulset-controller  create Claim manager-static-volume-acking-manager-0 Pod acking-manager-0 in StatefulSet acking-manager success
  Normal  SuccessfulCreate  6m38s  statefulset-controller  create Claim manager-dynamic-volume-acking-manager-0 Pod acking-manager-0 in StatefulSet acking-manager success
  Normal  SuccessfulCreate  6m38s  statefulset-controller  create Pod acking-manager-0 in StatefulSet acking-manager successful

Executor

Name:               acking-manager
Namespace:          default
CreationTimestamp:  Tue, 03 May 2022 11:33:31 -0400
Selector:           app=heron,topology=acking
Labels:             app=heron
                    topology=acking
Annotations:        <none>
Replicas:           1 desired | 1 total
Update Strategy:    RollingUpdate
  Partition:        0
Pods Status:        0 Running / 1 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:       app=heron
                topology=acking
  Annotations:  prometheus.io/port: 8080
                prometheus.io/scrape: true
  Containers:
   manager:
    Image:       apache/heron:testbuild
    Ports:       6001/TCP, 6002/TCP, 6003/TCP, 6004/TCP, 6005/TCP, 6006/TCP, 6007/TCP, 6008/TCP, 6009/TCP, 7775/TCP, 7776/UDP
    Host Ports:  0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/UDP
    Command:
      sh
      -c
      ./heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0--3944535742942496843.tar.gz . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor --topology-name=acking --topology-id=acking3ae6b6a0-bbb8-45a1-8015-37fd50fec7c6 --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181 --state-manager-root=/heron --state-manager-config-file=./heron-conf/statemgr.yaml --tmanager-binary=./heron-core/bin/heron-tmanager --stmgr-binary=./heron-core/bin/heron-stmgr --metrics-manager-classpath=./heron-core/lib/metricsmgr/* --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)" --classpath=heron-api-examples.jar --heron-internals-config-file=./heron-conf/heron_internals.yaml --override-config-file=./heron-conf/override.yaml --component-ram-map=exclaim1:1073741824,word:1073741824 --component-jvm-opts="" --pkg-type=jar --topology-binary-file=heron-api-examples.jar --heron-java-home=$JAVA_HOME --heron-shell-binary=./heron-core/bin/heron-shell --cluster=kubernetes --role=saad --environment=default --instance-classpath=./heron-core/lib/instance/* --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/* --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*: --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824 --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/* --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003 --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007 --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009
    Limits:
      cpu:     2
      memory:  3
    Requests:
      cpu:     1
      memory:  2
    Environment:
      HOST:                (v1:status.podIP)
      POD_NAME:            (v1:metadata.name)
      var_one_manager:    variable one on manager
      var_three_manager:  variable three on manager
      var_two_manager:    variable two on manager
    Mounts:
      /shared_volume/manager from shared-volume-manager (rw)
      empty/dir/path from manager-empty-dir (ro,path="empty/dir/sub/path")
      host/path/path from manager-host-path (ro,path="host/path/sub/path")
      nfs/path from manager-nfs (ro,path="nfs/sub/path")
      path/to/mount/dynamic/volume from manager-dynamic-volume (rw,path="sub/path/to/mount/dynamic/volume")
      path/to/mount/shared/volume from manager-shared-volume (rw,path="sub/path/to/mount/shared/volume")
      path/to/mount/static/volume from manager-static-volume (rw,path="sub/path/to/mount/static/volume")
   manager-sidecar-container:
    Image:        alpine
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:
      /shared_volume/manager from shared-volume-manager (rw)
  Volumes:
   manager-empty-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  50Mi
   manager-host-path:
    Type:          HostPath (bare host directory volume)
    Path:          /dev/null
    HostPathType:  File
   manager-nfs:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    nfs-server.address
    Path:      /dev/null
    ReadOnly:  true
   manager-shared-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  requested-claim-by-user
    ReadOnly:   false
   shared-volume-manager:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
Volume Claims:
  Name:          manager-static-volume
  StorageClass:  
  Labels:        onDemand=true
                 topology=acking
  Annotations:   <none>
  Capacity:      512Gi
  Access Modes:  [ReadWriteOnce ReadOnlyMany]
  Name:          manager-dynamic-volume
  StorageClass:  storage-class-name-manager
  Labels:        onDemand=true
                 topology=acking
  Annotations:   <none>
  Capacity:      256Gi
  Access Modes:  [ReadWriteOnce ReadOnlyMany]
Events:
  Type    Reason            Age    From                    Message
  ----    ------            ----   ----                    -------
  Normal  SuccessfulCreate  4m46s  statefulset-controller  create Claim manager-static-volume-acking-manager-0 Pod acking-manager-0 in StatefulSet acking-manager success
  Normal  SuccessfulCreate  4m46s  statefulset-controller  create Claim manager-dynamic-volume-acking-manager-0 Pod acking-manager-0 in StatefulSet acking-manager success
  Normal  SuccessfulCreate  4m46s  statefulset-controller  create Pod acking-manager-0 in StatefulSet acking-manager successful

Persistent Volume Claims

Name:          executor-dynamic-volume-acking-executor-0
Namespace:     default
StorageClass:  storage-class-name-executor
Status:        Pending
Volume:        
Labels:        app=heron
               onDemand=true
               topology=acking
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Block
Used By:       acking-executor-0
Events:
  Type     Reason              Age                   From                         Message
  ----     ------              ----                  ----                         -------
  Warning  ProvisioningFailed  2m8s (x26 over 8m9s)  persistentvolume-controller  storageclass.storage.k8s.io "storage-class-name-executor" not found


Name:          executor-dynamic-volume-acking-executor-1
Namespace:     default
StorageClass:  storage-class-name-executor
Status:        Pending
Volume:        
Labels:        app=heron
               onDemand=true
               topology=acking
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Block
Used By:       acking-executor-1
Events:
  Type     Reason              Age                   From                         Message
  ----     ------              ----                  ----                         -------
  Warning  ProvisioningFailed  2m8s (x26 over 8m9s)  persistentvolume-controller  storageclass.storage.k8s.io "storage-class-name-executor" not found


Name:          executor-static-volume-acking-executor-0
Namespace:     default
StorageClass:  
Status:        Pending
Volume:        
Labels:        app=heron
               onDemand=true
               topology=acking
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Block
Used By:       acking-executor-0
Events:
  Type    Reason         Age                   From                         Message
  ----    ------         ----                  ----                         -------
  Normal  FailedBinding  2m8s (x26 over 8m9s)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set


Name:          executor-static-volume-acking-executor-1
Namespace:     default
StorageClass:  
Status:        Pending
Volume:        
Labels:        app=heron
               onDemand=true
               topology=acking
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Block
Used By:       acking-executor-1
Events:
  Type    Reason         Age                   From                         Message
  ----    ------         ----                  ----                         -------
  Normal  FailedBinding  2m8s (x26 over 8m9s)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set


Name:          manager-dynamic-volume-acking-manager-0
Namespace:     default
StorageClass:  storage-class-name-manager
Status:        Pending
Volume:        
Labels:        app=heron
               onDemand=true
               topology=acking
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Block
Used By:       acking-manager-0
Events:
  Type     Reason              Age                   From                         Message
  ----     ------              ----                  ----                         -------
  Warning  ProvisioningFailed  2m8s (x26 over 8m9s)  persistentvolume-controller  storageclass.storage.k8s.io "storage-class-name-manager" not found


Name:          manager-static-volume-acking-manager-0
Namespace:     default
StorageClass:  
Status:        Pending
Volume:        
Labels:        app=heron
               onDemand=true
               topology=acking
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Block
Used By:       acking-manager-0
Events:
  Type    Reason         Age                   From                         Message
  ----    ------         ----                  ----                         -------
  Normal  FailedBinding  2m8s (x26 over 8m9s)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set

surahman · 2022-05-06T17:05:05Z

I employed a hybrid-factory pattern here because you cannot overload functions strictly on return types in Java. I would like to get this PR merged before we set a signed tag for the subsequent release. I know some folks are really busy and may not have time to review this.

@thinker0 I appreciate you are using Mesos as the scheduler of choice but there isn't a modern programming language in existence you are not comfortable with. Could you please have a look at the changes and provide some general feedback when you have time? I could not find you in the review request list so I cannot add you to the list of requested reviewers.

@windhamwong You are a Python expert but I am not sure how comfortable you are with Java, could you please have a look if you are comfortable? If I recall correctly you use Kubernetes as your scheduler of choice? Would you be so kind as to give this PR a deployment test if you have some time?

nicknezis · 2022-05-06T18:48:37Z

Are we sure this won't break existing functionality? I kind of was leaning to cut the release before putting this in. If you're confident in the changes, I'm ok adding it. But I made a tag yesterday and was starting to generate the artifacts.

If you've done a Helm install and tested things, I'm ok moving the branch/tag to include this. Will also try to review later tonight to better understand the cleanup changes.

surahman · 2022-05-06T19:07:16Z

If you've done a Helm install and tested things, I'm ok moving the branch/tag to include this. Will also try to review later tonight to better understand the cleanup changes.

I have not completed a Helm install but I have completed deployment testing on Minikube. If anyone else is uncomfortable/unsure with the changes I am fine with not having this in the release. With that said, this does remove deprecated code and would provide a clean tagged point to reset to if we ever needed to.

Edit: I should add that there are no permission changes or additional functionality added which would require additional permissions on Kubernetes. I am not sure how else there would be an impact on the Helm charts.

nicknezis · 2022-05-07T06:56:58Z

Oh, I just meant a test in k8s, like Minikube. I did a test install and it looks good. Haven't tested any PVC mounts, but it looks good.

surahman · 2022-05-07T15:30:03Z

Much obliged.

The test suites in this package should be robust and comprehensive enough to catch any broken functionality. The deployment test traces above indicate good volume creation. This PR will not negatively impact stability.

surahman added 12 commits April 30, 2022 11:50

[K8s] Removed EBS keys from context.

9619c05

[K8s] removed factory type for EBS.

b0706bf

[K8s] removed tests for EBS.

25c7c25

[K8s] Removed NFS keys from context.

b3383f6

[K8s] removed factory type for NFS.

06b5f84

[K8s] removed tests for NFS.

4f7f72c

[K8s] Removed Host Path keys from context.

4c2bb11

[K8s] removed factory type for Host Path.

0f7dc30

[K8s] removed tests for Host Path.

d427408

[K8s] Removed unneeded imports and constants.

0ef4430

[K8s] removed config volume de-duplicator.

8d701c5

Volumes are no longer loaded from Configs. As such there is no longer a need to de-duplicate against volumes loaded from Pod Templates and the CLI.

Merge branch 'HEAD' into saadurrahman/3821-Remove-Deprecated-Volumes-K8s

4315e5d

surahman added the Kubernetes label May 1, 2022

surahman self-assigned this May 1, 2022

surahman added 16 commits May 1, 2022 14:05

[K8s] Added support for Empty Directory.

b7999d2

[K8s] removed unused functions.

2707cd9

[K8s] Removed old code to add Empty Directory.

6c2e0f5

Updated V1Controller to use VolumeFactory for Empty Directory generation.

[K8s] Added support for Host Path.

053bb5f

[K8s] Removed old code to add Host Path.

e521247

Updated V1Controller to use VolumeFactory for Host Path generation.

[K8s] Added support for Network File System.

c13f0ee

[K8s] Removed old code to add Network File System.

dbd4fce

Updated V1Controller to use VolumeFactory for Network File System generation.

[K8s] cleanup in VolumeFactory.

cb2ccca

[K8s] Interfaces in Volume Factory updated.

616eb44

Added interfaces: - Volume Mounts - Persistent Volume Claim

Merge branch 'saadurrahman/3821-Remove-Deprecated-Volumes-K8s-dev' in…

b73dbf1

…to saadurrahman/3821-Remove-Deprecated-Volumes-K8s

[K8s] created Volume Mount in Volume Factory.

7bc2f02

Generating all volume mounts from within Volume Factory for maintainability.

[K8s] switched to using Volume Factory for Volume Mount generation.

44d859e

Merge branch 'saadurrahman/3821-Remove-Deprecated-Volumes-K8s-dev' in…

2d25581

…to saadurrahman/3821-Remove-Deprecated-Volumes-K8s

[K8s] created PVC generation in Volume Factory.

35c023e

Merge branch 'saadurrahman/3821-Remove-Deprecated-Volumes-K8s-dev' in…

db8472c

…to saadurrahman/3821-Remove-Deprecated-Volumes-K8s

[K8s] using Volume Factory to generate PVC.

2c0512e

surahman added 4 commits May 3, 2022 10:34

[K8s] test hardening.

1aa3289

Test in <testCreatePersistentVolumeClaims> hardened for size.

[K8s] generating volume with PVC in Volume Factory.

b4d7cd3

[K8s] using Volume Factory to create volumes with PVC.

2a5c30e

Merge branch 'saadurrahman/3821-Remove-Deprecated-Volumes-K8s-dev' in…

574efd5

…to saadurrahman/3821-Remove-Deprecated-Volumes-K8s

surahman force-pushed the saadurrahman/3821-Remove-Deprecated-Volumes-K8s branch from a1b1fcb to 574efd5 Compare May 3, 2022 14:58

surahman requested review from nicknezis, nwangtw and joshfischer1108 May 3, 2022 15:07

surahman added 2 commits May 3, 2022 12:31

[K8s] indentation fix.

f900679

Merge branch 'saadurrahman/3821-Remove-Deprecated-Volumes-K8s-dev' in…

49f00ae

…to saadurrahman/3821-Remove-Deprecated-Volumes-K8s

surahman marked this pull request as ready for review May 3, 2022 20:03

surahman requested a review from windhamwong May 6, 2022 17:05

nicknezis approved these changes May 7, 2022

View reviewed changes

surahman merged commit dc2d9f6 into master May 7, 2022

nicknezis deleted the saadurrahman/3821-Remove-Deprecated-Volumes-K8s branch May 7, 2022 16:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[3821] Remove deprecated Host Path, NFS, and EBS support for Kubernetes #3825

[3821] Remove deprecated Host Path, NFS, and EBS support for Kubernetes #3825

surahman commented May 1, 2022 •

edited

Loading

surahman commented May 3, 2022 •

edited

Loading

Manager

Executor

surahman commented May 6, 2022 •

edited

Loading

nicknezis commented May 6, 2022

surahman commented May 6, 2022 •

edited

Loading

nicknezis commented May 7, 2022

surahman commented May 7, 2022 •

edited

Loading

[3821] Remove deprecated Host Path, NFS, and EBS support for Kubernetes #3825

[3821] Remove deprecated Host Path, NFS, and EBS support for Kubernetes #3825

Conversation

surahman commented May 1, 2022 • edited Loading

surahman commented May 3, 2022 • edited Loading

Manager

Executor

surahman commented May 6, 2022 • edited Loading

nicknezis commented May 6, 2022

surahman commented May 6, 2022 • edited Loading

nicknezis commented May 7, 2022

surahman commented May 7, 2022 • edited Loading

surahman commented May 1, 2022 •

edited

Loading

surahman commented May 3, 2022 •

edited

Loading

surahman commented May 6, 2022 •

edited

Loading

surahman commented May 6, 2022 •

edited

Loading

surahman commented May 7, 2022 •

edited

Loading