EKS ServiseAccount AWS_WEB_IDENTITY_TOKEN_FILE unix path notation on Windows nodes

[ArturChe]

Having an EKS cluster with windows nodes and configured Service Account with the proper IAM role there is an error trying to operate with AWS API:

Other exception recorded! Cause: System.ArgumentException: The webIdentityTokenFile must be an absolute path. Parameter name: webIdentityTokenFile at Amazon.Runtime.AssumeRoleWithWebIdentityCredentials..ctor(String webIdentityTokenFile, String roleArn, String roleSessionName, AssumeRoleWithWebIdentityCredentialsOptions options) at Amazon.Runtime.AssumeRoleWithWebIdentityCredentials.FromEnvironmentVariables() at eChalk.Common.ConfigurationManagement.AwsProtectedConfigurationClient.FallbackCredentialsFactory.<>c.<Reset>b__10_1()

The problem is that on pod the env variable looks like:

AWS_WEB_IDENTITY_TOKEN_FILE    /var/run/secrets/eks.amazonaws.com/serviceaccount/token

But if you replace the path with '\' and add the 'C:\' on app startup then it works as expected.

Moreover, without changing the path the AWSCLI v2 does works inside this pod and reacts on revoking policies from the role.

Your Environment

• AWSSDK.S3 3.3.110.68
• AWSSDK.SQS 3.3.102.109
• AWSSDK.KeyManagementService 3.3.105.46
• Operating System and version: Windows_Server-2019-English-Full-EKS_Optimized-1.16-2020.05.13 (ami-0062672e50d3637a3)
• Docker image: mcr.microsoft.com/dotnet/framework/runtime:4.7.2
• Targeted .NET platform: 4.7.2

[wahidshalaly]

Hi. I'm facing the same issue now but not on Windows. I'm working on a solution that uses Mono and Target Framework is .NET 4.7.2 (same as in the issue). When I try to deploy it to an EKS cluster that has a service account I get the same issue. This issue is coming from the following code in AWSSDK.Core.AWSSDKUtils

        public static bool IsAbsolutePath(string path)
        {
            return IsWindows() ? !IsPartiallyQualifiedForWindows(path) : Path.IsPathRooted(path);
        }

        private static bool IsWindows()
        {
#if NETSTANDARD
            return RuntimeInformation.IsOSPlatform(OSPlatform.Windows);
#endif
            return true;
        }

Since that we're targeting .NET 4.7.2, then the code will return the hard-coded true for IsWindows() and since that the path for tokens is a unix path /var/run/secrets/eks.amazonaws.com/serviceaccount/tokenso it'll fail the IsAbsolutePath condition and won't be able to load credentials. Reproducing is tricky a bit to write here, you'll need a .NET 4.7.2 application & EKS cluster & IAM role for a service account.

Another hint: RuntimeInformation.IsOSPlatform(OSPlatform.Windows) is available since .NET Framework 4.7.1.

I hope that can be helpful to troubleshoot and solve this issue.

[ashishdhingra]

Hi @ArturChe,

Good afternoon.

I had recently worked on setting up Windows node on EKS cluster using the following working steps. However, the environment variable AWS_WEB_IDENTITY_TOKEN_FILE was correctly being set as C:\var\run\secrets\eks.amazonaws.com\serviceaccount\token. Not sure if this was something that was fixed by EKS team.

• Create cluster
  eksctl create cluster windows-powershell-test

• Associate IAM OIDC provider

eksctl utils associate-iam-oidc-provider \
               --name windows-powershell-test \
               --approve

• Take note of namespace of created Amazon Linux node group. It is kube-system. Use this in subsequent commands and deployment templates.

• Create IAM Service Account

eksctl create iamserviceaccount \
                --name my-serviceaccount \
                --namespace kube-system \
                --cluster windows-powershell-test \
                --attach-policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess \
                --approve

Good reference (for Windows): https://jtekds.com/amazon-eks-windows-container-support-now-generally-available/, https://betterprogramming.pub/build-a-windows-container-service-on-a-kubernetes-cluster-cac5087a4499

• For Windows support, install VPC controllers
  eksctl utils install-vpc-controllers --name windows-powershell-test --approve

• Create Windows node group
  eksctl create nodegroup --cluster=windows-powershell-test --node-ami-family=WindowsServer2019FullContainer

• Apply the deployment manifest using kubectl apply -f ./manifest-test.yml (also take note of namespace which is set to kube-system as noted in one of the previous steps):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: windows-server-iis
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: windows-server-iis
      tier: backend
      track: stable
  replicas: 1
  template:
    metadata:
      labels:
        app: windows-server-iis
        tier: backend
        track: stable
    spec:
      serviceAccountName: my-serviceaccount
      containers:
      - name: windows-server-iis
        image: mcr.microsoft.com/windows/servercore:1809
        ports:
        - name: http
          containerPort: 80
        imagePullPolicy: IfNotPresent
        command:
        - powershell.exe
        - -command
        - "Add-WindowsFeature Web-Server; Invoke-WebRequest -UseBasicParsing -Uri 'https://dotnetbinaries.blob.core.windows.net/servicemonitor/2.0.1.6/ServiceMonitor.exe' -OutFile 'C:\\ServiceMonitor.exe'; echo '<html><body><br/><br/><marquee><H1>Hello EKS!!!<H1><marquee></body><html>' > C:\\inetpub\\wwwroot\\default.html; C:\\ServiceMonitor.exe 'w3svc'; "
      nodeSelector:
        kubernetes.io/os: windows
---
apiVersion: v1
kind: Service
metadata:
  name: windows-server-iis-service
  namespace: kube-system
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: windows-server-iis
    tier: backend
    track: stable
  sessionAffinity: None
  type: LoadBalancer

NOTE: If we used the correct namespace, it should have Annotation vpc.amazonaws.com/PrivateIPv4Address on the Windows nodes.

• Watch for Windows Pod creation in the namespace:
  kubectl get pods --namespace kube-system -o wide --watch

• Open interactive PowerShell session on Windows node:
  kubectl exec windows-server-iis-7c5dfddfbb-jfgzw --namespace kube-system -it powershell.exe

• In interactive Powershell session, execute the below command Get-ChildItem env:
  Result:

AWS_DEFAULT_REGION             us-east-2                                                                                                               
AWS_REGION                     us-east-2                                                                                                               
AWS_ROLE_ARN                   arn:aws:iam::139480602983:role/eksctl-windows-powershell-test-addon-iamserv-Role1-10QX42EY0PHZJ                         
AWS_WEB_IDENTITY_TOKEN_FILE    C:\var\run\secrets\eks.amazonaws.com\serviceaccount\token

Could you please revalidate in the new setup and confirm?

Thanks,
Ashish

[ArturChe]

@ashishdhingra The problem has been resolved with a cluster upgrade.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.