Updating google cloud nio to 0.107.0 #6042

lbergelson · 2019-07-16T18:07:31Z

Updating google-cloud-nio 0.81.0-alpha:shaded -> 0.100.0-alpha:shaded
Fixes Strange behavior in ParallelCopyGCSDirectoryIntoHDFSSpark with NIO ver > 0.66 #5935

lbergelson · 2019-07-16T18:07:47Z

@SHuang-Broad Lets see if it passes tests...

lbergelson · 2019-07-16T20:47:11Z

Hmn. This is failing with 403 unauthorized errors. Seems like something about authentication changed.

SHuang-Broad · 2019-07-16T20:51:10Z

you mean tests?
I'm running the "failed" tool on an actual dataproc cluster right now. Should have a result in 10 minutes.

lbergelson · 2019-07-16T20:52:40Z

Yeah, the cloud tests are all failing.

SHuang-Broad · 2019-07-16T21:02:03Z

@lbergelson
Tested working.

#!/bin/bash

set -eu

echo "=============================="
export CRAM_BUCKET="gs://broad-dsde-methods-shuang/tmp/test/"
gsutil ls  -l  -h "${CRAM_BUCKET}"
echo "=============================="

# on Louis branch containing NIO 100
echo "=============================="
export GATK_DIR="/Users/shuang/GATK/gatk"
export CLUSTER_NAME="shuang-nio-100"
cd "${GATK_DIR}" && git pull && git checkout lb_update_nio && \
./gradlew clean installAll && \
cd scripts/sv/ && \
bash create_cluster.sh \
  "${GATK_DIR}" \
  broad-dsde-methods \
  "${CLUSTER_NAME}" \
  3h \
  120m \
  gs://broad-dsde-methods-sv/reference/GRCh38/ \
  "${CRAM_BUCKET}" \
  gs://broad-dsde-methods-shuang/init/default_init.sh
echo "=============================="

# on master (expected to fail)
echo "=============================="
export GATK_DIR="/Users/shuang/GATK/forks/GatkFork"
export CLUSTER_NAME="shuang-nio-81"
cd "${GATK_DIR}" && git checkout master && git fetch upstream && git rebase upstream/master && \
./gradlew clean installAll && \
cd scripts/sv/ && \
bash create_cluster.sh \
  "${GATK_DIR}" \
  broad-dsde-methods \
  "${CLUSTER_NAME}" \
  3h \
  120m \
  gs://broad-dsde-methods-sv/reference/GRCh38/ \
  "${CRAM_BUCKET}" \
  gs://broad-dsde-methods-shuang/init/default_init.sh
echo "=============================="

SHuang-Broad

👍
tested running fine on actual dataproc clusters.

lbergelson · 2019-07-16T21:05:50Z

We're seeing a lot of failures of the form:

  com.google.cloud.storage.StorageException: 806222273987-uilktks3j6i7962rp0v7nusveer58497@developer.gserviceaccount.com does not have serviceusage.services.use access to project 685190392835.
        Caused by:
        shaded.cloud_nio.com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
        {
          "code" : 403,
          "errors" : [ {
            "domain" : "global",
            "message" : "806222273987-uilktks3j6i7962rp0v7nusveer58497@developer.gserviceaccount.com does not have serviceusage.services.use access to project 685190392835.",
            "reason" : "forbidden"
          } ],
          "message" : "806222273987-uilktks3j6i7962rp0v7nusveer58497@developer.gserviceaccount.com does not have serviceusage.services.use access to project 685190392835."
        }

It looks like it now requires some new permission for the service accounts but our existing service account doesn't have that permission.

lbergelson · 2019-07-16T21:06:26Z

@SHuang-Broad Thanks, that's good to know that it actually fixes the problem you're having... I think we need to investigate this error a bit more though before we can merge.

SHuang-Broad · 2019-07-16T21:11:16Z

Ha? That's strange.
Is the project id 685190392835 dsde-methods?

lbergelson · 2019-07-16T21:25:18Z

No... it's not. It doesn't seem to correspond to any broad project I can see.

SHuang-Broad · 2019-07-17T15:34:22Z

@lbergelson I did an experiment in #6046 with NIO 94, same errors.

SHuang-Broad · 2019-07-19T14:07:36Z

@lbergelson

I've run a few more tests, and found the following table

NIO_VER	403_DENIED_PROJECT_ID
99	685190392835
98	685190392835
97	685190392835
96	539774316296
95	685190392835
94	685190392835
93	539774316296
92	685190392835
91	539774316296
90	539774316296
89	PASS
88	PASS
87	PASS
86	PASS
85	PASS

Googling both of the two mysterious project IDs, I landed onto tests by this Fiji project (here and here).
I parsed the test log and found the relevant part (for ID 685190392835)

"gce":{

    "instance":{
        "attributes":{
            "startup-script":"#!/usr/bin/env bash\necho poweroff | at now + 130 minutes\ncat > ~travis/.ssh/authorized_keys <<EOF\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdzVnIEg2ribEvhEvFjR9IFPAkIVtQwZhlgUAHu1BgjBugFRiqg3eaPMOeOuIZBvzwoyotHIVp3XvAfivGyCW4Ke7+2cqlcX1L8kcmoWLm2fdLGlLr/lZnAjQtexMC76uLtR8udqWA0e2sqrSJs4H/blOQmHWPrl/VSG7daoVptzqXihRmXN+/Huo7mTxAjTUEjk4IOBn7sv7G5qLrEPv78AJIZhWHdhUTGLvx+YpzQvX8pE53TMi9W4ovkZTCwhSO3WYyBOY7H1xjeYb9XWTeP563Du1b0JMpQgtFLQUVXio9NzXZE55ovvGDRSLds+VfPsv4G/Whhq76dEZ+wZO3\n\nEOF\n"
        },
        "cpuPlatform":"Intel Haswell",
        "description":"Travis CI python test VM",
        "disks":[{"deviceName":"persistent-disk-0","index":0,"mode":"READ_WRITE","type":"PERSISTENT"}],
        "hostname":"testing-gce-ec8614d2-40a2-4138-801e-d42d811590a2.c.travis-ci-prod-2.internal",
        "id":8221730359445041428,
        "image":"",
        "licenses":[{"id":"1000010"}],
        "machineType":"projects/685190392835/machineTypes/n1-standard-2",
        "maintenanceEvent":"NONE",
        "networkInterfaces":[{"accessConfigs":[{"externalIp":"104.198.203.242","type":"ONE_TO_ONE_NAT"}],"forwardedIps":[],"ip":"10.128.0.163","network":"projects/685190392835/networks/default"}],
        "scheduling":{"automaticRestart":"TRUE","onHostMaintenance":"MIGRATE","preemptible":"FALSE"},
        "serviceAccounts":{
            "[email protected]":{
                "aliases":["default"],
                "email":"[email protected]",
                "scopes":["https://www.googleapis.com/auth/userinfo.email",
                          "https://www.googleapis.com/auth/devstorage.full_control",
                          "https://www.googleapis.com/auth/compute"]
            },
            "default":{
                "aliases":["default"],
                "email":"[email protected]",
                "scopes":["https://www.googleapis.com/auth/userinfo.email",
                          "https://www.googleapis.com/auth/devstorage.full_control",
                          "https://www.googleapis.com/auth/compute"]}
            },
        "tags":["testing"],
        "virtualClock":{"driftToken":"11704388862566216373"},
        "zone":"projects/685190392835/zones/us-central1-b"
    },

    "project":{
        "attributes":{
            "sshKeys":"henrik:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQChY0pdGXohYN7KRnQa3VIcDoVBrxZVHkhOFc1SROV2T+gTOunYbOW5C4V1P2MGG6FcKeoQTJzXgPbZurM5l1AfEbKeCde778QyyxbcjpYvKyY5b4qVO79nOKAg1qHIqUl+2txv7X6tPv4Q99T7UBechuc5awnkJZKqP1s1qJ9BYYYAPukZPbhAkjkvPSaJfIi+py2p6L9mXFtrAhYNH1flE9GErAsf2Hq8zQvx4hmTseumv4Fb9rVogcBJOqhmDQmYwTg2rEbdLAjbqY7Sf4kjdOfF7uhwasZgVMjF1z5utnvHd2wC/cjkuDZB4UhetLTeOWDtvgZxF/uVJSTU2AGD google-ssh {\"userName\":\"[email protected]\",\"expireOn\":\"2016-03-04T00:52:57+0000\"}\nhenrik:ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ6IOlU4vY6QLWKOX52Opcdx/2zNgJyMq7ntIf8qD+CbMMfUy5C6WJnjn4E2lvYqYaVIotY196cVazh0Jj8E/co= google-ssh {\"userName\":\"[email protected]\",\"expireOn\":\"2016-03-04T00:52:42+0000\"}\n"
        },
        "numericProjectId":685190392835,
        "projectId":"travis-ci-prod-2"
    }
},

In the meantime, I've also found this site for ID 539774316296.

Not sure if these info are useful for you guys (who know much more about this than I do) to debug the issue.

codecov · 2019-07-25T19:13:53Z

Codecov Report

❗ No coverage uploaded for pull request base (master@8d88f6e). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             master     #6042   +/-   ##
==========================================
  Coverage          ?   87.013%           
  Complexity        ?     32636           
==========================================
  Files             ?      2011           
  Lines             ?    150967           
  Branches          ?     16134           
==========================================
  Hits              ?    131361           
  Misses            ?     14021           
  Partials          ?      5585

lbergelson · 2019-07-25T20:09:03Z

I finally opened an issue with google cloud to look into this... https://github.com/googleapis/google-cloud-java/issues/5884

droazen · 2019-08-29T18:39:36Z

build.gradle

@@ -69,7 +69,7 @@ final testNGVersion = '6.11'
 // Using the shaded version to avoid conflicts between its protobuf dependency
 // and that of Hadoop/Spark (either the one we reference explicitly, or the one
 // provided by dataproc).
-final googleCloudNioDependency = 'com.google.cloud:google-cloud-nio:0.81.0-alpha:shaded'
+final googleCloudNioDependency = 'com.google.cloud:google-cloud-nio:0.106.0-alpha:shaded'


0.107.0 is now out, so might as well update to that. 0.105.0 included the shading fix we needed to add a BigQuery dependency in master.

Oh, that happened in the last day I guess... will update. We're going to need a picard update before this will work though.

It seems like we have a nasty bug where we are using the shaded version but also including the non-shaded versions transitive dependencies. Resolving that might help with bigquery too.

droazen · 2019-09-03T16:12:22Z

@lbergelson As discussed, it's not essential that we update to 107. I'd be happy with either 105 or 106, since the shading fix was released in 105.

droazen · 2019-09-11T18:41:52Z

build.gradle

@@ -59,7 +59,7 @@ repositories {

 final requiredJavaVersion = "8"
 final htsjdkVersion = System.getProperty('htsjdk.version','2.20.3')
-final picardVersion = System.getProperty('picard.version','2.20.5')
+final picardVersion = System.getProperty('picard.version','2.20.6-5-g1b4178f-SNAPSHOT')


Update to actual released version of Picard

* Updating google-cloud-nio 0.81.0-alpha:shaded -> 0.100.0-alpha:shaded * Fixes #5935

We were importing the transitive dependencies of the shaded version, which I don't think we should be doing.

lbergelson requested a review from SHuang-Broad July 16, 2019 18:07

lbergelson assigned SHuang-Broad Jul 16, 2019

SHuang-Broad approved these changes Jul 16, 2019

View reviewed changes

SHuang-Broad mentioned this pull request Jul 17, 2019

NO MERGE #6046

Closed

lbergelson force-pushed the lb_update_nio branch from 500f9db to 9409933 Compare August 28, 2019 22:15

droazen self-requested a review August 29, 2019 18:38

droazen self-assigned this Aug 29, 2019

droazen reviewed Aug 29, 2019

View reviewed changes

droazen changed the title ~~Updating google cloud nio to 0.100.0~~ Updating google cloud nio to 0.106.0 Aug 29, 2019

droazen changed the title ~~Updating google cloud nio to 0.106.0~~ Updating google cloud nio to 0.107.0 Aug 30, 2019

droazen assigned lbergelson and unassigned droazen and SHuang-Broad Aug 30, 2019

tomwhite mentioned this pull request Sep 3, 2019

Upgrade GCS Connector to 1.9.17 #6135

Closed

lbergelson force-pushed the lb_update_nio branch from cf06772 to c7654f1 Compare September 3, 2019 15:53

droazen reviewed Sep 11, 2019

View reviewed changes

lbergelson added 9 commits September 11, 2019 14:47

Updating google cloud nio to 0.100.0

dcd5664

* Updating google-cloud-nio 0.81.0-alpha:shaded -> 0.100.0-alpha:shaded * Fixes #5935

try .90

2d4dd2c

add workaround and return to vs 100

20e7d32

update to 106 and adapt to shaded class change

3baaea0

fixupe, forget $

b400217

change shading and go to 107

00bf644

We were importing the transitive dependencies of the shaded version, which I don't think we should be doing.

bump picard version to snapshot

bbf0606

update to new picard snapshot

091aa35

update to fix ? issue

03f8070

lbergelson force-pushed the lb_update_nio branch from 76cc1e3 to 03f8070 Compare September 11, 2019 18:47

lbergelson merged commit 78bf002 into master Sep 11, 2019

lbergelson deleted the lb_update_nio branch September 11, 2019 20:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updating google cloud nio to 0.107.0 #6042

Updating google cloud nio to 0.107.0 #6042

lbergelson commented Jul 16, 2019

lbergelson commented Jul 16, 2019

lbergelson commented Jul 16, 2019

SHuang-Broad commented Jul 16, 2019

lbergelson commented Jul 16, 2019

SHuang-Broad commented Jul 16, 2019

SHuang-Broad left a comment

lbergelson commented Jul 16, 2019

lbergelson commented Jul 16, 2019

SHuang-Broad commented Jul 16, 2019

lbergelson commented Jul 16, 2019

SHuang-Broad commented Jul 17, 2019

SHuang-Broad commented Jul 19, 2019

codecov bot commented Jul 25, 2019 •

edited

Loading

lbergelson commented Jul 25, 2019

droazen Aug 29, 2019

lbergelson Aug 29, 2019

lbergelson Aug 29, 2019

droazen commented Sep 3, 2019

droazen Sep 11, 2019

Updating google cloud nio to 0.107.0 #6042

Updating google cloud nio to 0.107.0 #6042

Conversation

lbergelson commented Jul 16, 2019

lbergelson commented Jul 16, 2019

lbergelson commented Jul 16, 2019

SHuang-Broad commented Jul 16, 2019

lbergelson commented Jul 16, 2019

SHuang-Broad commented Jul 16, 2019

SHuang-Broad left a comment

Choose a reason for hiding this comment

lbergelson commented Jul 16, 2019

lbergelson commented Jul 16, 2019

SHuang-Broad commented Jul 16, 2019

lbergelson commented Jul 16, 2019

SHuang-Broad commented Jul 17, 2019

SHuang-Broad commented Jul 19, 2019

codecov bot commented Jul 25, 2019 • edited Loading

Codecov Report

lbergelson commented Jul 25, 2019

droazen Aug 29, 2019

Choose a reason for hiding this comment

lbergelson Aug 29, 2019

Choose a reason for hiding this comment

lbergelson Aug 29, 2019

Choose a reason for hiding this comment

droazen commented Sep 3, 2019

droazen Sep 11, 2019

Choose a reason for hiding this comment

codecov bot commented Jul 25, 2019 •

edited

Loading