Migrate away from n1-machine-{1-4} for k8s e2e testing

[kannon92]

N1 machines are slower and the suggestion is to start migrating the machines to use e2. These are faster and more available so this should help mitigate stock outage errors.

[kannon92]

cc @BenTheElder @ameukam

/sig node

[SergeyKanzhelev]

/sig testing

[SergeyKanzhelev]

@BenTheElder who is the best person to decide on this? Ultimately for the most tests we do not care of a VM shape

[BenTheElder]

/sig k8s-infra
@upodroid was looking into these trade-offs recently.

IMHO, we should move to newer machine types over time unless it's way more expensive.

We can also move to more expensive when we are getting a bit performance win ... e.g. the CI cluster is moving to C4 and C4D because they will let things like builds go a fair bit faster (20-25% for some workloads IIRC?) but I wouldn't move everything yet because there are some availability issues and we can keep some warm minimum scaled capacity there but e2e VMs cannot.

We also should probably be careful with jobs like scale tests where giving the test different machines might impact detecting performance regressions.

For workloads that aren't very sensitive, e2 seems reasonable:
https://cloud.google.com/compute/docs/machine-resource#recommendations_for_machine_types

[upodroid]

We should fix the node gce launcher to do the following:

• pass a region instead of a zone and also a list of machine types that you want to use
• us-central1 should be preferred CI region as it's closest to the CI cluster and also has the most types of VMs available. Its also one of the largest regions in GCP
• it should attempt to launch the instance in a random zone of the region with the first machine type on the list, if it fails attempt to launch the VM in a different zone with the same machine type and then retry the whole step again with a different machine type.
• after 4 attempts, we should consider the job failed and return the current error

[SergeyKanzhelev]

Having a priority list of VM shapes, zones, or even regions sounds good. I don't care if tests will run in asia or europe.

One thing is we need to have a clear and easy way to check in logs what was finally picked. Even better if this will be a metadata for the triage tool. So flakes can be grouped by this machine type or zone.