Skip to content

Commit 0d9c44e

Browse files
Sam Naserkaariger
Sam Naser
andauthored
feat!: Rewrite ASM module (#1140)
* Remove previous ASM module and add initial implementation of new * Move from kubernetes_manifest to bash CPR creation * Add meshconfig.googleapis.com service * Add namespace to CPR creation script wait * Change example to use google_container_cluster * Improve CPR status wait duration * Use retries rather than sleeping to wait for CPR CRD existence * Enable servicemesh feature in module * Revert changes to examples * Fix enable_mdp to just enable CNI * Lint * minor fixes * Bump timeout on status wait * Don't create MeshConfig if unset * Fix ASM sample * Update README autogen * Minor fixes * Remove meshConfig from module * fix end to end tests * lint fixes * Remove enable_mdp, add enable_vpc_sc and fleet_id * move VPC-SC from labels to annotations * update README * use default node pool size * use wip for exmaple cluster creation * add feature enablement back to module * lint * remove feature enablement from module * remove depends_on * fix unspecified channel bug * minor fixes * add more testing * update docs * change required module version * fix cclb * change registry source to run in CI * update CROSS_CLUSTER_SERVICE_DISCOVERY to multicluster_mode * update README * fix test * fix wording * remove from README * iterate on comments Co-authored-by: kaariger <[email protected]>
1 parent d5ceafb commit 0d9c44e

File tree

12 files changed

+211
-547
lines changed

12 files changed

+211
-547
lines changed

docs/upgrading_to_v20.0.md

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# Upgrading to v20.0
2+
3+
The v20.0 release of *kubernetes-engine* is a backwards incompatible
4+
release for the Anthos Service Mesh (ASM) module.
5+
6+
### ASM module rewrite
7+
8+
The [ASM submodule](https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/tree/master/modules/asm) has been rewritten to use the `ControlPlaneRevision` API to provision
9+
a managed control plane rather than using an installer script. Due to the drastic difference in implementation the module does not support an upgrade path
10+
from the previous version.

examples/simple_zonal_with_asm/hub.tf

+16-7
Original file line numberDiff line numberDiff line change
@@ -14,11 +14,20 @@
1414
* limitations under the License.
1515
*/
1616

17-
module "hub" {
18-
source = "../../modules/hub"
19-
project_id = var.project_id
20-
location = module.gke.location
21-
cluster_name = module.gke.name
22-
cluster_endpoint = module.gke.endpoint
23-
gke_hub_membership_name = "gke-asm-membership"
17+
resource "google_gke_hub_membership" "cluster_membership" {
18+
provider = google-beta
19+
project = var.project_id
20+
membership_id = "gke-asm-membership"
21+
endpoint {
22+
gke_cluster {
23+
resource_link = "//container.googleapis.com/${module.gke.cluster_id}"
24+
}
25+
}
26+
}
27+
28+
resource "google_gke_hub_feature" "mesh" {
29+
name = "servicemesh"
30+
project = var.project_id
31+
location = "global"
32+
provider = google-beta
2433
}

examples/simple_zonal_with_asm/main.tf

+8-14
Original file line numberDiff line numberDiff line change
@@ -44,29 +44,23 @@ module "gke" {
4444
ip_range_services = var.ip_range_services
4545
network_policy = false
4646
cluster_resource_labels = { "mesh_id" : "proj-${data.google_project.project.number}" }
47+
identity_namespace = "${var.project_id}.svc.id.goog"
4748
node_pools = [
4849
{
4950
name = "asm-node-pool"
5051
autoscaling = false
5152
auto_upgrade = true
52-
# ASM requires minimum 4 nodes and e2-standard-4
53-
node_count = 4
53+
node_count = 3
5454
machine_type = "e2-standard-4"
5555
},
5656
]
5757
}
5858

5959
module "asm" {
60-
source = "../../modules/asm"
61-
cluster_name = module.gke.name
62-
cluster_endpoint = module.gke.endpoint
63-
project_id = var.project_id
64-
location = module.gke.location
65-
enable_cluster_roles = true
66-
enable_cluster_labels = true
67-
enable_gcp_apis = true
68-
enable_gcp_components = true
69-
enable_namespace_creation = true
70-
options = ["envoy-access-log"]
71-
outdir = "./${module.gke.name}-outdir"
60+
source = "../../modules/asm"
61+
project_id = var.project_id
62+
cluster_name = module.gke.name
63+
cluster_location = module.gke.location
64+
multicluster_mode = "connected"
65+
enable_cni = true
7266
}

modules/asm/README.md

+21-59
Original file line numberDiff line numberDiff line change
@@ -2,87 +2,49 @@
22

33
This module installs [Anthos Service Mesh](https://cloud.google.com/service-mesh/docs) (ASM) in a Kubernetes Engine (GKE) cluster.
44

5-
Specifically, this module automates installing the ASM Istio Operator on your cluster ([installing ASM](https://cloud.google.com/service-mesh/docs/install)).
6-
75
## Usage
86

9-
There is a [full example](../../examples/simple_zonal_with_asm) provided. Detailed usage example is as follows:
7+
There are a few prerequisites to using this module that can be done either through Terraform or manually:
8+
9+
1. Enable the `mesh.cloud.googleapis.com` service
10+
1. Enable the `servicemesh` feature on the cluster hub
11+
1. Register target cluster to the servicemesh-enabled hub
12+
13+
There is a full example provided [here](../../examples/simple_zonal_with_asm). Detailed usage example is as follows:
1014

1115
```tf
1216
module "asm" {
13-
source = "terraform-google-modules/kubernetes-engine/google//modules/asm"
14-
15-
project_id = "my-project-id"
16-
cluster_name = "my-cluster-name"
17-
location = module.gke.location
18-
cluster_endpoint = module.gke.endpoint
19-
enable_all = false
20-
enable_cluster_roles = true
21-
enable_cluster_labels = false
22-
enable_gcp_apis = false
23-
enable_gcp_iam_roles = true
24-
enable_gcp_components = true
25-
enable_registration = false
26-
managed_control_plane = false
27-
options = ["envoy-access-log,egressgateways"]
28-
custom_overlays = ["./custom_ingress_gateway.yaml"]
29-
skip_validation = true
30-
outdir = "./${module.gke.name}-outdir-${var.asm_version}"
17+
source = "../../modules/asm"
18+
project_id = var.project_id
19+
cluster_name = module.gke.name
20+
cluster_location = module.gke.location
21+
enable_cni = true
3122
}
3223
```
3324

3425
To deploy this config:
3526

3627
1. Run `terraform apply`
3728

38-
## Requirements
39-
40-
- Anthos Service Mesh on GCP no longer requires an active Anthos license. You can use Anthos Service Mesh as a standalone product on GCP (on GKE) or as part of your Anthos subscription for hybrid and multi-cloud architectures.
41-
- GKE cluster must have minimum four nodes.
42-
- Minimum machine type is `e2-standard-4`.
43-
- GKE cluster must be enrolled in a release channel. ASM does not support static version.
44-
- ASM on a private GKE cluster requires adding a firewall rule to open port 15017 if you want to use [automatic sidecar injection](https://cloud.google.com/service-mesh/docs/proxy-injection).
45-
- One ASM mesh per Google Cloud project is supported.
46-
4729
<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
4830
## Inputs
4931

5032
| Name | Description | Type | Default | Required |
5133
|------|-------------|------|---------|:--------:|
52-
| asm\_git\_tag | ASM git tag to deploy. This module supports versions `1.8`, `1.9` and `1.10`. You can get the exact `asm_git_tag` by running the command `install_asm --version`. The ASM git tab should be of the form `1.9.3-asm.2+config5`. You can also see all ASM git tags by running `curl https://storage.googleapis.com/csm-artifacts/asm/STABLE_VERSIONS`. You must provide the full and exact git tag. This variable is optional. Leaving it empty (default) will download the latest `install_asm` script for the version provided by the `asm_version` variable. | `string` | `""` | no |
53-
| asm\_version | ASM version to deploy. This module supports versions `1.8`, `1.9` and `1.10`. Available versions are documented in https://github.com/GoogleCloudPlatform/anthos-service-mesh-packages | `string` | `"1.9"` | no |
54-
| ca | Sets CA option. Possible values are `meshca` or `citadel`. Additional documentation on Citadel is available at https://cloud.google.com/service-mesh/docs/scripted-install/gke-install#installation_with_citadel_as_the_ca. | `string` | `"meshca"` | no |
55-
| ca\_certs | Sets CA certificate file paths when `ca` is set to `citadel`. These values must be provided when using Citadel as CA. Additional documentation on Citadel is available at https://cloud.google.com/service-mesh/docs/scripted-install/gke-install#installation_with_citadel_as_the_ca. | `map(any)` | `{}` | no |
56-
| cluster\_endpoint | The GKE cluster endpoint. | `string` | n/a | yes |
34+
| channel | The channel to use for this ASM installation. | `string` | `""` | no |
35+
| cluster\_location | The cluster location for this ASM installation. | `string` | n/a | yes |
5736
| cluster\_name | The unique name to identify the cluster in ASM. | `string` | n/a | yes |
58-
| custom\_overlays | Comma separated list of custom\_overlay file paths. Works with in-cluster control plane only. Additional documentation available at https://cloud.google.com/service-mesh/docs/scripted-install/gke-install#installation_with_an_overlay_file | `list(any)` | `[]` | no |
59-
| enable\_all | Sets `--enable_all` option if true. | `bool` | `false` | no |
60-
| enable\_cluster\_labels | Sets `--enable_cluster_labels` option if true. | `bool` | `false` | no |
61-
| enable\_cluster\_roles | Sets `--enable_cluster_roles` option if true. | `bool` | `false` | no |
62-
| enable\_gcp\_apis | Sets `--enable_gcp_apis` option if true. | `bool` | `false` | no |
63-
| enable\_gcp\_components | Sets --enable\_gcp\_components option if true. Can be true or false. Available versions are documented in https://github.com/GoogleCloudPlatform/anthos-service-mesh-packages | `bool` | `false` | no |
64-
| enable\_gcp\_iam\_roles | Grants IAM roles required for ASM if true. If enable\_gcp\_iam\_roles, one of impersonate\_service\_account, service\_account, or iam\_member must be set. | `bool` | `false` | no |
65-
| enable\_namespace\_creation | Sets `--enable_namespace_creation` option if true. | `bool` | `false` | no |
66-
| enable\_registration | Sets `--enable_registration` option if true. | `bool` | `false` | no |
67-
| gcloud\_sdk\_version | The gcloud sdk version to use. Minimum required version is 293.0.0 | `string` | `"296.0.1"` | no |
68-
| iam\_member | The GCP member email address to grant IAM roles to. If impersonate\_service\_account or service\_account is set, roles are granted to that SA. | `string` | `""` | no |
69-
| impersonate\_service\_account | An optional service account to impersonate for gcloud commands. If this service account is not specified, the module will use Application Default Credentials. | `string` | `""` | no |
70-
| key\_file | The GCP Service Account credentials file path used to deploy ASM. | `string` | `""` | no |
71-
| location | The location (zone or region) this cluster has been created in. | `string` | n/a | yes |
72-
| managed\_control\_plane | ASM managed control plane boolean. Determines whether to install ASM managed control plane. Installing ASM managed control plane does not install gateways. Documentation on how to install gateways with ASM MCP can be found at https://cloud.google.com/service-mesh/docs/managed-control-plane#install_istio_gateways_optional. | `bool` | `false` | no |
73-
| mode | ASM mode for deployment. Supported modes are `install` and `upgrade`. | `string` | `"install"` | no |
74-
| options | Comma separated list of options. Works with in-cluster control plane only. Supported options are documented in https://cloud.google.com/service-mesh/docs/enable-optional-features. | `list(any)` | `[]` | no |
75-
| outdir | Sets `--outdir` option. | `string` | `"none"` | no |
37+
| enable\_cni | Determines whether to enable CNI for this ASM installation. Required to use Managed Data Plane (MDP). | `bool` | `false` | no |
38+
| enable\_vpc\_sc | Determines whether to enable VPC-SC for this ASM installation. For more information read https://cloud.google.com/service-mesh/docs/managed/vpc-sc | `bool` | `false` | no |
39+
| fleet\_id | The fleet to use for this ASM installation. | `string` | `""` | no |
40+
| multicluster\_mode | [Preview] Determines whether remote secrets should be autogenerated across fleet cluster. | `string` | `"manual"` | no |
7641
| project\_id | The project in which the resource belongs. | `string` | n/a | yes |
77-
| revision\_name | Sets `--revision-name` option. | `string` | `"none"` | no |
78-
| service\_account | The GCP Service Account email address used to deploy ASM. | `string` | `""` | no |
79-
| service\_account\_key\_file | Path to service account key file to auth as for running `gcloud container clusters get-credentials`. | `string` | `""` | no |
80-
| skip\_validation | Sets `_CI_NO_VALIDATE` variable. Determines whether the script should perform validation checks for prerequisites such as IAM roles, Google APIs etc. | `bool` | `false` | no |
8142

8243
## Outputs
8344

8445
| Name | Description |
8546
|------|-------------|
86-
| asm\_wait | An output to use when you want to depend on ASM finishing |
47+
| revision\_name | The name of the installed managed ASM revision. |
48+
| wait | An output to use when depending on the ASM installation finishing. |
8749

88-
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
50+
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

modules/asm/main.tf

+32-77
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/**
2-
* Copyright 2018 Google LLC
2+
* Copyright 2022 Google LLC
33
*
44
* Licensed under the Apache License, Version 2.0 (the "License");
55
* you may not use this file except in compliance with the License.
@@ -14,95 +14,50 @@
1414
* limitations under the License.
1515
*/
1616

17-
data "google_project" "asm_project" {
18-
project_id = var.project_id
19-
}
20-
2117
locals {
22-
options_string = length(var.options) > 0 ? join(",", var.options) : "none"
23-
custom_overlays_string = length(var.custom_overlays) > 0 ? join(",", var.custom_overlays) : "none"
24-
asm_git_tag_string = (var.asm_git_tag == "" ? "none" : var.asm_git_tag)
25-
service_account_string = (var.service_account == "" ? "none" : var.service_account)
26-
key_file_string = (var.key_file == "" ? "none" : var.key_file)
27-
ca_cert = lookup(var.ca_certs, "ca_cert", "none")
28-
ca_key = lookup(var.ca_certs, "ca_key", "none")
29-
root_cert = lookup(var.ca_certs, "root_cert", "none")
30-
cert_chain = lookup(var.ca_certs, "cert_chain", "none")
31-
revision_name_string = (var.revision_name == "" ? "none" : var.revision_name)
32-
asm_minor_version = tonumber(split(".", var.asm_version)[1])
33-
# https://github.com/GoogleCloudPlatform/anthos-service-mesh-packages/blob/1cf61b679cd369f42a0e735f8e201de1a6a6433b/scripts/asm-installer/install_asm#L1970
34-
iam_roles = [
35-
"roles/container.admin",
36-
"roles/meshconfig.admin",
37-
"roles/gkehub.admin",
38-
]
39-
# https://github.com/GoogleCloudPlatform/anthos-service-mesh-packages/blob/1cf61b679cd369f42a0e735f8e201de1a6a6433b/scripts/asm-installer/install_asm#L1958
40-
mcp_iam_roles = [
41-
"roles/serviceusage.serviceUsageConsumer",
42-
"roles/container.admin",
43-
"roles/monitoring.metricWriter",
44-
"roles/logging.logWriter",
45-
"roles/gkehub.viewer",
46-
"roles/gkehub.gatewayAdmin",
47-
]
48-
# if enable_gcp_iam_roles is set, grant IAM roles to first non null principal in the order below
49-
asm_iam_member = var.enable_gcp_iam_roles ? coalesce(var.impersonate_service_account, var.service_account, var.iam_member) : ""
50-
# compute any additonal resources that ASM provisioner should depend on
51-
additional_depends_on = concat(var.enable_gcp_apis ? [module.asm-services[0].project_id] : [], local.asm_iam_member != "" ? [for k, v in google_project_iam_member.asm_iam : v.etag] : [])
52-
# base command template for ASM installation
53-
kubectl_create_command_base = "${path.module}/scripts/install_asm.sh ${var.project_id} ${var.cluster_name} ${var.location} ${var.asm_version} ${var.mode} ${var.managed_control_plane} ${var.skip_validation} ${local.options_string} ${local.custom_overlays_string} ${var.enable_all} ${var.enable_cluster_roles} ${var.enable_cluster_labels} ${var.enable_gcp_components} ${var.enable_registration} ${var.outdir} ${var.ca} ${local.ca_cert} ${local.ca_key} ${local.root_cert} ${local.cert_chain} ${local.service_account_string} ${local.key_file_string} ${local.asm_git_tag_string} ${local.revision_name_string}"
18+
// GKE release channel is a list with max length 1 https://github.com/hashicorp/terraform-provider-google/blob/9d5f69f9f0f74f1a8245f1a52dd6cffb572bbce4/google/resource_container_cluster.go#L954
19+
gke_release_channel = data.google_container_cluster.asm.release_channel != null ? data.google_container_cluster.asm.release_channel[0].channel : ""
20+
gke_release_channel_filtered = lower(local.gke_release_channel) == "unspecified" ? "" : local.gke_release_channel
21+
// In order or precedence, use (1) user specified channel, (2) GKE release channel, and (3) regular channel
22+
channel = lower(coalesce(var.channel, local.gke_release_channel_filtered, "regular"))
23+
revision_name = "asm-managed${local.channel == "regular" ? "" : "-${local.channel}"}"
24+
// Fleet ID should default to project ID if unset
25+
fleet_id = coalesce(var.fleet_id, var.project_id)
5426
}
5527

56-
resource "google_project_iam_member" "asm_iam" {
57-
for_each = toset(local.asm_iam_member != "" ? (var.managed_control_plane ? local.mcp_iam_roles : local.iam_roles) : [])
28+
data "google_container_cluster" "asm" {
5829
project = var.project_id
59-
role = each.value
60-
member = "serviceAccount:${local.asm_iam_member}"
30+
name = var.cluster_name
31+
location = var.cluster_location
6132
}
6233

63-
module "asm-services" {
64-
source = "terraform-google-modules/project-factory/google//modules/project_services"
65-
version = "~> 11.3"
66-
67-
count = var.enable_gcp_apis ? 1 : 0
34+
resource "kubernetes_namespace" "system" {
35+
metadata {
36+
name = "istio-system"
37+
}
38+
}
6839

69-
project_id = var.project_id
70-
disable_services_on_destroy = false
71-
disable_dependent_services = false
40+
resource "kubernetes_config_map" "asm_options" {
41+
metadata {
42+
name = "asm-options"
43+
namespace = kubernetes_namespace.system.metadata[0].name
44+
}
7245

73-
# https://github.com/GoogleCloudPlatform/anthos-service-mesh-packages/blob/1cf61b679cd369f42a0e735f8e201de1a6a6433b/scripts/asm-installer/install_asm#L2005
74-
activate_apis = [
75-
"container.googleapis.com",
76-
"monitoring.googleapis.com",
77-
"logging.googleapis.com",
78-
"cloudtrace.googleapis.com",
79-
"meshtelemetry.googleapis.com",
80-
"meshconfig.googleapis.com",
81-
"meshca.googleapis.com",
82-
"iamcredentials.googleapis.com",
83-
"gkeconnect.googleapis.com",
84-
"gkehub.googleapis.com",
85-
"cloudresourcemanager.googleapis.com",
86-
"stackdriver.googleapis.com",
87-
]
46+
data = {
47+
multicluster_mode = var.multicluster_mode
48+
}
8849
}
8950

90-
module "asm_install" {
51+
module "cpr" {
9152
source = "terraform-google-modules/gcloud/google//modules/kubectl-wrapper"
9253
version = "~> 3.1"
9354

94-
module_depends_on = concat([var.cluster_endpoint], local.additional_depends_on)
55+
project_id = var.project_id
56+
cluster_name = var.cluster_name
57+
cluster_location = var.cluster_location
9558

96-
gcloud_sdk_version = var.gcloud_sdk_version
97-
upgrade = true
98-
additional_components = ["kubectl", "kpt", "beta"]
99-
cluster_name = var.cluster_name
100-
cluster_location = var.location
101-
project_id = var.project_id
102-
service_account_key_file = var.service_account_key_file
103-
impersonate_service_account = var.impersonate_service_account
59+
kubectl_create_command = "${path.module}/scripts/create_cpr.sh ${local.revision_name} ${local.channel} ${var.enable_cni} ${var.enable_vpc_sc}"
60+
kubectl_destroy_command = "${path.module}/scripts/destroy_cpr.sh ${local.revision_name}"
10461

105-
# enable_namespace_creation flag is only available starting 1.10
106-
kubectl_create_command = (local.asm_minor_version > 9 ? "${local.kubectl_create_command_base} ${var.enable_namespace_creation}" : local.kubectl_create_command_base)
107-
kubectl_destroy_command = "${path.module}/scripts/destroy_asm.sh"
62+
module_depends_on = [kubernetes_config_map.asm_options]
10863
}

modules/asm/outputs.tf

+9-4
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/**
2-
* Copyright 2018 Google LLC
2+
* Copyright 2022 Google LLC
33
*
44
* Licensed under the Apache License, Version 2.0 (the "License");
55
* you may not use this file except in compliance with the License.
@@ -14,7 +14,12 @@
1414
* limitations under the License.
1515
*/
1616

17-
output "asm_wait" {
18-
description = "An output to use when you want to depend on ASM finishing"
19-
value = module.asm_install.wait
17+
output "revision_name" {
18+
value = local.revision_name
19+
description = "The name of the installed managed ASM revision."
20+
}
21+
22+
output "wait" {
23+
value = module.cpr.wait
24+
description = "An output to use when depending on the ASM installation finishing."
2025
}

modules/asm/scripts/.gitignore

-1
This file was deleted.

0 commit comments

Comments
 (0)