Skip to content

feat: LogPipeline HTTP output mTLS support #439

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Oct 5, 2023
Merged

feat: LogPipeline HTTP output mTLS support #439

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
1968acc
feat: LogPipeline HTTP Output mTLS support
dennis-ge Sep 28, 2023
56f7c05
fix e2e test
dennis-ge Sep 29, 2023
84a8a5a
add unit tests
dennis-ge Sep 29, 2023
b1dff81
further unit test improvements
dennis-ge Sep 29, 2023
5c84ffb
add docs and .golangci.yaml
dennis-ge Sep 29, 2023
aa20b0d
fix resources test
dennis-ge Sep 29, 2023
8aa9b27
apply suggestions
dennis-ge Oct 5, 2023
d9e8ce9
properly handle env secret and tls config secret
dennis-ge Oct 5, 2023
43cbe27
fix assignment
dennis-ge Oct 5, 2023
41edb19
make certs pointers
dennis-ge Oct 5, 2023
e82dc11
update docs
dennis-ge Oct 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions apis/telemetry/v1alpha1/logpipeline_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,12 @@ type TLSConfig struct {
Disabled bool `json:"disabled,omitempty"`
// If `true`, the validation of certificates is skipped. Default is `false`.
SkipCertificateValidation bool `json:"skipCertificateValidation,omitempty"`
// Defines an optional CA certificate for server certificate verification when using TLS. The certificate must be provided in PEM format.
CA ValueType `json:"ca,omitempty"`
// Defines a client certificate to use when using TLS. The certificate must be provided in PEM format.
Cert ValueType `json:"cert,omitempty"`
// Defines the client key to use when using TLS. The key must be provided in PEM format.
Key ValueType `json:"key,omitempty"`
}

// Output describes a Fluent Bit output configuration section.
Expand Down
6 changes: 3 additions & 3 deletions apis/telemetry/v1alpha1/shared_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,11 +57,11 @@ type OtlpTLS struct {
Insecure bool `json:"insecure,omitempty"`
// Defines whether to skip server certificate verification when using TLS.
InsecureSkipVerify bool `json:"insecureSkipVerify,omitempty"`
// Defines an optional CA certificate for server certificate verification when using TLS. The certificate needs to be provided in PEM format.
// Defines an optional CA certificate for server certificate verification when using TLS. The certificate must be provided in PEM format.
CA ValueType `json:"ca,omitempty"`
// Defines a client certificate to use when using TLS. The certificate needs to be provided in PEM format.
// Defines a client certificate to use when using TLS. The certificate must be provided in PEM format.
Cert ValueType `json:"cert,omitempty"`
// Defines the client key to use when using TLS. The key needs to be provided in PEM format.
// Defines the client key to use when using TLS. The key must be provided in PEM format.
Key ValueType `json:"key,omitempty"`
}

Expand Down
94 changes: 94 additions & 0 deletions config/crd/bases/telemetry.kyma-project.io_logpipelines.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -264,10 +264,104 @@ spec:
tls:
description: Configures TLS for the HTTP target server.
properties:
ca:
description: Defines an optional CA certificate for server
certificate verification when using TLS. The certificate
must be provided in PEM format.
properties:
value:
description: The value as plain text.
type: string
valueFrom:
description: The value as a reference to a resource.
properties:
secretKeyRef:
description: Refers to the value of a specific
key in a Secret. You must provide `name` and
`namespace` of the Secret, as well as the name
of the `key`.
properties:
key:
description: The name of the attribute of
the Secret holding the referenced value.
type: string
name:
description: The name of the Secret containing
the referenced value
type: string
namespace:
description: The name of the Namespace containing
the Secret with the referenced value.
type: string
type: object
type: object
type: object
cert:
description: Defines a client certificate to use when
using TLS. The certificate must be provided in PEM format.
properties:
value:
description: The value as plain text.
type: string
valueFrom:
description: The value as a reference to a resource.
properties:
secretKeyRef:
description: Refers to the value of a specific
key in a Secret. You must provide `name` and
`namespace` of the Secret, as well as the name
of the `key`.
properties:
key:
description: The name of the attribute of
the Secret holding the referenced value.
type: string
name:
description: The name of the Secret containing
the referenced value
type: string
namespace:
description: The name of the Namespace containing
the Secret with the referenced value.
type: string
type: object
type: object
type: object
disabled:
description: Indicates if TLS is disabled or enabled.
Default is `false`.
type: boolean
key:
description: Defines the client key to use when using
TLS. The key must be provided in PEM format.
properties:
value:
description: The value as plain text.
type: string
valueFrom:
description: The value as a reference to a resource.
properties:
secretKeyRef:
description: Refers to the value of a specific
key in a Secret. You must provide `name` and
`namespace` of the Secret, as well as the name
of the `key`.
properties:
key:
description: The name of the attribute of
the Secret holding the referenced value.
type: string
name:
description: The name of the Secret containing
the referenced value
type: string
namespace:
description: The name of the Namespace containing
the Secret with the referenced value.
type: string
type: object
type: object
type: object
skipCertificateValidation:
description: If `true`, the validation of certificates
is skipped. Default is `false`.
Expand Down
7 changes: 3 additions & 4 deletions config/crd/bases/telemetry.kyma-project.io_tracepipelines.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,7 @@ spec:
ca:
description: Defines an optional CA certificate for server
certificate verification when using TLS. The certificate
needs to be provided in PEM format.
must be provided in PEM format.
properties:
value:
description: The value as plain text.
Expand Down Expand Up @@ -243,8 +243,7 @@ spec:
type: object
cert:
description: Defines a client certificate to use when
using TLS. The certificate needs to be provided in PEM
format.
using TLS. The certificate must be provided in PEM format.
properties:
value:
description: The value as plain text.
Expand Down Expand Up @@ -283,7 +282,7 @@ spec:
type: boolean
key:
description: Defines the client key to use when using
TLS. The key needs to be provided in PEM format.
TLS. The key must be provided in PEM format.
properties:
value:
description: The value as plain text.
Expand Down
7 changes: 3 additions & 4 deletions config/development/telemetry.kyma-project.io_metricpipelines.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -247,7 +247,7 @@ spec:
ca:
description: Defines an optional CA certificate for server
certificate verification when using TLS. The certificate
needs to be provided in PEM format.
must be provided in PEM format.
properties:
value:
description: The value as plain text.
Expand Down Expand Up @@ -278,8 +278,7 @@ spec:
type: object
cert:
description: Defines a client certificate to use when
using TLS. The certificate needs to be provided in PEM
format.
using TLS. The certificate must be provided in PEM format.
properties:
value:
description: The value as plain text.
Expand Down Expand Up @@ -318,7 +317,7 @@ spec:
type: boolean
key:
description: Defines the client key to use when using
TLS. The key needs to be provided in PEM format.
TLS. The key must be provided in PEM format.
properties:
value:
description: The value as plain text.
Expand Down
15 changes: 8 additions & 7 deletions controllers/telemetry/logpipeline_controller_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,13 +27,14 @@ import (

var (
testLogPipelineConfig = logpipelinereconciler.Config{
DaemonSet: types.NamespacedName{Name: "test-telemetry-fluent-bit", Namespace: "default"},
ParsersConfigMap: types.NamespacedName{Name: "test-telemetry-fluent-bit-parsers", Namespace: "default"},
LuaConfigMap: types.NamespacedName{Name: "test-telemetry-fluent-bit-luascripts", Namespace: "default"},
SectionsConfigMap: types.NamespacedName{Name: "test-telemetry-fluent-bit-sections", Namespace: "default"},
FilesConfigMap: types.NamespacedName{Name: "test-telemetry-fluent-bit-files", Namespace: "default"},
EnvSecret: types.NamespacedName{Name: "test-telemetry-fluent-bit-env", Namespace: "default"},
OverrideConfigMap: types.NamespacedName{Name: "override-config", Namespace: "default"},
DaemonSet: types.NamespacedName{Name: "test-telemetry-fluent-bit", Namespace: "default"},
ParsersConfigMap: types.NamespacedName{Name: "test-telemetry-fluent-bit-parsers", Namespace: "default"},
LuaConfigMap: types.NamespacedName{Name: "test-telemetry-fluent-bit-luascripts", Namespace: "default"},
SectionsConfigMap: types.NamespacedName{Name: "test-telemetry-fluent-bit-sections", Namespace: "default"},
FilesConfigMap: types.NamespacedName{Name: "test-telemetry-fluent-bit-files", Namespace: "default"},
EnvSecret: types.NamespacedName{Name: "test-telemetry-fluent-bit-env", Namespace: "default"},
OutputTLSConfigSecret: types.NamespacedName{Name: "test-telemetry-fluent-bit-output-tls-config", Namespace: "default"},
OverrideConfigMap: types.NamespacedName{Name: "override-config", Namespace: "default"},
DaemonSetConfig: logpipelineresources.DaemonSetConfig{
FluentBitImage: "my-fluent-bit-image",
FluentBitConfigPrepperImage: "my-fluent-bit-config-image",
Expand Down
108 changes: 81 additions & 27 deletions docs/user/02-logs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,8 @@ In the following steps, you can see how to construct and deploy a typical LogPip

### Step 1: Create a LogPipeline and output

To ship application logs to a new output, create a resource of the kind `LogPipeline`:
To ship application logs to a new output, create a resource of the kind `LogPipeline`:

```yaml
kind: LogPipeline
apiVersion: telemetry.kyma-project.io/v1alpha1
Expand All @@ -71,15 +72,17 @@ spec:
password:
value: "not-required"
```

An output is a data destination configured by a [Fluent Bit output](https://docs.fluentbit.io/manual/pipeline/outputs) of the relevant type. The LogPipeline supports the following output types:

- **http**, which sends the data to the specified HTTP destination. The output is designed to integrate with a [Fluentd HTTP Input](https://docs.fluentd.org/input/http), which opens up a huge ecosystem of integration possibilities.
- **grafana-loki**, which sends the data to the Kyma-internal Loki instance.
>**NOTE:** This output is considered legacy and is only provided for backward compatibility with the [deprecated](https://github.com/kyma-project/kyma/releases/tag/2.9.0) in-cluster Loki instance. It might not be compatible with the latest Loki versions. For integration with a custom Loki installation, use the `custom` output with the name `loki` instead. See also [Installing a custom Loki stack in Kyma](https://github.com/kyma-project/examples/tree/main/loki).
- **custom**, which supports the configuration of any destination in the Fluent Bit configuration syntax.
- **custom**, which supports the configuration of any destination in the Fluent Bit configuration syntax.
>**CAUTION:** If you use a `custom` output, you put the LogPipeline in the [unsupported mode](#unsupported-mode).

See the following example of the `custom` output:

```yaml
spec:
output:
Expand All @@ -101,6 +104,7 @@ If you need selection mechanisms for application logs on the Namespace or contai
If you don't define any input, it's collected from all Namespaces, except the system Namespaces `kube-system`, `istio-system`, `kyma-system`, which are excluded by default. For example, you can define the Namespaces to include in the input collection, exclude Namespaces from the input collection, or choose that only system Namespaces are included. Learn more about the available [parameters and attributes](resources/02-logpipeline.md).

The following example collects input from all Namespaces excluding `kyma-system` and only from the `istio-proxy` containers:

```yaml
kind: LogPipeline
apiVersion: telemetry.kyma-project.io/v1alpha1
Expand Down Expand Up @@ -170,45 +174,93 @@ Telemetry Manager supports different types of [Fluent Bit filter](https://docs.f

### Step 4: Add authentication details from Secrets

Integrations into external systems usually need authentication details dealing with sensitive data. To handle that data properly in Secrets, the LogPipeline supports the reference of Secrets.
Integrations into external systems usually need authentication details dealing with sensitive data. To handle that data properly in Secrets, the LogPipeline supports the reference of Secrets. At the moment, mutual TLS (mTLS) and Basic Authentication are supported.

Using the **http** output definition and the **valueFrom** attribute, you can map Secret keys as in the following **http** output example:
Using the **http** output definition and the **valueFrom** attribute, you can map Secret keys as in the following examples:

```yaml
kind: LogPipeline
apiVersion: telemetry.kyma-project.io/v1alpha1
metadata:
name: http-backend
spec:
output:
http:
<div tabs>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent idea to make the code samples expandable, much better reading experience!

<details>
<summary>Mutual TLS</summary>

```yaml
apiVersion: telemetry.kyma-project.io/v1alpha1
kind: LogPipeline
metadata:
name: http-backend
spec:
output:
http:
dedot: false
port: "80"
uri: "/"
host:
valueFrom:
valueFrom:
secretKeyRef:
name: http-backend-credentials
namespace: default
key: HTTP_ENDPOINT
user:
valueFrom:
name: http-backend-credentials
namespace: default
key: HTTP_ENDPOINT
tls:
cert:
valueFrom:
secretKeyRef:
name: http-backend-credentials
namespace: default
key: HTTP_USER
password:
valueFrom:
name: http-backend-credentials
namespace: default
key: TLS_CERT
key:
valueFrom:
secretKeyRef:
name: http-backend-credentials
namespace: default
key: HTTP_PASSWORD
name: http-backend-credentials
namespace: default
key: TLS_KEY
input:
...
filters:
...
```
```

</details>

<details>
<summary>Basic Authentication</summary>

```yaml
apiVersion: telemetry.kyma-project.io/v1alpha1
kind: LogPipeline
metadata:
name: http-backend
spec:
output:
http:
dedot: false
port: "80"
uri: "/"
host:
valueFrom:
secretKeyRef:
name: http-backend-credentials
namespace: default
key: HTTP_ENDPOINT
user:
valueFrom:
secretKeyRef:
name: http-backend-credentials
namespace: default
key: HTTP_USER
password:
valueFrom:
secretKeyRef:
name: http-backend-credentials
namespace: default
key: HTTP_PASSWORD
input:
...
filters:
...
```

</details>

</div>

The related Secret must fulfill the referenced name and Namespace, and contain the mapped key as in the following example:

Expand All @@ -221,6 +273,8 @@ stringData:
HTTP_ENDPOINT: https://myhost/logs
HTTP_USER: myUser
HTTP_PASSWORD: XXX
TLS_CERT: ...
TLS_KEY: ...
```

To leverage data provided by the Kubernetes Secrets in a `custom` output definition, use placeholder expressions for the data provided by the Secret, then specify the actual mapping to the Secret keys in the **variables** section, like in the following example:
Expand Down
Loading