Skip to content

Commit 4c554b8

Browse files
tssuryatssurya
committed
NPEP: Iron out Egress Support API Design
Signed-off-by: Surya Seetharaman <[email protected]>
1 parent bb2cafa commit 4c554b8

File tree

1 file changed

+250
-4
lines changed

1 file changed

+250
-4
lines changed

‎npep/npep-126-egress-traffic-control.md

+250-4
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# NPEP-126: Add northbound traffic support in (B)ANP API
22

33
* Issue: [#126](https://github.com/kubernetes-sigs/network-policy-api/issues/126)
4-
* Status: Provisional
4+
* Status: Implementable
55

66
## TLDR
77

@@ -76,13 +76,259 @@ selected cluster workloads to k8s-apiservers for securing the server.
7676

7777
## API
7878

79-
(... details, can point to PR with changes)
79+
Proof of Concept for the API design details can be found here: https://github.com/kubernetes-sigs/network-policy-api/pull/143
8080

81+
### Implementing egress traffic control towards cluster nodes
82+
83+
This NPEP proposes to add a new type of `AdminNetworkPolicyEgressPeer` called `Nodes`
84+
to be able to explicitly select nodes (based on the node's labels) in the cluster.
85+
This ensures that if the list of IPs on a node OR list of nodes change, the users
86+
don't need to manually intervene to include those new IPs. The label selectors will
87+
take care of this automatically. Note that the nodeIPs that this type of peer matches
88+
on are the IPs present in `Node.Status.Addresses` field of the node.
89+
90+
```
91+
// AdminNetworkPolicyEgressPeer defines a peer to allow traffic to.
92+
// Exactly one of the selector pointers must be set for a given peer. If a
93+
// consumer observes none of its fields are set, they must assume an unknown
94+
// option has been specified and fail closed.
95+
// +kubebuilder:validation:MaxProperties=1
96+
// +kubebuilder:validation:MinProperties=1
97+
type AdminNetworkPolicyEgressPeer struct {
98+
<snipped>
99+
// Nodes defines a way to select a set of nodes in
100+
// in the cluster. This field follows standard label selector
101+
// semantics; if present but empty, it selects all Nodes.
102+
//
103+
// Support: Core
104+
//
105+
// +optional
106+
Nodes *metav1.LabelSelector `json:"nodes,omitempty"`
107+
}
108+
```
109+
110+
Note that `AdminNetworkPolicyPeer` will be changed to
111+
`AdminNetworkPolicyEgressPeer` and `AdminNetworkPolicyIngressPeer` since ingress and
112+
egress peers have started to diverge at this point and it is easy to
113+
maintain it with two sets of peer definitions.
114+
This ensures nodes can be referred to only as "egress peers".
115+
116+
Example: Admin wants to deny egress traffic from tenants who don't have
117+
`restricted`, `confidential` or `internal` level security clearance
118+
to control-plane nodes at 443 and 6443 ports in the cluster
119+
120+
```
121+
apiVersion: policy.networking.k8s.io/v1alpha1
122+
kind: AdminNetworkPolicy
123+
metadata:
124+
name: node-as-egress-peers
125+
spec:
126+
priority: 55
127+
subject:
128+
namespaces:
129+
matchExpressions:
130+
- {key: security, operator: notIn, values: [restricted, confidential, internal]}
131+
egress:
132+
- name: "deny-all-egress-to-kapi-server"
133+
action: "Deny"
134+
to:
135+
- nodes:
136+
matchLabels:
137+
node-role.kubernetes.io/control-plane:
138+
ports:
139+
- portNumber:
140+
protocol: TCP
141+
port: 443
142+
- portNumber:
143+
protocol: TCP
144+
port: 6443
145+
```
146+
147+
### Implementing egress traffic control towards external destinations
148+
149+
This NPEP proposes to add a new type of `AdminNetworkPolicyEgressPeer` called `ExternalNetworks`
150+
to be able to explicitly select external destinations (based on the `externalNetworkSet`'s
151+
labels) in the cluster. This peer type will not be supported in `AdminNetworkPolicyIngressPeer`.
152+
153+
```
154+
// AdminNetworkPolicyEgressPeer defines a peer to allow traffic to.
155+
// Exactly one of the selector pointers must be set for a given peer. If a
156+
// consumer observes none of its fields are set, they must assume an unknown
157+
// option has been specified and fail closed.
158+
// +kubebuilder:validation:MaxProperties=1
159+
// +kubebuilder:validation:MinProperties=1
160+
type AdminNetworkPolicyEgressPeer struct {
161+
<snipped>
162+
// ExternalNetworks defines a way to select ExternalNetworkSets
163+
// that consist of network CIDRs that live outside the cluster as a peer.
164+
// This field follows standard label selector semantics; if present
165+
// but empty, it selects all ExternalNetworkSets defined in the cluster.
166+
//
167+
// Support: Core
168+
//
169+
// +optional
170+
ExternalNetworks *metav1.LabelSelector `json:"externalNetworks,omitempty"`
171+
}
172+
```
173+
174+
An `ExternalNetworkSet` is a new object used to define a set of networks outside
175+
the cluster. This is defined as a new object so that it's easy for users to group
176+
their external entities and then refer to that group from the different egress rules.
177+
This ensures consistency without the users having to manually change the relevant egress
178+
rules in the ANP/BANP objects while accommodating the changing needs of the network cidrs.
179+
They can directly edit the new object which guarantees all rules effecting that peer will
180+
take effect automatically.
181+
In the future if we define `DeveloperNetworkPolicy`, this will come even more handy
182+
for uniformity, especially across `Pass` action.
183+
184+
```
185+
// ExternalNetworkSet is a cluster level resource that is used to define
186+
// a set of networks outsides the cluster which can be referred to from
187+
// the AdminNetworkPolicy && BaselineAdminNetworkPolicy APIs as an external peer
188+
type ExternalNetworkSet struct {
189+
metav1.TypeMeta `json:",inline"`
190+
metav1.ObjectMeta `json:"metadata"`
191+
192+
// Specification of the desired behavior of ExternalNetworkSet.
193+
Spec ExternalNetworkSetSpec `json:"spec"`
194+
}
195+
196+
// ExternalNetworkSetSpec defines the desired state of ExternalNetworkSet.
197+
type ExternalNetworkSetSpec struct {
198+
// Networks is the list of CIDRs (both v4 & v6) that can be used to define
199+
// external destinations.
200+
// A total of 100 CIDRs will be allowed in each NetworkSet instance.
201+
// ANP & BANP APIs may use the .spec.egress.to.externalNetworks selector
202+
// to select a set of external networks
203+
// +optional
204+
// +kubebuilder:validation:MaxItems=100
205+
Networks []string `json:"networks,omitempty" validate:"omitempty,dive,cidr"`
206+
}
207+
```
208+
209+
Note that the API expects `networks` to be a set of "external" destinations.
210+
However if user puts a podCIDR, nodeCIDR, serviceCIDR or other intra-cluster
211+
networks, it is implementation-defined as to what the behaviour will be. Not
212+
all implementations can correctly define the boundary between "internal" and
213+
"external" destinations with respect to a Kubernetes cluster.
214+
215+
Example: Let's define some sample sets of externalNetworks.
216+
217+
```
218+
apiVersion: policy.networking.k8s.io/v1alpha1
219+
kind: ExternalNetworkSet
220+
metadata:
221+
name: api-vm-network
222+
labels:
223+
network: intranet
224+
spec:
225+
networks:
226+
- 10.0.0.10/32
227+
- 10.0.0.11/32
228+
- 10.0.0.12/32
229+
---
230+
apiVersion: policy.networking.k8s.io/v1alpha1
231+
kind: ExternalNetworkSet
232+
metadata:
233+
name: storage-network
234+
labels:
235+
network: intranet
236+
spec:
237+
networks:
238+
- 95.90.241.22/28
239+
- 200.0.65.0/24
240+
---
241+
apiVersion: policy.networking.k8s.io/v1alpha1
242+
kind: ExternalNetworkSet
243+
metadata:
244+
name: dns-network
245+
labels:
246+
network: dns
247+
spec:
248+
networks:
249+
- 195.90.241.22/28
250+
- 111.0.65.0/29
251+
---
252+
apiVersion: policy.networking.k8s.io/v1alpha1
253+
kind: ExternalNetworkSet
254+
metadata:
255+
name: global-network
256+
labels:
257+
network: internet
258+
spec:
259+
networks:
260+
- 0.0.0.0/0
261+
```
262+
```
263+
$ oc get externalnetworkset
264+
NAME NETWORKS AGE
265+
api-vm-network ["10.0.0.10/32","10.0.0.11/32","10.0.0.12/32"] 2s
266+
dns-network ["195.90.241.22/28","111.0.65.0/29"] 2s
267+
global-network ["0.0.0.0/0"] 2s
268+
storage-network ["95.90.241.22/28","200.0.65.0/24"] 2s
269+
```
270+
271+
Let's define ANP and BANP that refer to these external networks:
272+
```
273+
apiVersion: policy.networking.k8s.io/v1alpha1
274+
kind: AdminNetworkPolicy
275+
metadata:
276+
name: cluster-egress-controls
277+
spec:
278+
priority: 70
279+
subject:
280+
namespaces: {}
281+
egress:
282+
- name: "allow-all-egress-to-intranet"
283+
action: "Allow"
284+
to:
285+
- externalNetworks:
286+
matchLabels:
287+
network: intranet
288+
- name: "allow-egress-to-external-dns"
289+
action: "Allow"
290+
to:
291+
- externalNetworks:
292+
matchLabels:
293+
network: dns
294+
ports:
295+
- portNumber:
296+
protocol: UDP
297+
port: 53
298+
- name: "pass-egress-to-internet"
299+
action: "Pass"
300+
to:
301+
- externalNetworks:
302+
matchLabels:
303+
network: internet
304+
---
305+
apiVersion: policy.networking.k8s.io/v1alpha1
306+
kind: BaselineAdminNetworkPolicy
307+
metadata:
308+
name: default
309+
spec:
310+
subject:
311+
namespaces: {}
312+
egress:
313+
- name: "deny-egress-to-internet"
314+
action: "Deny"
315+
to:
316+
- externalNetworks:
317+
matchLabels:
318+
network: internet
319+
```
320+
This allows admins to specify all cluster workloads can talk to
321+
the company's internet CIDR's and the external DNS network. It
322+
also allows them to put up default guardrails of not allowing
323+
any other egress traffic towards internet in a BANP.
81324

82325
## Alternatives
83326

84-
(List other design alternatives and why we did not go in that
85-
direction)
327+
* Instead of defining a new object called `ExternalNetworkSet`,
328+
we can go ahead with putting the CIDR ranges directly into the
329+
main ANP/BANP objects. This alternative is better if the set of
330+
CIDRs are mostly a constant and are only referred to from one or two
331+
egress rules. It increases readability.
86332

87333
## References
88334

0 commit comments

Comments
 (0)