Skip to content

Commit d6a0c93

Browse files
tssuryatssurya
committed
NPEP: Iron out Egress Support API Design
Signed-off-by: Surya Seetharaman <[email protected]>
1 parent bb2cafa commit d6a0c93

File tree

1 file changed

+240
-4
lines changed

1 file changed

+240
-4
lines changed

‎npep/npep-126-egress-traffic-control.md

+240-4
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# NPEP-126: Add northbound traffic support in (B)ANP API
22

33
* Issue: [#126](https://github.com/kubernetes-sigs/network-policy-api/issues/126)
4-
* Status: Provisional
4+
* Status: Implementable
55

66
## TLDR
77

@@ -76,13 +76,249 @@ selected cluster workloads to k8s-apiservers for securing the server.
7676

7777
## API
7878

79-
(... details, can point to PR with changes)
79+
Proof of Concept for the API design details can be found here: https://github.com/kubernetes-sigs/network-policy-api/pull/143
8080

81+
### Implementing egress traffic control towards cluster nodes
82+
83+
This NPEP proposes to add a new type of `AdminNetworkPolicyEgressPeer` called `Nodes`
84+
to be able to explicitly select nodes (based on the node's labels) in the cluster.
85+
This ensures that if the list of IPs on a node OR list of nodes change, the users
86+
don't need to manually intervene to include those new IPs. The label selectors will
87+
take care of this automatically.
88+
89+
```
90+
// AdminNetworkPolicyEgressPeer defines a peer to allow traffic to.
91+
// Exactly one of the selector pointers must be set for a given peer. If a
92+
// consumer observes none of its fields are set, they must assume an unknown
93+
// option has been specified and fail closed.
94+
// +kubebuilder:validation:MaxProperties=1
95+
// +kubebuilder:validation:MinProperties=1
96+
type AdminNetworkPolicyEgressPeer struct {
97+
<snipped>
98+
// Nodes defines a way to select a set of nodes in
99+
// in the cluster. This field follows standard label selector
100+
// semantics; if present but empty, it selects all Nodes.
101+
//
102+
// Support: Core
103+
//
104+
// +optional
105+
Nodes *metav1.LabelSelector `json:"nodes,omitempty"`
106+
}
107+
```
108+
109+
Note that `AdminNetworkPolicyPeer` will changed to
110+
`AdminNetworkPolicyEgressPeer` and `AdminNetworkPolicyIngressPeer` since ingress and
111+
egress peers have started to diverge at this point and it is easy to
112+
maintain it with two sets of peer definitions.
113+
This ensures nodes can be referred to only as "egress peers".
114+
115+
Example: Admin wants to deny egress traffic from tenants who don't have
116+
`restricted`, `confidential` or `internal` level security clearance
117+
to control-plane nodes at 443 and 6443 ports in the cluster
118+
119+
```
120+
apiVersion: policy.networking.k8s.io/v1alpha1
121+
kind: AdminNetworkPolicy
122+
metadata:
123+
name: node-as-egress-peers
124+
spec:
125+
priority: 55
126+
subject:
127+
namespaces:
128+
matchExpressions:
129+
- {key: security, operator: notIn, values: [restricted, confidential, internal]}
130+
egress:
131+
- name: "deny-all-egress-to-kapi-server"
132+
action: "Deny"
133+
to:
134+
- nodes:
135+
matchLabels:
136+
node-role.kubernetes.io/control-plane: true
137+
ports:
138+
- portNumber:
139+
protocol: TCP
140+
port: 443
141+
- portNumber:
142+
protocol: TCP
143+
port: 6443
144+
```
145+
146+
### Implementing egress traffic control towards external destinations
147+
148+
This NPEP proposes to add a new type of `AdminNetworkPolicyEgressPeer` called `ExternalNetworks`
149+
to be able to explicitly select external destinations (based on the `externalNetworkSet`'s
150+
labels) in the cluster. This peer type will not be supported in `AdminNetworkPolicyIngressPeer`.
151+
152+
```
153+
// AdminNetworkPolicyEgressPeer defines a peer to allow traffic to.
154+
// Exactly one of the selector pointers must be set for a given peer. If a
155+
// consumer observes none of its fields are set, they must assume an unknown
156+
// option has been specified and fail closed.
157+
// +kubebuilder:validation:MaxProperties=1
158+
// +kubebuilder:validation:MinProperties=1
159+
type AdminNetworkPolicyEgressPeer struct {
160+
<snipped>
161+
// ExternalNetworks defines a way to select ExternalNetworkSets
162+
// that consist of network CIDRs that live outside the cluster as a peer.
163+
// This field follows standard label selector semantics; if present
164+
// but empty, it selects all ExternalNetworkSets defined in the cluster.
165+
//
166+
// Support: Core
167+
//
168+
// +optional
169+
ExternalNetworks *metav1.LabelSelector `json:"externalNetworks,omitempty"`
170+
}
171+
```
172+
173+
An `externalNetworkSet` is a new object used to define a set of networks outside
174+
the cluster. This is defined as a new object so that its easy for users to group
175+
their external entities and then refer to that group from the different egress rules.
176+
This ensures consistency without the users having to manually change the ANP/BANP
177+
to accommodate the changing needs of the network cidrs. They can directly edit the
178+
new CRD which guarantees all rules effecting that peer will take effect automatically.
179+
In the future when we define `DeveloperNetworkPolicies`, this will come even more handy
180+
for uniformity specially across `Pass` actions.
181+
182+
```
183+
// ExternalNetworkSet is a cluster level resource that is used to define
184+
// a set of networks outsides the cluster which can be referred to from
185+
// the AdminNetworkPolicy && BaselineAdminNetworkPolicy APIs as an external peer
186+
type ExternalNetworkSet struct {
187+
metav1.TypeMeta `json:",inline"`
188+
metav1.ObjectMeta `json:"metadata"`
189+
190+
// Specification of the desired behavior of ExternalNetworkSet.
191+
Spec ExternalNetworkSetSpec `json:"spec"`
192+
}
193+
194+
// ExternalNetworkSetSpec defines the desired state of ExternalNetworkSet.
195+
type ExternalNetworkSetSpec struct {
196+
// Networks is the list of NetworkCIDR (both v4 & v6) that can be used to define
197+
// external destinations.
198+
// A total of 100 CIDRs will be allowed in each NetworkSet instance.
199+
// ANP & BANP APIs may use the .spec.in(e)gress.from(to).externalNetworks selector
200+
// to select a set of external networks
201+
// +optional
202+
// +kubebuilder:validation:MaxItems=100
203+
Networks []string `json:"networks,omitempty" validate:"omitempty,dive,cidr"`
204+
}
205+
```
206+
207+
Example: Let's define some sample sets of externalNetworks.
208+
209+
```
210+
apiVersion: policy.networking.k8s.io/v1alpha1
211+
kind: ExternalNetworkSet
212+
metadata:
213+
name: api-vm-network
214+
labels:
215+
network: intranet
216+
spec:
217+
networks:
218+
- 10.0.0.10/32
219+
- 10.0.0.11/32
220+
- 10.0.0.12/32
221+
---
222+
apiVersion: policy.networking.k8s.io/v1alpha1
223+
kind: ExternalNetworkSet
224+
metadata:
225+
name: storage-network
226+
labels:
227+
network: intranet
228+
spec:
229+
networks:
230+
- 95.90.241.22/28
231+
- 200.0.65.0/24
232+
---
233+
apiVersion: policy.networking.k8s.io/v1alpha1
234+
kind: ExternalNetworkSet
235+
metadata:
236+
name: dns-network
237+
labels:
238+
network: dns
239+
spec:
240+
networks:
241+
- 195.90.241.22/28
242+
- 111.0.65.0/29
243+
---
244+
apiVersion: policy.networking.k8s.io/v1alpha1
245+
kind: ExternalNetworkSet
246+
metadata:
247+
name: global-network
248+
labels:
249+
network: internet
250+
spec:
251+
networks:
252+
- 0.0.0.0/0
253+
```
254+
```
255+
$ oc get ens
256+
NAME NETWORKS AGE
257+
api-vm-network ["10.0.0.10/32","10.0.0.11/32","10.0.0.12/32"] 2s
258+
dns-network ["195.90.241.22/28","111.0.65.0/29"] 2s
259+
global-network ["0.0.0.0/0"] 2s
260+
storage-network ["95.90.241.22/28","200.0.65.0/24"] 2s
261+
```
262+
263+
Let's define ANP and BANP that refer to these external networks:
264+
```
265+
apiVersion: policy.networking.k8s.io/v1alpha1
266+
kind: AdminNetworkPolicy
267+
metadata:
268+
name: cluster-egress-controls
269+
spec:
270+
priority: 70
271+
subject:
272+
namespaces: {}
273+
egress:
274+
- name: "allow-all-egress-to-intranet"
275+
action: "Allow"
276+
to:
277+
- externalNetworks:
278+
matchLabels:
279+
network: intranet
280+
- name: "allow-egress-to-external-dns"
281+
action: "Allow"
282+
to:
283+
- externalNetworks:
284+
matchLabels:
285+
network: dns
286+
ports:
287+
- portNumber:
288+
protocol: UDP
289+
port: 53
290+
- name: "pass-egress-to-internet"
291+
action: "Pass"
292+
to:
293+
- externalNetworks:
294+
matchLabels:
295+
network: internet
296+
---
297+
apiVersion: policy.networking.k8s.io/v1alpha1
298+
kind: BaselineAdminNetworkPolicy
299+
metadata:
300+
name: default
301+
spec:
302+
subject:
303+
namespaces: {}
304+
egress:
305+
- name: "deny-egress-to-internet"
306+
action: "Deny"
307+
to:
308+
- externalNetworks:
309+
matchLabels:
310+
network: internet
311+
```
312+
This allows admins to specify all cluster workloads can talk to
313+
the company's internet CIDR's and the external DNS network. It
314+
also allows them to put up default guardrails of not allowing
315+
any other egress traffic towards internet in a BANP.
81316

82317
## Alternatives
83318

84-
(List other design alternatives and why we did not go in that
85-
direction)
319+
* Instead of defining a new object called `ExternalNetworkSet`,
320+
we can go ahead with putting the CIDR ranges directly into the
321+
main ANP/BANP objects.
86322

87323
## References
88324

0 commit comments

Comments
 (0)