feat(client/v3/naming): add Attributes into Endpoint and deprecate Metadata #19785

amosehiguese · 2025-04-23T14:09:08Z

This implementation adds the new Attributes *attributes.Attributes into Endpoint and deprecate Metadata accordingly. The metadata field is still supported for backward compatibility

Fixes #19706

cc: @ahrtr @siyuanfoundation

Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.

k8s-ci-robot · 2025-04-23T14:09:13Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: amosehiguese
Once this PR has been reviewed and has the lgtm label, please assign serathius for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-04-23T14:09:19Z

Hi @amosehiguese. Thanks for your PR.

I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ahrtr · 2025-04-23T14:19:45Z

client/v3/naming/resolver/resolver.go

 			r.cc.UpdateState(gresolver.State{Endpoints: eps})
 		}
 	}
 }

-func convertToGRPCEndpoint(ups map[string]*endpoints.Update) []gresolver.Endpoint {
+func convertToGRPCEndpoint(lg *zap.Logger, ups map[string]*endpoints.Update) []gresolver.Endpoint {


suggest not to change the signature of the function. We don't know how users use the resovler, it may generate huge number of warning message.

Suggested change

func convertToGRPCEndpoint(lg *zap.Logger, ups map[string]*endpoints.Update) []gresolver.Endpoint {

func convertToGRPCEndpoint(ups map[string]*endpoints.Update) []gresolver.Endpoint {

I was thinking since it is a private function and only called within the watch function(in the entire etcd repo), it wouldn't have much impact.

That said, do i revert or leave it as is?

I could take another approach though

please revert this change.

usually it isn't a good pattern to print log in a lower level or utility package

I will revert then. Thank you for the feedback

ahrtr · 2025-04-23T14:46:55Z

please squash the commits.

We also need to update the doc https://etcd.io/docs/v3.5/dev-guide/grpc_naming/

the Metadata has deprecated, use Attributes instead
Mark the gRPC naming and discovery feature as experimental, depending on how the resolver package in grpc-go evolves.

amosehiguese · 2025-04-23T14:49:57Z

Alright. I'll do just that.

ahrtr · 2025-04-23T15:56:30Z

the Metadata has deprecated, use Attributes instead

let's update this first together with this PR.

after this PR gets merged, backport it to 3.6 and 3.5.
raise a PR in etcd-io/website to update the doc

Mark the gRPC naming and discovery feature as experimental, depending on how the resolver package in grpc-go evolves.

we will discuss & resolve this separately.

cc @dfawley @fuweid @serathius @ivanvc

ahrtr · 2025-04-23T15:58:43Z

/ok-to-test

ahrtr · 2025-04-23T16:17:01Z

client/v3/naming/resolver/resolver.go

-					Metadata: up.Endpoint.Metadata,
+					Addr:       up.Endpoint.Addr,
+					Metadata:   up.Endpoint.Metadata,
+					Attributes: up.Endpoint.Attributes,


@dfawley

Probably eventually we might want to use Endpoint.Attributes instead of Address.Attributes, as we only care about the LB policy?

Currently we assume each endpoint has only one address, but actually it may contain multiple addresses (we won't change the behavior for now).

Yeah, I saw it and then realised that the attributes field in the Endpoint was specific to LB-Policy which is the usecase for etcd and wondered why that wasn't used instead. But I have clarity on that now. Thanks

Probably eventually we might want to use Endpoint.Attributes instead of Address.Attributes, as we only care about the LB policy?

It seems that this comment hasn't been resolved yet.

We'll have to wait on @dfawley, I guess

The attributes on the address are passed to our code that creates and manages the connection to that address. This means if there are two Addresses between name resolver updates with the same IP address but non-equal attributes, we will consider them different and reconnect.

The attributes on the endpoint are for anything else you might want to do. gRPC won't look at your attributes.

One important thing that is still not clear to me: what is your use case for attributes? Why are you adding them into gRPC endpoints/addresses?

One important thing that is still not clear to me: what is your use case for attributes? Why are you adding them into gRPC endpoints/addresses?

This is a good question, which I am not 100% sure either. But I believe the concept should be tightly coupled with gprc-go.

In the first place, we (previous maintainer) added Metadata (see below), which was passed to grpc-go's Address.Metadata. Based on the comment, it's used for load balancing decision.

etcd/client/v3/naming/endpoints/endpoints.go

Lines 32 to 35 in f5dbde1

// Metadata is the information associated with Addr, which may be used

// to make load balancing decision.

// Since etcd 3.1

Metadata any

This means if there are two Addresses between name resolver updates with the same IP address but non-equal attributes, we will consider them different and reconnect.

It seems the only usage of the Address.Attribute is to differentiate two addresses with the same IP address? I originally thought grpc-go would transparently pass through any data users attach—Attributes—to the server.

Actually based on current implementation, each endpoint has only one address inside.

The attributes on the endpoint are for anything else you might want to do. gRPC won't look at your attributes.

In that case, then I don't think we should use it if users do not use it either. Would grpc-go transparently pass through the endpoint Attributes—to the server?

codecov · 2025-04-23T16:21:50Z

Codecov Report

Attention: Patch coverage is 85.71429% with 6 lines in your changes missing coverage. Please review.

Project coverage is 68.81%. Comparing base (8f933a5) to head (1264be5).
Report is 9 commits behind head on main.

Files with missing lines	Patch %	Lines
client/v3/naming/resolver/resolver.go	50.00%	5 Missing and 1 partial ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
client/v3/naming/endpoints/endpoints.go	`100.00% <ø> (ø)`
client/v3/naming/endpoints/endpoints_impl.go	`75.20% <100.00%> (+3.50%)`	⬆️
server/proxy/grpcproxy/register.go	`70.58% <100.00%> (-4.97%)`	⬇️
client/v3/naming/resolver/resolver.go	`74.24% <50.00%> (-6.12%)`	⬇️

... and 16 files with indirect coverage changes

@@           Coverage Diff           @@
##             main   #19785   +/-   ##
=======================================
  Coverage   68.81%   68.81%           
=======================================
  Files         421      421           
  Lines       35863    35894   +31     
=======================================
+ Hits        24678    24701   +23     
- Misses       9754     9764   +10     
+ Partials     1431     1429    -2

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8f933a5...1264be5. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ahrtr · 2025-04-23T16:27:50Z

You also need to persist the new added attributes (line 41 in internal/update.go), and read it back in List,

etcd/client/v3/naming/endpoints/endpoints_impl.go

Lines 175 to 192 in 8f933a5

    
           func (m *endpointManager) List(ctx context.Context) (Key2EndpointMap, error) { 
        
           	key := m.target + "/" 
        
           	resp, err := m.client.Get(ctx, key, clientv3.WithPrefix(), clientv3.WithSerializable()) 
        
           	if err != nil { 
        
           		return nil, err 
        
           	} 
        
           	eps := make(Key2EndpointMap) 
        
           	for _, kv := range resp.Kvs { 
        
           		var iup internal.Update 
        
           		if err := json.Unmarshal(kv.Value, &iup); err != nil { 
        
           			continue 
        
           		} 
        
           		eps[string(kv.Key)] = Endpoint{Addr: iup.Addr, Metadata: iup.Metadata} 
        
           	} 
        
           	return eps, nil 
        
           }

amosehiguese · 2025-04-23T16:27:59Z

let's update this first together with this PR.

Alright.

after this PR gets merged, backport it to 3.6 and 3.5.

I'm on standby.

raise a PR in etcd-io/website to update the doc

I will hold off on this then

ahrtr · 2025-04-23T16:33:11Z

You might also need to update grpcproxy

etcd/server/proxy/grpcproxy/register.go

Line 75 in 8f933a5

endpoint := endpoints.Endpoint{Addr: addr, Metadata: getMeta()}

amosehiguese · 2025-04-23T16:37:51Z

You also need to persist the new added attributes (line 41 in internal/update.go), and read it back in List,

etcd/client/v3/naming/endpoints/endpoints_impl.go

Lines 175 to 192 in 8f933a5

func (m *endpointManager) List(ctx context.Context) (Key2EndpointMap, error) {

key := m.target + "/"

resp, err := m.client.Get(ctx, key, clientv3.WithPrefix(), clientv3.WithSerializable())

if err != nil {

return nil, err

}

eps := make(Key2EndpointMap)

for _, kv := range resp.Kvs {

var iup internal.Update

if err := json.Unmarshal(kv.Value, &iup); err != nil {

continue

}

eps[string(kv.Key)] = Endpoint{Addr: iup.Addr, Metadata: iup.Metadata}

}

return eps, nil

}

Got it.

amosehiguese · 2025-04-23T16:45:12Z

You might also need to update grpcproxy

Pretty much adjust every other usage of the endpoint struct to account for the new Attributes field.

Will do that.

amosehiguese · 2025-04-24T00:49:13Z

Two files may also need adjustments but I thought to run it by you first.

👉 endpoints_test.go [integration tests]

etcd/tests/integration/clientv3/naming/endpoints_test.go

Line 46 in 8f933a5

e1 := endpoints.Endpoint{Addr: "127.0.0.1", Metadata: "metadata"}

👉 cluster.go [grpc-proxy]
In MembersList, If proxies are registered, it will lists them using their metadata and endpoint addresses.

etcd/server/proxy/grpcproxy/cluster.go

Line 146 in 8f933a5

m, err := decodeMeta(fmt.Sprint(upt.Metadata))

What would you have me do to these files?

ahrtr · 2025-04-24T09:26:31Z

I think

we need update (or add) the test to cover the new field Attributes
we shouldn't break any usage on the existing field Metadata.

amosehiguese · 2025-04-24T09:38:37Z

I think

we need update (or add) the test to cover the new field Attributes

we shouldn't break any usage on the existing field Metadata.

Alright then.

ahrtr · 2025-04-24T10:01:47Z

I just mark this as a priority/important-soon task.

As mentioned above #19785 (comment), I think we need to backport this PR to release-3.6 and release-3.5, of course we will evaluate the impact. The goal is to get it included in 3.6.0, so that we can remove it in 3.7; otherwise, we will have to completely get rid of the deprecated Metadata field in 3.8 (~6 years later based on current cadence)

cc @fuweid @serathius @ivanvc @jmhbnz

ahrtr · 2025-04-24T10:02:35Z

also cc grpc-go tech-lead @dfawley, thx for all the help so far.

amosehiguese · 2025-04-24T13:42:16Z

@ahrtr
Quick question. Are flaky tests expected in the end-to-end test suite? I’ve been running make test-e2e and sometimes it passes, but other times it fails without me changing anything. Just want to be sure if that’s expected or if something else is going on, because it’s been slowing down my push.

...and those tests take quite a bit of time to run

amosehiguese · 2025-04-24T13:43:48Z

I just mark this as a priority/important-soon task.

I will be done with this from my end once I'm able to resolve whether or not flaky tests are expected in the e2e suite

ahrtr · 2025-04-24T13:48:36Z

Quick question. Are flaky tests expected in the end-to-end test suite?

You need to ensure the test failures are not caused by your changes.

amosehiguese · 2025-04-24T14:27:33Z

Quick question. Are flaky tests expected in the end-to-end test suite?

You need to ensure the test failures are not caused by your changes.

I think I’ll have to run it against the main branch a couple of times to be sure.

amosehiguese · 2025-04-24T17:09:26Z

Same error occured on the main branch after running successfully in the previous run. Seems to look like a connection timeout.

What do you reckon I do? push?

cc: @ahrtr

ahrtr · 2025-04-24T17:36:30Z

Same error occured on the main branch after running successfully in the previous run. Seems to look like a connection timeout.

Windows is only tier 3 support, please try to test it on a linux platform.

What do you reckon I do? push?

Pushing to your dev branch won't break anything. Please review your PR yourself first to ensure it's the best you can deliver.

amosehiguese · 2025-04-24T17:44:23Z

Oh, I am actually working on a linux machine running on Azure cloud. This is just my backup pc, so I am forced to doing everything on the cloud.

Pushing to your dev branch won't break anything. Please review your PR yourself first to ensure it's the best you can deliver.

I definitely will. Thanks for the feedback

amosehiguese · 2025-04-25T05:53:47Z

@dfawley

Just a bit of clarification in your docs. This line to be precise. Is this meant to be, "Is it impossible..." or "It is impossible"?

https://github.com/grpc/grpc-go/blob/030938e543b4a721cd426d15611d50ecf097dcf3/attributes/attributes.go#L137

cc: @ahrtr

amosehiguese · 2025-04-25T09:14:46Z

I think

we need update (or add) the test to cover the new field Attributes

Attempting to do this would break these test cases.

👇

etcd/tests/integration/clientv3/naming/endpoints_test.go

Line 29 in f5dbde1

func TestEndpointManager(t *testing.T) {

👇

etcd/tests/integration/clientv3/naming/endpoints_test.go

Line 120 in f5dbde1

func TestEndpointManagerCRUD(t *testing.T) {

Mostly because of the design decision by grpc on the Attributes type to make it impossible to unmarshal attributes from a JSON representation

cc: @ahrtr

ahrtr · 2025-04-25T13:14:31Z

client/v3/naming/endpoints/endpoints.go

+	// Attributes contains arbitrary data about this address intended for
+	// consumption by the SubConn.
+	// Since etcd 3.5
+	Attributes *attributes.Attributes


We can assign this map to attributes.Attributes using the utilities New and WithValue

https://github.com/grpc/grpc-go/blob/4cedec40eb2ccfbe3f56bb15e894903111ada2d2/attributes/attributes.go#L44-L62

Suggested change

Attributes *attributes.Attributes

Attributes map[string]any

Unfortunately the the grpc attributes package is also Experimental

This approach will solve the JSON problem for sure. I will adjust it now. Meanwhile, you also mentioned the grpc attributes package is also Experimental. I mean we've already decided to take this direction so, things like this should be expected. Except you have something else in mind, of course. Are you having second thoughts?

cc: @ahrtr

Are you having second thoughts?

The alternative is to support generic type (a comparable type), so that the key of the map doesn't have to be a string. But let's keep it simple for now, as we are playing with grpc-go's experimental feature/packages.

+1 you should not use any experimental APIs in your own APIs, unless you want to declare them as experimental and your users are aware that you reserve the right to break them in the future.

You also need that disclaimer on any package that internally uses experimental APIs. But if you refrain from exposing it in your own APIs, then there should be some path forward that doesn't require changes to the user's code, too.

…tadata This implementation adds the new Attributes *attributes.Attributes into Endpoint and deprecate Metadata accordingly. The metadata field is still supported for backward compatibility Signed-off-by: amosehiguese <[email protected]>

dfawley · 2025-04-25T17:41:13Z

client/v3/naming/resolver/resolver.go

+			if attr != nil {
+				attr = attr.WithValue(k, v)
+			} else {
+				attr = attributes.New(k, v)


WithValue already does this, so you can simplify by deleting the else here.

dfawley · 2025-04-25T17:43:39Z

client/v3/naming/endpoints/internal/update.go

@@ -49,4 +49,11 @@ type Update struct {
 	// Metadata is not required for a custom naming implementation.
 	// Since etcd 3.1.
 	Metadata any
+
+	// Attributes is for storing and retrieving generic key/value pairs.
+	// Keys must be hashable, and users should define their own types for keys.


Keys must be hashable

This needs to be updated with the change to map.

dfawley · 2025-04-25T17:45:48Z

client/v3/naming/endpoints/endpoints.go

+	// Attributes contains arbitrary data about this address intended for
+	// consumption by the SubConn.
+	// Since etcd 3.5
+	Attributes *attributes.Attributes


+1 you should not use any experimental APIs in your own APIs, unless you want to declare them as experimental and your users are aware that you reserve the right to break them in the future.

You also need that disclaimer on any package that internally uses experimental APIs. But if you refrain from exposing it in your own APIs, then there should be some path forward that doesn't require changes to the user's code, too.

k8s-ci-robot added the area/clientv3 label Apr 23, 2025

k8s-ci-robot added the needs-ok-to-test label Apr 23, 2025

k8s-ci-robot added the size/M label Apr 23, 2025

ahrtr reviewed Apr 23, 2025

View reviewed changes

ahrtr mentioned this pull request Apr 23, 2025

Remove the deprecated grpc.DialContext #19772

Open

k8s-ci-robot added size/S and removed size/M labels Apr 23, 2025

amosehiguese force-pushed the issue-19706/deprecate-address-metadata branch from 0c64765 to cbf86a8 Compare April 23, 2025 15:36

k8s-ci-robot added ok-to-test and removed needs-ok-to-test labels Apr 23, 2025

ahrtr reviewed Apr 23, 2025

View reviewed changes

amosehiguese force-pushed the issue-19706/deprecate-address-metadata branch from cbf86a8 to da7ba13 Compare April 24, 2025 00:03

k8s-ci-robot added size/M and removed size/S labels Apr 24, 2025

ahrtr added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Apr 24, 2025

ahrtr mentioned this pull request Apr 24, 2025

Remove the dependency on grpc-go's experimental API (e.g. resolver & balancer) #15145

Open

amosehiguese force-pushed the issue-19706/deprecate-address-metadata branch from da7ba13 to 524789c Compare April 25, 2025 07:19

ahrtr reviewed Apr 25, 2025

View reviewed changes

amosehiguese force-pushed the issue-19706/deprecate-address-metadata branch from 524789c to 3ad2fb6 Compare April 25, 2025 16:16

k8s-ci-robot added the area/testing label Apr 25, 2025

amosehiguese force-pushed the issue-19706/deprecate-address-metadata branch from 3ad2fb6 to 1264be5 Compare April 25, 2025 16:23

dfawley reviewed Apr 25, 2025

View reviewed changes

	func convertToGRPCEndpoint(lg zap.Logger, ups map[string]endpoints.Update) []gresolver.Endpoint {
	func convertToGRPCEndpoint(ups map[string]*endpoints.Update) []gresolver.Endpoint {

	// Metadata is the information associated with Addr, which may be used
	// to make load balancing decision.
	// Since etcd 3.1
	Metadata any

feat(client/v3/naming): add Attributes into Endpoint and deprecate Metadata #19785

Are you sure you want to change the base?

feat(client/v3/naming): add Attributes into Endpoint and deprecate Metadata #19785

Conversation

amosehiguese commented Apr 23, 2025 • edited Loading

k8s-ci-robot commented Apr 23, 2025

k8s-ci-robot commented Apr 23, 2025

Choose a reason for hiding this comment

amosehiguese Apr 23, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amosehiguese Apr 23, 2025 • edited Loading

Choose a reason for hiding this comment

ahrtr commented Apr 23, 2025

amosehiguese commented Apr 23, 2025

ahrtr commented Apr 23, 2025

ahrtr commented Apr 23, 2025

Choose a reason for hiding this comment

amosehiguese Apr 23, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Apr 23, 2025 • edited Loading

Codecov Report

ahrtr commented Apr 23, 2025

amosehiguese commented Apr 23, 2025 • edited Loading

ahrtr commented Apr 23, 2025

amosehiguese commented Apr 23, 2025

amosehiguese commented Apr 23, 2025

amosehiguese commented Apr 24, 2025

ahrtr commented Apr 24, 2025

amosehiguese commented Apr 24, 2025

ahrtr commented Apr 24, 2025

ahrtr commented Apr 24, 2025

amosehiguese commented Apr 24, 2025 • edited Loading

amosehiguese commented Apr 24, 2025 • edited Loading

ahrtr commented Apr 24, 2025

amosehiguese commented Apr 24, 2025

amosehiguese commented Apr 24, 2025

ahrtr commented Apr 24, 2025

amosehiguese commented Apr 24, 2025 • edited Loading

amosehiguese commented Apr 25, 2025 • edited Loading

amosehiguese commented Apr 25, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amosehiguese commented Apr 23, 2025 •

edited

Loading

amosehiguese Apr 23, 2025 •

edited

Loading

amosehiguese Apr 23, 2025 •

edited

Loading

amosehiguese Apr 23, 2025 •

edited

Loading

codecov bot commented Apr 23, 2025 •

edited

Loading

amosehiguese commented Apr 23, 2025 •

edited

Loading

amosehiguese commented Apr 24, 2025 •

edited

Loading

amosehiguese commented Apr 24, 2025 •

edited

Loading

amosehiguese commented Apr 24, 2025 •

edited

Loading

amosehiguese commented Apr 25, 2025 •

edited

Loading

amosehiguese commented Apr 25, 2025 •

edited

Loading