DB Size small in one etcd node

[jmorciva]

Hi folks!

We have a three etcd node cluster and one of the nodes is not catching up, even after removing and adding it.
We are under 3.5.15 version. This is the actual status

+--------------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|               ENDPOINT               |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| tv2ml0186.pre.cm2.yyyyyyyyy.com:2379 | 84916b8cf9714100 |  3.5.15 |  3.0 MB |     false |      false |         9 |      15204 |              15204 |        |
| tv2ml0187.pre.cm2.yyyyyyyyy.com:2379 | b39a4c9e9902d4b0 |  3.5.15 |   25 MB |      true |      false |         9 |      15204 |              15204 |        |
| tv2ml0188.pre.cm2.yyyyyyyyy.com:2379 | 416213b72873f08a |  3.5.15 |   25 MB |     false |      false |         9 |      15204 |              15204 |        |
+--------------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

As you can see tv2ml0186 db size is 3.0 MB and the rest 25 MB.
The following message is being repeated

Feb 07 13:47:12 tv2ml0186.pre.cm2.yyyyyyyyy.com bash[3026066]: {"level":"warn","ts":"2025-02-07T13:47:12.994046+0100","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.879µs","request":"header:<ID:15325913222176503856 > lease_revoke:<id:772d947458ac665b>","response":"size:29","error":"lease not found"}

Any ideas!?

Thanks for your help!!

José Antonio

[ahrtr]

As you can see tv2ml0186 db size is 3.0 MB and the rest 25 MB.

This isn't a problem. Once you compact & defragment all the members, they will have similar size.

Feb 07 13:47:12 tv2ml0186.pre.cm2.yyyyyyyyy.com bash[3026066]: {"level":"warn","ts":"2025-02-07T13:47:12.994046+0100","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.879µs","request":"header:<ID:15325913222176503856 > lease_revoke:id:772d947458ac665b","response":"size:29","error":"lease not found"}

Did you only see this error on tv2ml0186 or on all members? Would you mind providing the complete logs? Can you manually check the leaseID 772d947458ac665b (run etcdctl lease list on each member)?

Do you have any idea what triggered this issue?

Looks like there is a potential bug for the scenario below,

• One member (assuming A) of a 3 members cluster is down;
• A lease is created.
• The cluster keeps running, and a snapshot gets created;
• The lease is revoked
• The member A gets started again. Then it will apply the lease revoking request, but the lease doesn't exist at all, so it will complain "lease not found"

even after removing and adding it.

Did you cleanup the data before you added the member into the cluster again? If not, please try again and do cleanup the data.

[ahrtr]

Looks like there is a potential bug for the scenario below,

• One member (assuming A) of a 3 members cluster is down;
• A lease is created.
• The cluster keeps running, and a snapshot gets created;
• The lease is revoked
• The member A gets started again. Then it will apply the lease revoking request, but the lease doesn't exist at all, so it will complain "lease not found"

This isn't correct. There isn't any issue in this scenario.