Selfhost dns api dev talk

[Marvo2011]

Please Report all bugs to selfhost dns api here:
acmesh-official#4291

Usage:

• create a new TXT record for a subdomainname with the needed prefix e.g. "_acme-challenge.example.com" (default) or "alias.example.com" (dns alias mode)
  • for wildcard subdomains add a second TXT record for the identical subdomainname
• edit the TXT record and note the ID in (...) behind the subdomainname
• export each subdomainname (including the prefix) and the corresponding record IDs in SELFHOSTDNS_MAP like "subdomainname:RID1:RID2"
  • at least one RID must be set, up to two are supported for wildcard subdomains
  • each entry must be seperated by a space
• export username and password in SELFHOSTDNS_USERNAME and SELFHOSTDNS_PASSWORD

Note: For username you have to use your account / customer number. You can find them in any invoice or on the right top of the selfhost dashboard.

export SELFHOSTDNS_USERNAME="myname"
export SELFHOSTDNS_PASSWORD="mypass"
export SELFHOSTDNS_MAP="_acme-challenge.example.com:12345:98765 alias.example.com:11111"

acme.sh --issue -d example.com --dns dns_selfhost

[AlvinSchiller]

Hey, this works great. Thanks for adding selfhost support!
Can you maybe point out that for the selfhost username you have to use to the accountnumber as shown on the selfhost dashboard. I normally use my email for login and this gets rejected in the API call. Took a bit to figure out ;)

[Marvo2011]

Thanks, I will add it to all docs and comments. I never tried to use the mail :)

[AlvinSchiller]

Hey, today I notices one thing about the RIDs, when testing around.
I created two subdomains for which I issued single certificates (wildcards of subsubdomains included, but i thinks thats not relevant):

1. Domain: test.example.de + *.test.example.de - RID: 11 + RID2: 12
2. Domain test2.example.de + *.test2.example.de - RID: 21 + RID: 22

After the creation I deployed them to my Synology DSM and then wanted to test an automatic renewal of all domains.

The renewal of the first certificate now failed cause the _acme-challenge wasnt successfull. The reason is, that the SELFHOST entries are stored in the account.conf, so the issueing of the second domain with RIDs 21 + 22 overrode the RIDs 11 + 12, so only the renewal of the last domain would be working.

Is this intented?
Must for this case the SELFHOSTDNS_MAP be written with all relevant mappings of all domains? ( which wasn't working in a first test).
Wouldn't it be more straight foreward if the RIDs would be stored within the certificate conf?

[Marvo2011]

Have the Synology a GUI for configuring ACME.sh? I pass the SELFHOSTDNS_RID / SELFHOSTDNS_RID2 exports to every issue / renew. So I don't have any issue with the stored RIDs in the account.conf

SELFHOSTDNS_MAP is thought for multi san like test1.domain.com, test2.domain.com, test3.domain.com (Here is only one RID for every single (sub)domain needed.)

[AlvinSchiller]

No there is no gui for acme in dsm. I only use the functions from acme.sh (running in docker).

Maybe I got something wrong in my test. Just for clarification:
Will the txt record be reused for renewal of a certificate in any way? Or is it just needed for the initial issuing?
Edit: I found the answer: the verification is valid 30 Days, so after that the TXT- Record is needed for renewal.

Cause when it is needed after there is the discribed behavior.
Here with a more detailed step by step description:

Steps from a configuration without any present certificates and exports (if Docker Container restarts, all exports are cleared and only read from confs):

• Add TXT-Records on selfhost _acme-challenge.test1.example.de (11 + 12)

• Set exports: RID=11, RID2=12

• Issue: acme.sh --issue -d test1.example.de --dns dns_selfhost --server letsencrypt

• account.conf: RID=11, RID2=12

• Add TXT-Records on selfhost _acme-challenge.test2.example.de (21 + 22)

• Set exports: RID=21, RID2=22

• Issue: acme.sh --issue -d test2.example.de --dns dns_selfhost --server letsencrypt

• account.conf: RID=21, RID2=22

• renewall: acme.sh --renew-all --force
  -- precondition: dns-01 must be checked again, and is not skipped

-> test1.example.de will be passed RID=21, RID2=22 as it is the last configuration, but these are the IDs of "test2.example.de" so the challenge will fail
-> if I would renew only a single certificate I would need to manually set the rights RIDs again, to make it work ( as you described)
-> this certificate.conf behavior works for example in the deploy_hook synology_dsm. Here the Name(SYNO_Certificate) is configured per certificate and is corretly used for every certificate on renewall

[AlvinSchiller]

Hi, i had problems while using the SELFHOSTDNS_MAP. On issuing a cert (test.example.de + testsub1.example.de) i get the error: "tr: range-endpoints of 'e-c' are in reverse collating sequence order" and the RID is not taken from the MAP.

On debugging the line "rid=$(echo "$SELFHOSTDNS_MAP" | grep -Eoi "$domain:(\d+)" | tr -d "$domain:")" i found some problems:

• The tr-command will fail with the message ""tr: range-endpoints of 'e-c' are in reverse collating sequence order" because the domain given is "_acme-challenge.*". The "-" is taken as range operator. This will also be a problem if the domain contains a dash e.g. "ex-ample.de".
• The replacement with the tr-command cuts digits from the RID if the domain contains digits. e.g. test1.example.de:123 -> 23
• In most cases the fulldomain given is "_acme-challenge.*" which will not match the DNS_MAP entries if there are only the domains are definied
• The grep-command has more then one hit if the subdomains only differ at the beginning, e.g. DNS_MAP="test.example.de:1 mytest.example.de:2 sub.test.example.de;3" , domain=test.example.de -> Result of grep = "test.example.de:1 mytest.example.de:2 sub.test.example.de;3"
• The DEV-Guide states that the commands "grep -o" and "grep -E" should not be used for cross-platform reasons (https://github.com/acmesh-official/acme.sh/wiki/DNS-API-Dev-Guide#12-cross-platform-compatibility-guide)
• The fulldomain can by either of those formats (https://github.com/acmesh-official/acme.sh/wiki/DNS-API-Dev-Guide#3-detect-which-part-is-your-root-zone) so DNS_MAP entries must be defined acording to the chosen mode

I implemented and tested a fix for the usecases i found. Mayby you can check the changes and validate (especially with the plugins).

[Marvo2011]

Thanks. Which distribution do you use?

[AlvinSchiller]

The Docker container from dockerhub / build from Dockerfile (fork) on Linux (Synology DSM)

[Marvo2011]

@AlvinSchiller can you try my pending change on your docker container?

[AlvinSchiller]

Where have you made the changes? I don't see any and got no notification.

[Marvo2011]

@AlvinSchiller I commented directly to the code in your PR.
I have some trouble with sed and regex on some distros.

Can you try the following, in your docker container:

rid=$(echo "$SELFHOSTDNS_MAP" | sed -E "s/(^|^.* )($lookupdomain:|$fulldomain:)([0-9][0-9]*)(.*)/\3/")

[AlvinSchiller]

@Marvo2011 ok very strange that i dont see any comments. Are there settings to the PR that must be activated?
Answer to the change is in the PR

[AlvinSchiller]

@Marvo2011 regarding the pending change: I did some reading and most likely you started a review? In that case you must complete the review so the comments will be published.
"Single comments" should be directly visible.

[Marvo2011]

@AlvinSchiller it's strange I have complete it. Now there are removed. But no problem [[:space:]] works in my tests and I have merged it.

[Marvo2011]

Please report bugs here: acmesh-official#4291

[ralhako]

I have problems use selfHOST and dns_selfhost. DNS record are set correct. dig _acme-challenge.xxxxx show correct content, but acme.sh is in loop. Only cancel is possible.
TTL of DNS-record is 60 sec.
Any idea? Thanks!

[Marvo2011]

@ralhako can you post your log? You can replace your domain with xxx etc...

[Marvo2011]

10/20s are very short I use the original 2min, what does the curl request on cloudflare ?

[Marvo2011]

No problem, I also speak German 😅 I will check the cloudflare request tomorrow. I think it is only a DNS request over a HTTP/Rest API. But for me it was new that the log includes the request by let's encrypt. Do you use also a opnsense ?

[ralhako]

No use/install of opnsense. I use get_acme.sh. Transfer form root to home. Change acme.sh.env to: export LE_WORKING_DIR="/home/.acme.sh/cert" alias acme.sh="/home/.acme.sh/acme.sh". Set acme.sh --set-default-ca --server letsencrypt. And try the first domain - after 5 Error I get: :: too many failed authorizations. Now I use --test with an other domain.

[Marvo2011]

@ralhako Do you move your domain yesterday from United-Domains to selfhost DNS? Such DNS moves can take some time (~24h), please try the acme request again.

[ralhako]

No move from United-Domains, but an update to DNS DYNAMIC to change TTL to 1 minute instead of 1 hour. If 24 hours have passed I will test and report.

[ralhako]

It did not work.
Here my report https://cardinizer.com/public/acme-challenge.ccccccccc_20230603.txt

Try with acme.sh --issue --test --log -d ccccccccc.com -d *.ccccccccc.com --dns --yes-I-know-dns-manual-mode-enough-go-ahead-please will work.

[Marvo2011]

@ralhako ccccccccc.com is your replaced domain?

Seams that Lets Encrypt can't get your actual DNS Records:
[Sa 3. Jun 07:52:34 CEST 2023] ccccccccc.com:Verify error:DNS problem: NXDOMAIN looking up TXT for _acme-challenge.ccccccccc.com - check that a DNS record exists for this domain

Same for the cloudflare-dns option. Cloudflare is used to prevent failed request to the Lets Encrypt API.
https://github.com/acmesh-official/acme.sh/wiki/dnscheck

Can you try to check the DNS with dig at Cloudflare itself?
dig TXT _acme-challenge.ccccccccc.com @1.1.1.1

EDIT: To check it with the real API use:
curl --http2 -H "accept: application/dns-json" "https://1.1.1.1/dns-query?name=_acme-challenge.ccccccccc.com&type=TXT"

[ralhako]

dig and curl is ok, but same error:

https://cardinizer.com/public/acme-challenge.ccccccccc_20230605.txt

[Marvo2011]

@ralhako very strange, which distro and version do you use? Is this a local machine or a vm by selfhost? Do you use any firewall which redirects DNS?

[ralhako]

Hello Marvo2011,
I was only fixated on the cccccccc.com. Now I tried my other domains and they all worked. Then I tried cccccccc.com with --force, same error. I think cccccccc.com is blocked by LetsEncrypt!?
Thanks for your help and your dns_selfhost!!!!!!!!!!

[ralhako]

Is use of a modified version of your dns_selfhost as dns_checkipnet allowed, feasible, approved?: https://checkip.net/getmyowndyndns/

[Marvo2011]

I have no problem, when some one want to use the code from dns_selfhost. It's open source and ok to use it for any one and for any project.