Hi,
No response to this post as yet?
Any help appreciated.
RayG
Hi RayG,
You are correct that the file should be written by unbound. Are you sure that unbound has write permissions in that directory?
You could also use IP addresses for XFRs and they will be probed for the SOA value and also tried if the url does not work.
However, I don't think that they offer the service over XFR. At least they only advertise the url on their website.
Best regards,
-- George
Hi RayG,
On verbosity >= 4 you could see the following entries that relate to rpz (from my own run where download and file creation succeed):
debug: auth zone rpz.urlhaus.abuse.ch. transfer next HTTP fetch from
debug: http download downloads/rpz of size
info: auth zone http downloaded content preview:
debug: auth zone rpz.urlhaus.abuse.ch. updated to serial
debug: write zonefile file.name for rpz.urlhaus.abuse.ch.
local-zone answers are before the rpz zones, so you will not see entries in the log file for those.
Best regards,
-- George
OK so with log level set at 4 I don’t see in the log file lines like you list below
The log file is very large so I cannot attach it to this email even as pasted text.
Here is a link:
https://1drv.ms/u/s!As73rPtzISrU4mTvPONvZCWVSCWD?e=MJ18Lx
A couple of points.
1). Do I have to create the zone file before starting unbound?
2). Do I need to populate the file with anything?
I have tried all ways with no success.
Thanks
RayG
Hi RayG,
You don't have to create the file before starting unbound. If the file is there unbound will try to parse it.
You don't have to manually populate the file with anything unless the rpz source is only a file. Not for your case though.
I see in your log:
...
10/11/2020 15:00:14 C:\Program Files\Unbound\unbound.exe[15932:0] debug: read zonefile C:\ProgramData\Unbound\Logs\rpz.urlhaus.abuse.ch for rpz.urlhaus.abuse.ch.
...
10/11/2020 15:05:24 C:\Program Files\Unbound\unbound.exe[15932:0] debug: auth zone rpz.urlhaus.abuse.ch. transfer failed, wait
...
The first one shows unbound reading from the file it created from a previous run probably.
The second one shows that unbound could not complete the transfer and will try later.
Best regards,
-- George
Hi George,
OK thanks for that info.
I have no issue getting the data using the URL in a browser but unbound flatly refuses to successfully retrieve it.
name: "URLHaus"
zonefile: "C:\ProgramData\Unbound\Logs\urlhaus.zone"
url: https://urlhaus.abuse.ch/downloads/rpz
rpz-log: yes
rpz-log-name: "URLHausRPZ"
rpz-action-override: nxdomain
I have tried with an empty file, a populated file and no file but nothing works. No file is ever created.
What else can I try?
This is the configuration file I have removed my local network configuration:
RayG,
You can try stop unbound and run it in foreground:
unbound -d -vvvvv
And look for some errors.
Hi Eduardo,
Thanks for the suggestion, that is certainly an easier way to get the debugging output.
Looking through the logs and in greater detail I wonder if I have seen the issue.
See these two commands:
C:\Program Files\Unbound>I:\wget64.exe https://151.101.130.49/downloads/rpz
--2020-11-11 16:01:48-- https://151.101.130.49/downloads/rpz
Connecting to 151.101.130.49:443... connected.
ERROR: certificate common name 'c.sni.fastly.net' doesn't match requested host name '151.101.130.49'.
To connect to 151.101.130.49 insecurely, use `--no-check-certificate'.
C:\Program Files\Unbound>I:\wget64.exe https://urlhaus.abuse.ch/downloads/rpz
--2020-11-11 16:02:54-- https://urlhaus.abuse.ch/downloads/rpz
Resolving urlhaus.abuse.ch (urlhaus.abuse.ch)... 151.101.66.49, 151.101.2.49, 151.101.194.49, ...
Connecting to urlhaus.abuse.ch (urlhaus.abuse.ch)|151.101.66.49|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 130762 (128K) [text/plain]
Saving to: 'rpz'
rpz 100%[=================================================>] 127.70K --.-KB/s in 0.04s
2020-11-11 16:02:54 (3.15 MB/s) - 'rpz' saved [130762/130762]
And the data is there in the "rpz" file.
I see in the unbound log file:
10/11/2020 15:05:14 C:\Program Files\Unbound\unbound.exe[15932:0] debug: auth zone rpz.urlhaus.abuse.ch. transfer next HTTP fetch from 151.101.122.49 started
...
10/11/2020 15:05:24 C:\Program Files\Unbound\unbound.exe[15932:0] debug: xfr stopped, connection timeout to urlhaus.abuse.ch
...
10/11/2020 15:05:24 C:\Program Files\Unbound\unbound.exe[15932:0] debug: auth zone rpz.urlhaus.abuse.ch. transfer failed, wait
Which suggests the transfer is being done using the IP address rather than the DNS name and as we can see from above with wget we get a certificate error.
Is this what is causing things to go wrong?
Is unbound using the DNS name or the IP address?
RayG
Hi George,
I have tried many things over the weekend but despite everything I cannot see why this is not working.
All I see in the log is "Transfer failed"
rpz: # MyResponsePolicyZones.conf
name: "URLHaus"
zonefile: "C:\ProgramData\Unbound\Logs\urlhaus.zone"
url: https://urlhaus.abuse.ch/downloads/rpz/
rpz-log: yes
rpz-log-name: "URLHausRPZ"
rpz-action-override: nxdomain
There is no indication of why it failed or if indeed it did download the data and failed while processing it.
Until I can get a handle on what is going on I cannot see a way to resolve the situation.
I got the impression that the configuration worked for you but why not for me?
RayG
Hi RayG,
Indeed it works for me on linux.
It *seems* like an issue on windows.
It would be great if other windows users could try and verify the same behavior before I start debugging there.
To clarify this is not about writing the file but succeeding with the transfer to begin with.
-- George
Hi George,
I have captured a network trace for all communications to URAHaus following a restart of Unbound:
https://1drv.ms/t/s!As73rPtzISrU6R4U71vH0s5rCTT5?e=RzjhtF (3MB approx.)
At this time this is a straight text file but I can save it in a different network sniffer format to make searching/decoding easier.
If you let me know what format I will capture and save as required.
The trace is filtered on all communications with these 4 IP addresses (151.*)
urlhaus.abuse.ch. 1179 IN CNAME p2.shared.global.fastly.net.
p2.shared.global.fastly.net. 29 IN A 151.101.194.49
p2.shared.global.fastly.net. 29 IN A 151.101.2.49
p2.shared.global.fastly.net. 29 IN A 151.101.66.49
p2.shared.global.fastly.net. 29 IN A 151.101.130.49
RayG
Hi RayG,
Thanks for this. From a first quick look it seems that the TLS handshake is not completed and I suspect a timeout is thrown at the end. At least this is what I expect the "Encrypted Alert" to be; it seems to trigger after 10 seconds exactly.
Will have to look closer on what is happening though.
-- George
Hi George,
I forgot to mention that I do run unbound under the "Network Service" Account which I successfully configured way back some time. The solutions were in the thread around: 10/06/2016 07:55:00. Since then Unbound has run this way without issue relating to the Network Service Account.
FYI I am using Windows 10/64 (B19042.630-V20H2)
RayG
Hello,
I like to attach myself to this longer thread...
I have no issue getting the data using the URL in a browser but unbound flatly refuses to successfully retrieve it.
name: "URLHaus"
zonefile: "C:\ProgramData\Unbound\Logs\urlhaus.zone"
url: https://urlhaus.abuse.ch/downloads/rpz
rpz-log: yes
rpz-log-name: "URLHausRPZ"
rpz-action-override: nxdomainI have tried with an empty file, a populated file and no file but nothing works. No file is ever created.
My idea was to use RPZ and also start with urlhaus available as https://urlhaus.abuse.ch/downloads/rpz
Running 1.13.1 this failed partially. So here are my findings.
To be as reproducable as possible, I describe a docker setup and show some unnessesary exlicit options:
$ install -d /tmp/rpz/ && cd /tmp/rpz/
$ cat <<EOF > /tmp/rpz/unbound.conf
server:
chroot: ""
do-daemonize: no
do-ip6: no
logfile: ""
log-replies: yes
module-config: "respip validator iterator"
tls-cert-bundle: /usr/share/ca-certificates/mozilla/GlobalSign_Root_CA_-_R3.crt
rpz:
name: rpz.urlhaus.abuse.ch.
zonefile: /tmp/spool/rpz.urlhaus.abuse.ch
url: https://urlhaus.abuse.ch/downloads/rpz/
rpz-log: yes
EOF
$ docker run --name unbound --rm -ti -v /tmp/rpz:/tmp/rpz:rw debian:testing-slim
root@f94664d2bfc7:/# apt-get update
...
root@f94664d2bfc7:/# apt-get -qq --no-install-recommends install ca-certificates knot-dnsutils unbound wget
...
root@f94664d2bfc7:/# install -d --owner unbound /tmp/spool/
root@f94664d2bfc7:/# unbound -c /tmp/rpz/unbound.conf
[1619098696] unbound[2749:0] notice: init module 0: respip
[1619098696] unbound[2749:0] notice: init module 1: validator
[1619098696] unbound[2749:0] notice: init module 2: iterator
[1619098696] unbound[2749:0] info: start of service (unbound 1.13.1).
According to unbound documentation I expext unbound to start downloading
https://urlhaus.abuse.ch/downloads/rpz/ to /tmp/spool/rpz.urlhaus.abuse.ch
The directory /tmp/spool/ *is* writable for the unbound user.
That download does not happen.
Consequently dns queries are answered like no rpz is present at all.
Let's login from an other shell into the unbound container and ask for the test entry:
$ docker exec -ti unbound bash
root@f94664d2bfc7:/# kdig @127.0.0.1 testentry.rpz.urlhaus.abuse.ch.
;; ->>HEADER<<- opcode: QUERY; status: NXDOMAIN; id: 16488
;; Flags: qr rd ra; QUERY: 1; ANSWER: 0; AUTHORITY: 1; ADDITIONAL: 0
;; QUESTION SECTION:
;; testentry.rpz.urlhaus.abuse.ch. IN A
;; AUTHORITY SECTION:
abuse.ch. 1800 IN SOA ns1.p04.dynect.net. dnsadmin.abuse.ch. 2021042100 3600 600 604800 1800
;; Received 111 B
;; Time 2021-04-22 14:00:35 UTC
;; From 127.0.0.1@53(UDP) in 21.7 ms
the other terminal shows the log:
...
[1619098696] unbound[2749:0] info: start of service (unbound 1.13.1).
[1619100035] unbound[2874:0] info: 127.0.0.1 testentry.rpz.urlhaus.abuse.ch. A IN NXDOMAIN 0.021310 0 111
This is unexpected because there is an AUTHORITY SECTION
Now let's download https://urlhaus.abuse.ch/downloads/rpz/ manually.
We use 'wget' and explicit set ca-directory to an empty /opt/ and ca-certificate the the one matching CA so only this one download is 'permitted'
This is only to demonstrate there are *no* TLS certificate validatation issues.
Also, we run 'wget' as user unbound. This makes the resulting file owned an thus writable by unbound.
press CTRL+C in the unbound terminal
^C[1619099541] unbound[2856:0] info: service stopped (unbound 1.13.1).
[1619099541] unbound[2856:0] info: server stats for thread 0: 1 queries, 0 answers from cache, 1 recursions, 0 prefetch, 0 rejected by ip ratelimiting
[1619099541] unbound[2856:0] info: server stats for thread 0: requestlist max 0 avg 0 exceeded 0 jostled 0
[1619099541] unbound[2856:0] info: average recursion processing time 0.031167 sec
[1619099541] unbound[2856:0] info: histogram of recursion processing times
[1619099541] unbound[2856:0] info: [25%]=0 median[50%]=0 [75%]=0
[1619099541] unbound[2856:0] info: lower(secs) upper(secs) recursions
[1619099541] unbound[2856:0] info: 0.016384 0.032768 1
root@f94664d2bfc7:/# su --shell /bin/sh --command 'wget --no-verbose -O /tmp/spool/rpz.urlhaus.abuse.ch --ca-directory=/opt --ca-certificate=/usr/share/ca-certificates/mozilla/GlobalSign_Root_CA_-_R3.crt https://urlhaus.abuse.ch/downloads/rpz/’ unbound
2021-04-22 14:29:19 URL:https://urlhaus.abuse.ch/downloads/rpz/ [163814/163814] -> "/tmp/spool/rpz.urlhaus.abuse.ch" [1]
root@f94664d2bfc7:/# stat /tmp/spool/rpz.urlhaus.abuse.ch
File: /tmp/spool/rpz.urlhaus.abuse.ch
Size: 163814 Blocks: 320 IO Block: 4096 regular file
Device: 34h/52d Inode: 123433959 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 101/ unbound) Gid: ( 101/ unbound)
Access: 2021-04-22 14:29:19.000000000 +0000
Modify: 2021-04-22 14:25:03.000000000 +0000
Change: 2021-04-22 14:29:19.891804798 +0000
Birth: 2021-04-22 14:29:19.619804800 +0000
again, start unbound:
root@f94664d2bfc7:/# unbound -c /tmp/rpz/unbound.conf
[1619099932] unbound[2866:0] notice: init module 0: respip
[1619099932] unbound[2866:0] notice: init module 1: validator
[1619099932] unbound[2866:0] notice: init module 2: iterator
[1619099932] unbound[2866:0] info: start of service (unbound 1.13.1).
in the other terminal now ask the *same* question:
root@f94664d2bfc7:/# kdig @127.0.0.1 testentry.rpz.urlhaus.abuse.ch. ns
;; ->>HEADER<<- opcode: QUERY; status: NXDOMAIN; id: 29041
;; Flags: qr aa rd ra; QUERY: 1; ANSWER: 0; AUTHORITY: 0; ADDITIONAL: 0
;; QUESTION SECTION:
;; testentry.rpz.urlhaus.abuse.ch. IN NS
;; Received 48 B
;; Time 2021-04-22 14:04:32 UTC
;; From 127.0.0.1@53(UDP) in 0.2 ms
and unbound logs this:
[1619100269] unbound[2880:0] info: start of service (unbound 1.13.1).
[1619100272] unbound[2880:0] info: RPZ applied testentry.rpz.urlhaus.abuse.ch. nxdomain 127.0.0.1@39411 testentry.rpz.urlhaus.abuse.ch. NS IN
[1619100272] unbound[2880:0] info: 127.0.0.1 testentry.rpz.urlhaus.abuse.ch. NS IN NXDOMAIN 0.000000 1 48
so we see, the same question is now handled by the rpz and is answered (in this case also) with NXDOMAIN
But this happen only if I manually provide the rpz data file.
I also setup a second nginx container that can serve http://local-nginx/rpz.urlhaus.abuse.ch to unbound
This eliminates https and allow running tcpdump on the local docker network: There is no communication from unbound to the local-nginx
But the documentation (https://github.com/NLnetLabs/unbound/blob/release-1.13.1/doc/unbound.conf.5.in#L2404)
promise somehow "If the file does not exist or is empty, unbound will attempt to fetch zone data (eg. from the primary servers)"
Conclusion: rpz with URL works but bootstrapping is unsure.
I've also not yet monitored, if and when updates happen.
Andreas
Hi Andreas,
I can't reproduce what you are experiencing. That configuration with URL zone transferring is working fine on my machine (linux). And it works with or without TLS in place.
Although, I didn't use docker so I can't comment if something doesn't work there specifically.
What was described previously on the list is an issue with fetching the zone file via URL on windows specifically. I tracked this down to HTTPS specifically but didn't have time to look further on that yet.
Some points that may help you further:
1.
You can skip the 'zonefile:' directive. That way you force unbound to not look at a possible file when starting and instead fetch the zone from the url.
If a zonefile is already present at the configured location unbound will use it without going out to the network.
Zone updates then happen based on the SOA.Refresh timer of the zone.
2.
You could use 'unbound-control auth_zone_transfer <auth_zone>' and see what happens. Unbound should try to refetch the zone data from the URL and hopefully print out some errors in your case.
Best regards,
-- George
Hello George,
the problem here looks similar, fetching via HTTPS fail.
I login into a running container. "unbound-control verbosity 5" and "unbound-control auth_zone_transfer rpz.urlhaus.abuse.ch." start the relevant things...
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: auth zone rpz.urlhaus.abuse.ch. transfer next HTTP fetch from 151.101.14.49 started
unbound_1 | Apr 28 20:14:10 unbound[1:0] info: mesh_run: end 0 recursion states (0 with reply, 0 detached), 0 waiting replies, 0 recursion replies sent, 0 replies dropped, 0 states jostled out
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: cache memory msg=105467 rrset=106995 infra=39527 val=72984 subnet=0
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: svcd callbacks end
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: serviced_delete
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: close of port 28986
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: comm_point_close of 20: event_del
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: close fd 20
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: comm point listen_for_rw 21 0
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: peer certificate:
unbound_1 | Issuer: C=BE, O=GlobalSign nv-sa, CN=GlobalSign Atlas R3 DV TLS CA 2020
unbound_1 | Validity
unbound_1 | Not Before: Mar 23 15:34:33 2021 GMT
unbound_1 | Not After : Apr 24 15:34:32 2022 GMT
unbound_1 | Subject: CN=*.abuse.ch
unbound_1 | X509v3 extensions:
unbound_1 | X509v3 Subject Alternative Name:
unbound_1 | DNS:*.abuse.ch
unbound_1 | X509v3 Key Usage: critical
unbound_1 | Digital Signature, Key Encipherment
unbound_1 | X509v3 Extended Key Usage:
unbound_1 | TLS Web Server Authentication, TLS Web Client Authentication
unbound_1 | X509v3 Subject Key Identifier:
unbound_1 | E3:9E:17:C0:43:1E:FB:CF:43:B6:CB:B5:FF:C9:DE:AF:08:81:3A:49
unbound_1 | X509v3 Certificate Policies:
unbound_1 | Policy: 1.3.6.1.4.1.4146.1.10
unbound_1 | CPS: Repository | GlobalSign
unbound_1 | Policy: 2.23.140.1.2.1
unbound_1 |
unbound_1 | X509v3 Basic Constraints:
unbound_1 | CA:FALSE
unbound_1 | Authority Information Access:
unbound_1 | OCSP - URI:http://ocsp.globalsign.com/ca/gsatlasr3dvtlsca2020
unbound_1 | CA Issuers - URI:http://secure.globalsign.com/cacert/gsatlasr3dvtlsca2020.crt
unbound_1 |
unbound_1 | X509v3 Authority Key Identifier:
unbound_1 | keyid:42:6D:57:2D:4F:1F:26:77:74:A6:27:64:F6:80:FA:8F:48:68:FE:7C
unbound_1 |
unbound_1 | X509v3 CRL Distribution Points:
unbound_1 |
unbound_1 | Full Name:
unbound_1 | URI:http://crl.globalsign.com/ca/gsatlasr3dvtlsca2020.crl
unbound_1 |
unbound_1 | CT Precertificate SCTs:
unbound_1 | Signed Certificate Timestamp:
unbound_1 | Version : v1 (0x0)
unbound_1 | Log ID : 46:A5:55:EB:75:FA:91:20:30:B5:A2:89:69:F4:F3:7D:
unbound_1 | 11:2C:41:74:BE:FD:49:B8:85:AB:F2:FC:70:FE:6D:47
unbound_1 | Timestamp : Mar 23 15:34:33.859 2021 GMT
unbound_1 | Extensions: none
unbound_1 | Signature : ecdsa-with-SHA256
unbound_1 | 30:45:02:21:00:99:20:2E:E7:63:02:8B:EE:BB:C7:07:
unbound_1 | 84:FE:70:AF:BA:CC:77:E8:AD:CA:B2:9A:82:60:85:E6:
unbound_1 | C6:7D:45:68:13:02:20:2E:3B:FA:16:3D:1C:8A:87:51:
unbound_1 | 9B:BA:45:58:36:0D:38:D1:8E:F4:D2:22:80:8A:24:F6:
unbound_1 | 3B:18:B5:64:E9:85:87
unbound_1 | Signed Certificate Timestamp:
unbound_1 | Version : v1 (0x0)
unbound_1 | Log ID : 6F:53:76:AC:31:F0:31:19:D8:99:00:A4:51:15:FF:77:
unbound_1 | 15:1C:11:D9:02:C1:00:29:06:8D:B2:08:9A:37:D9:13
unbound_1 | Timestamp : Mar 23 15:34:34.022 2021 GMT
unbound_1 | Extensions: none
unbound_1 | Signature : ecdsa-with-SHA256
unbound_1 | 30:46:02:21:00:97:1F:A1:2A:0E:08:0C:2D:6F:14:3A:
unbound_1 | 30:50:C6:C4:37:7E:55:8A:B1:83:9B:E3:7F:8E:EA:41:
unbound_1 | 53:CF:88:E4:19:02:21:00:E6:9D:17:2E:CE:A0:93:C8:
unbound_1 | 54:04:61:2C:AC:56:B7:6E:CE:DA:FB:73:34:F4:EE:5D:
unbound_1 | 76:EE:9B:A1:E6:25:D0:CF
unbound_1 | Signed Certificate Timestamp:
unbound_1 | Version : v1 (0x0)
unbound_1 | Log ID : 55:81:D4:C2:16:90:36:01:4A:EA:0B:9B:57:3C:53:F0:
unbound_1 | C0:E4:38:78:70:25:08:17:2F:A3:AA:1D:07:13:D3:0C
unbound_1 | Timestamp : Mar 23 15:34:34.059 2021 GMT
unbound_1 | Extensions: none
unbound_1 | Signature : ecdsa-with-SHA256
unbound_1 | 30:45:02:21:00:D5:AC:D7:20:A0:D4:7A:A3:9D:3A:7A:
unbound_1 | A7:67:06:D6:19:A8:DE:B4:E8:BC:E7:00:C0:4F:76:B9:
unbound_1 | C8:42:C0:16:81:02:20:62:91:72:CA:FB:B5:51:15:4E:
unbound_1 | 94:8E:1D:3A:98:2A:2C:30:AF:60:FA:0A:D4:BC:0B:E0:
unbound_1 | 72:3A:F1:00:D6:20:28
unbound_1 |
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: SSL connection to *.abuse.ch authenticated ip4 151.101.14.49 port 443 (len 16)
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: comm point listen_for_rw 21 1
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: comm point stop listening 21
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: comm point start listening 21 (-1 msec)
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: startlistening 21 mode r
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: HTTP/1.1 200 OK
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: Connection: keep-alive
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: Content-Length: 138924
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: Server: Apache
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: Strict-Transport-Security: max-age=15768000 ; includeSubDomains
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: Expect-CT: enforce, max-age=86400
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: Permissions-Policy: accelerometer=(), ambient-light-sensor=(), autoplay=(), camera=(), encrypted-media=(), fullscreen=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), midi=(), payment=(), picture-in-picture=(), speaker=(), usb=(), vr=()
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: Referrer-Policy: strict-origin-when-cross-origin
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: Content-Security-Policy: default-src 'self' https://fonts.gstatic.com:443 data:; style-src 'self' 'unsafe-inline' https://www.gstatic.com:443 https://fonts.googleapis.com:443; script-src 'self' 'unsafe-inline' 'unsafe-eval' https://www.gstatic.com:443 https://www.google.com/recaptcha/; frame-src https://www.google.com/recaptcha/; img-src 'self' data: https://syndication.twitter.com:443; object-src 'none'
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: Cross-Origin-Opener-Policy: same-origin; report-to="default"
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: Cross-Origin-Resource-Policy: same-site
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: Last-Modified: Wed, 28 Apr 2021 18:10:05 GMT
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: ETag: "21eac-5c10c49cbaab3"
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: Cache-Control: max-age=300
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: Expires: Wed, 28 Apr 2021 18:16:51 GMT
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: X-Content-Type-Options: nosniff
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: X-Frame-Options: sameorigin
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: X-XSS-Protection: 1; mode=block
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: Content-Type: text/plain
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: Via: 1.1 varnish, 1.1 varnish
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: Accept-Ranges: bytes
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: Date: Wed, 28 Apr 2021 18:14:10 GMT
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: Age: 138
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: X-Served-By: cache-lhr7352-LHR, cache-fra19165-FRA
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: X-Cache: HIT, HIT
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: X-Cache-Hits: 1, 1
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: X-Timer: S1619633650.192137,VS0,VE1
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header: Vary: Accept-Encoding
unbound_1 | Apr 28 20:14:10 unbound[1:0] debug: http header:
unbound_1 | Apr 28 20:14:20 unbound[1:0] debug: xfr stopped, connection timeout to urlhaus.abuse.ch
unbound_1 | Apr 28 20:14:20 unbound[1:0] debug: comm_point_close of 21: event_del
unbound_1 | Apr 28 20:14:20 unbound[1:0] debug: close fd 21
unbound_1 | Apr 28 20:14:20 unbound[1:0] debug: auth zone rpz.urlhaus.abuse.ch. transfer failed, wait
So just after 10s the transfer time out. Attached a trace. It show some data are arriving but then the connection somehow get out of state/sync and errors happen.
This don't happen when I run "wget" as separate process parallel to unbound but in the same container.
Andreas
(attachments)
trace (170 KB)
Hi Andreas,
Thanks for the extra information!
The windows issue does not establish the HTTPS handshake IIRC, so no further data flowing there.
A couple more questions:
- Does this also happen without HTTPS? You mentioned an nginx serving non-HTTPS content. Could you retry with auth_zone_transfer?
- Do you see that behavior also without docker?
Best regards,
-- George
The windows issue does not establish the HTTPS handshake IIRC, so no further data flowing there.
ok, two different things ...
- Does this also happen without HTTPS? You mentioned an nginx serving non-HTTPS content. Could you retry with auth_zone_transfer?
I copied https://urlhaus.abuse.ch/downloads/rpz/ to a local webserver reachable by http as well as https
with "url: https://andreasschulze.de/testing/" the transfer failed in the same manner as from https://urlhaus.abuse.ch/downloads/rpz/
with "url: http://andreasschulze.de/testing/" the transfer succeeded immediately.
- Do you see that behavior also without docker?
yes, the behavior describded above is the same for unbound-1.13.1 inside docker-ce (the Debian Bullseye unbound binary/package)
as well as unbound-1.13.1 (selfcompiled, on Debian Buster) without the Docker Layer.
HTTP works, HTTPS don't.
Q: did you consider using libcurl for downloads?
optional; if unbound is build with libcurl, "url" options are available otherwise AXFR must be used
Andreas
Hi Andreas,
Thanks for the detailed info!
Non-working HTTPS url on linux is news to me, I'll look into it.
Not sure if it will help but which version of openssl do you use? unbound -V should have that information.
For the libcurl question: I wasn't part of the development but I guess pulling that big of a library into unbound is not an easy task, especially when libcurl comes with so much functionality that unbound will never need.
Best regards,
-- George
yes, it's not a simple task and I'm aware this may introduce other problems. Was just any idea. OpenDKIM for example, choosed to include libcurl instead of writing dedicated SMTP client. https://github.com/trusteddomainproject/OpenDKIM/blob/master/RELEASE_NOTES#L253
Oh, and this was also initiated by myself 
Andreas