www.heroesonline.com unresolvable via local unbound instance?

I’ve got pi-hole installed locally on my LAN and it forwards it’s uncached requests to a local unbound instance, listening on 127.0.0.1:5353. I recently discovered that the FQDN www.heroesonline.com was unresolvable. After hitting my unbound instance directly, I found it to be the culprit. As near as I can tell, the hosts listed in it’s NS records, simply answer with an empty reply. I don’t know how better to describe it than “empty” but I’m hoping the attached debugging I have sheds some light on it.

If I hit the nameservers for the domain directly, I get what appears to be an empty reply.
If I hit up google dns at 8.8.8.8, I get a valid reply.
If I hit up localhost at 127.0.0.1:5353 I get a timeout, and the unbound logs appear to show unbound retrying after numerous empty replys from the NS servers for that domain.

I’ve:

  1. turned up the verbosity on unbound
  2. requested a valid domain that works (cnn.com), and copied out the log
  3. requested the failing domain, and copied out the log
  4. Ran some host and dig commands to emulate to the best of my ability how this would function. i.e. hit a root server, hit a .com server, hit the servers for heroesonline.com.

My best guess is that the DNS servers for the domain, (ns*.kpmedia.org) are configured to only respond to major ISP DNS servers? Is that a thing these days? I can’t imagine what else a complete lack of an answer would dictate.

Thankfully I just hopped on LTE to get to the website to do what I needed too, I need my local mini-con tickets ;-). But I’m curious why this would be this way. I’m wondering if maybe they’re blocking requests from un-authoritative nameservers or something?

Below you’ll see my dig attempts. I uploaded the listed files to pastebin:

unbound config files:
https://pastebin.com/fRYKKrQB /etc/unbound/unbound.conf
https://pastebin.com/0JMCXeAW /etc/unbound/unbound.conf.d/pi-hole.conf
https://pastebin.com/08gQF4mj /etc/unbound/unbound.conf.d/qname-minimisation.conf
https://pastebin.com/6eNNhcT8 /etc/unbound/unbound.conf.d/root-auto-trust-anchor-file.conf

logfile for a working domain (www.cnn.com)

https://pastebin.com/c5auFtfM unbound.log.cnn.com.txt

logfile for the failing domain (www.heroesonline.com)

https://pastebin.com/FGrDXwEk unbound.log.heroesonline.com.txt

dig directly against google all looks good

root@stretch:~# dig www.heroesonline.com @8.8.8.8

; <<>> DiG 9.10.3-P4-Debian <<>> www.heroesonline.com @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49963
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;www.heroesonline.com. IN A

;; ANSWER SECTION:
www.heroesonline.com. 7 IN A 162.213.254.70

;; Query time: 19 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Tue Apr 30 16:57:12 EDT 2019
;; MSG SIZE rcvd: 65

dig directly against my local unbound instance, times out. I’m assuming that unbound does the same as what my next step does ultimately, hits up nameservers for heroesonline.com to find out the A record for www.heroesonline.com. I’d also assumed that since unbound timed out back to me, that it retries (from what I’m reading in the logs) and eventually just returns me nothing since it gets empty answers from ns*.kpmedia.org

root@stretch:~# dig www.heroesonline.com @127.0.0.1 -p 5353

; <<>> DiG 9.10.3-P4-Debian <<>> www.heroesonline.com @127.0.0.1 -p 5353
;; global options: +cmd
;; connection timed out; no servers could be reached

dig directly against the nameservers for heroesonline.com, gets me an empty answer if I’m reading it right

root@stretch:~# host -t NS heroesonline.com 8.8.8.8
Using domain server:
Name: 8.8.8.8
Address: 8.8.8.8#53
Aliases:

heroesonline.com name server ns21.kpmedia.org.
heroesonline.com name server ns19.kpmedia.org.
heroesonline.com name server ns20.kpmedia.org.
root@stretch:~# host ns21.kpmedia.org. 8.8.8.8
Using domain server:
Name: 8.8.8.8
Address: 8.8.8.8#53
Aliases:

ns21.kpmedia.org has address 37.61.235.107
root@stretch:~# dig www.heroesonline.com @37.61.235.107

; <<>> DiG 9.10.3-P4-Debian <<>> www.heroesonline.com @37.61.235.107
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 8231
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.heroesonline.com. IN A

;; Query time: 104 msec
;; SERVER: 37.61.235.107#53(37.61.235.107)
;; WHEN: Tue Apr 30 16:58:40 EDT 2019
;; MSG SIZE rcvd: 49

Of the three authoritative servers, ns21.kpmedia.org is responding with a REFUSED response.

$ dig www.heroesonline.com @ns21.kpmedia.org

; <<>> DiG 9.10.6 <<>> www.heroesonline.com @ns21.kpmedia.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 506
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.heroesonline.com. IN A

;; Query time: 348 msec
;; SERVER: 37.61.235.107#53(37.61.235.107)
;; WHEN: Wed May 01 20:40:38 PDT 2019
;; MSG SIZE rcvd: 49

So 33% of lookups will fail. It is possible that Google retries different NS servers, if a REFUSED is received. Or perhaps they hit the two working servers, and are just returning the cached result.

Either way, this is not specifically an Unbound issue, as the REFUSED behaviour can be seen anywhere on the Internet.

Tom, This doesn’t explain the behavior of the other two though. To test, I ran dig’s from my workplace. I get the REFUSED as you do to ns21.kpmedia.org from work and home, but from my workplace, dig’s to the other two work just fine. a dig from home times out. See below. I’m wondering if they’re just not answering queries from known cable internet user space…

Home:

[root@topo ~]# host ns19.kpmedia.org. 8.8.8.8 | grep has
ns19.kpmedia.org has address 192.64.118.171
[root@topo ~]# host ns20.kpmedia.org. 8.8.8.8 | grep has
ns20.kpmedia.org has address 162.213.254.70
[root@topo ~]# host ns21.kpmedia.org. 8.8.8.8 | grep has
ns21.kpmedia.org has address 37.61.235.107
[root@topo ~]# dig www.heroesonline.com @192.64.118.171

; <<>> DiG 9.9.4-RedHat-9.9.4-73.el7_6 <<>> www.heroesonline.com @192.64.118.171
;; global options: +cmd
;; connection timed out; no servers could be reached
[root@topo ~]# dig www.heroesonline.com @162.213.254.70

; <<>> DiG 9.9.4-RedHat-9.9.4-73.el7_6 <<>> www.heroesonline.com @162.213.254.70
;; global options: +cmd
;; connection timed out; no servers could be reached
[root@topo ~]# dig www.heroesonline.com @37.61.235.107

; <<>> DiG 9.9.4-RedHat-9.9.4-73.el7_6 <<>> www.heroesonline.com @37.61.235.107
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 23571
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.heroesonline.com. IN A

;; Query time: 106 msec
;; SERVER: 37.61.235.107#53(37.61.235.107)
;; WHEN: Thu May 02 07:31:52 EDT 2019
;; MSG SIZE rcvd: 49

Work:

[blaket@topo2 ~]$ host www.heroesonline.com
www.heroesonline.com has address 162.213.254.70
[blaket@topo2 ~]$ host -t NS heroesonline.com
heroesonline.com name server ns21.kpmedia.org.
heroesonline.com name server ns20.kpmedia.org.
heroesonline.com name server ns19.kpmedia.org.
[blaket@topo2 ~]$ dig www.heroesonline.com @ns19.kpmedia.org.

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.68.rc1.el6_10.1 <<>> www.heroesonline.com @ns19.kpmedia.org.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40206
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;www.heroesonline.com. IN A

;; ANSWER SECTION:
www.heroesonline.com. 3600 IN A 162.213.254.70

;; AUTHORITY SECTION:
heroesonline.com. 86400 IN NS ns19.kpmedia.org.
heroesonline.com. 86400 IN NS ns20.kpmedia.org.
heroesonline.com. 86400 IN NS ns21.kpmedia.org.

;; Query time: 28 msec
;; SERVER: 192.64.118.171#53(192.64.118.171)
;; WHEN: Thu May 2 11:28:01 2019
;; MSG SIZE rcvd: 122

[blaket@topo2 ~]$ dig www.heroesonline.com @ns20.kpmedia.org.

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.68.rc1.el6_10.1 <<>> www.heroesonline.com @ns20.kpmedia.org.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54332
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;www.heroesonline.com. IN A

;; ANSWER SECTION:
www.heroesonline.com. 3600 IN A 162.213.254.70

;; AUTHORITY SECTION:
heroesonline.com. 86400 IN NS ns21.kpmedia.org.
heroesonline.com. 86400 IN NS ns19.kpmedia.org.
heroesonline.com. 86400 IN NS ns20.kpmedia.org.

;; Query time: 18 msec
;; SERVER: 162.213.254.70#53(162.213.254.70)
;; WHEN: Thu May 2 11:28:06 2019
;; MSG SIZE rcvd: 122

[blaket@topo2 ~]$ dig www.heroesonline.com @ns21.kpmedia.org.

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.68.rc1.el6_10.1 <<>> www.heroesonline.com @ns21.kpmedia.org.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 18346
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;www.heroesonline.com. IN A

;; Query time: 140 msec
;; SERVER: 37.61.235.107#53(37.61.235.107)
;; WHEN: Thu May 2 11:28:09 2019
;; MSG SIZE rcvd: 38

[blaket@topo2 ~]$

It is fairly common for ISPs to block all udp port 53 across their network, and only permit udp port 53 to their own DNS servers. That is only two ACL rules, so it is very simple to implement. I would say that in general, port 53 blocking is something that happens less and less.

Ask your ISP for their list of blocked ports. Nearly all residential ISPs have a list of block ports. It might even be published in their support portal.

You are totally off about the DNS servers not “… answering queries from known cable internet user space.” I’ve never heard of that. It is also would terribly difficult to implement. Nor can I imagine what benefit it would provide.

Tom

Hi Tom,

  It is fairly common for ISPs to block all udp port 53 across their network, and only permit udp port 53 to their own DNS servers. That is only two ACL rules, so it is very simple to implement. I would say that in general, port 53 blocking is something that happens less and less.

That would spell "support apocalypse" in any residential ISP I've ever
used, and a shortcut to "we can't make payroll" via "all the customers
have gone". I have never seen it outside hotel/retail guest networks.

Do you have any measurements to support "fairly common"? If that's
right and my experience is atypical it's the kind of thing I'd like to
understand.

Joe

My ISP is Spectrum (Charter) in the US, from what I can tell, they don’t seem to block much of anything, if anything, at all.

The “fairly common” comment was just a guess on my part as I’ve never had to troubleshoot DNS like this. I see no logical reason why I shouldn’t be able to contact their DNS servers. I was just at a loss as to what the issue might be.

Though I also noticed, that if I set my DNS to 8.8.8.8 on my desktop PC, and can thusly resolve the domain, I still can’t reach the website. This smells less like intentional blocking, and more like maybe routing, or some other misconfiguration at the host for the domain.

This seems like a block form the auth servers, not from ISP.

Maybe someone using your ISP did some attack and got you also in a /24 drop or even /16.

I agree with Joe. ISPs that want all DNS traffic simply route all port 53 traffic to their own nameserver. Which only now end users are getting protection from with DoT and DoH