DNS poisoning - any ideas how this can happen?

Unbound
1

Hi all,

We’ve run into a dns poisoning issue in our company network since Friday. The issue is being discussed here: https://forum.pfsense.org/index.php?topic=87491.0 - we use Unbound on a pfSense. A few other users have the same problem:

  • All of a sudden, all host names resolve to a malware host.
  • It stops automatically after some time
  • There’s no arp poisoning going on, so it really comes from Unbound on the pfSense

Example:

While “on”:

$ host omx.ch
omx.ch has address 195.22.26.248
omx.ch mail is handled by 10 mx1.csof.net.
omx.ch mail is handled by 10 mx2.csof.net.

Normally:

$host omx.ch
omx.ch has address 62.48.3.132
omx.ch mail is handled by 10 mxhost1.omx.ch

Other wrongly resolved ips lead to sso.mlwr.io (which tries to redirect back to xsso.<correcthost.com>/)

Any ideas?

  • Martin
2

Hi Martin,

Hi all,

We've run into a dns poisoning issue in our company network since
Friday. The issue is being discussed here:
https://forum.pfsense.org/index.php?topic=87491.0 - we use Unbound
on a pfSense. A few other users have the same problem:

- All of a sudden, all host names resolve to a malware host. - It
stops automatically after some time - There's no arp poisoning
going on, so it really comes from Unbound on the pfSense

So, unbound comes with a set of commands for unbound-control that
allow you to monitor the runtime settings, and these are exactly meant
to be able to audit the settings in the runtime daemon and if they are
still correct. unbound-control list_forwards.

You could try to use packet capture of traffic going to 8.8.8.8 and
the responses. Or you can get unbound to log verbosely (high
verbosity setting), although much slower at level 4 it'll print a
dig-like output for packets received from upstream, so you can see
where the malicious data comes from.

Or just dig @8.8.8.8 from the commandline, that has the same routing
as the pfSense firewall box with unbound on it, and look at the result.

Unbound has DNSSEC capabilities that are meant to protect against
these sorts of things (only for DNSSEC signed domains of course). You
can easily turn it on with unbound-anchor -a /etc/root.key and putting
auto-trust-anchor-file: "/etc/root.key" in unbound.conf.

Best regards,
   Wouter

3

Hi Martin,

Looking through that forum thread, I see that unbound's cache contains
bad items and that the NS records for com, net org edu are point to
nsX.csof.net. And I wonder if unbound is getting cache poisoned or if
your 'upstream resolver' or upstream captive-resolver (i.e. the
8.8.8.8 has been hijacked by your ISP and is serviced by other
software) is getting cache poisoned.

The unbound logs should tell you if you enable verbose logging (I
would recommend level 4, or perhaps level 5 so you can see who
requests those bad domain names in your network). Or when unbound is
misbehaving dig @8.8.8.8 from a box with similar routing, and see if
those responses have been cache poisoned.

When I try to resolve api-nyc01.exip.org here I see unbound complain
that it has to remove potential poisonous DNS RRsets from the answers.
It does so and is not poisoned:

reply from <exip.org.> 54.77.72.254#53
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 26841
;; flags: qr aa ; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 4
;; QUESTION SECTION:
;; api-nyc01.exip.org. IN A

;; ANSWER SECTION:
api-nyc01.exip.org. 10 IN A 195.22.26.248

;; AUTHORITY SECTION:
org. 172800 IN NS ns1.csof.net.
org. 172800 IN NS ns2.csof.net.
org. 172800 IN NS ns3.csof.net.
org. 172800 IN NS ns4.csof.net.

;; ADDITIONAL SECTION:
ns1.csof.net. 100 IN A 54.77.72.254
ns2.csof.net. 100 IN A 212.6.183.201
ns3.csof.net. 100 IN A 195.22.26.199
ns4.csof.net. 100 IN A 54.72.8.183

Humourously (if you think this is funny), it could be a
misconfiguration and bad DNS ISP practices interfering here. Some
domain hosters serve .org and .com zones and they have *.com and *.org
wildcards on those servers. Thus they do not reconfigure their DNS
servers when they buy or sell a domain. But that produces these types
of 'poisonous' answers. Badly written DNS software could then get DNS
cache poisoned as a result. Unbound should not get DNS cache
poisoned, there is explicit fixup code for this.

Because setting different upstream IP forwarders changes the outcome,
I think it may be the upstream DNS server that gets cache poisoned
(after a lookup of one of these affected domains), and some ISPs
interpose all DNS traffic with their own DNS servers, that may be
getting poisoned or maybe only traffic to 'popular' DNS sites. I
doubt google DNS itself is getting cache poisoned, but it is a
technical possibility.

Best regards,
   Wouter

4

Hi,

After off-list conversation (with conf and logs), the solution is
harden-glue: yes in unbound.conf. The default is yes, but in pfSense
it was turned off.

There is a code fix that still protects NS records if config has
harden-glue no.

Best regards,
   Wouter

5

Ouch, that seems like a sub-optimal configuration.

Was this in the unbound package available for pfSense 2.1 and earlier, or the native unbound implementation in pfSense 2.2? Did you log any bugs on the pfSense side or does this still need to be done?

6

I had a chance to play with pfSense 2.2 a bit, and this is a UI option, so there's no concern here. Sorry for the additional noise, when I first read the thread I was assuming that you meant the unbound default is set, but pfSense turned it off.

tl;dr: Everything is good.