Learning how to dig made me a better programmer
“You never know what your next dig is going to find.”
This week at work we had a massive DNS resolution failure with one of our
datacenters. The issue was quickly resolved, but in the process, they assigned
our services new IPs which took forever to propagate. While it is known to most
people that DNS propagation can take up to 72 hours, this meant total
downtime for our customers in the meantime. Also, because our team could not do
anything about it, I simply updated all my tickets, and then took the time to
finally sit down and understand how dig works.
Digging with no shovel
A typical dig query looks like dig @server name type. But if you simply
dig and do nothing else, you end up getting the 13 root nameservers. They
play a fundamental role in the operation of DNS infrastructure.
The root nameservers do not directly handle queries for specific domain names but rather provide information about the authoritative nameservers for each top-level domain. These authoritative nameservers, in turn, handle queries for their respective domains.
dig
; <<>> DiG 9.10.6 <<>>
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11663
;; flags: qr rd ra; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;. IN NS
;; ANSWER SECTION:
. 1800 IN NS d.root-servers.net.
. 1800 IN NS h.root-servers.net.
. 1800 IN NS j.root-servers.net.
. 1800 IN NS a.root-servers.net.
. 1800 IN NS k.root-servers.net.
. 1800 IN NS i.root-servers.net.
. 1800 IN NS e.root-servers.net.
. 1800 IN NS g.root-servers.net.
. 1800 IN NS l.root-servers.net.
. 1800 IN NS m.root-servers.net.
. 1800 IN NS f.root-servers.net.
. 1800 IN NS b.root-servers.net.
. 1800 IN NS c.root-servers.net.
;; Query time: 127 msec
;; SERVER: 100.64.0.2#53(100.64.0.2)
;; WHEN: Sun Jun 25 18:21:17 IST 2023
;; MSG SIZE rcvd: 228That… is a lot of noise. Perhaps that’s because I don’t fully know what every line means. Today, I’m going to change that.
The first thing that caught my attention was that dig was using (at the time)
an unknown IP for resolving DNS queries. I know I set my primary DNS resolver
to Cloudflare’s 1.1.1.1, so where was this IP coming from?
man dig to the rescue:
If no server argument is provided, dig consults
/etc/resolv.conf; if an address is found there, it queries the nameserver at that address.
Let’s check what my /etc/resolve.conf looks like:
#
# macOS Notice
#
# This file is not consulted for DNS hostname resolution, address
# resolution, or the DNS query routing mechanism used by most
# processes on this system.
#
# To view the DNS configuration used by this system, use:
# scutil --dns
#
# SEE ALSO
# dns-sd(1), scutil(8)
#
# This file is automatically generated.
#
nameserver 100.64.0.2Ah, macOS, you never fail to amaze piss me (off). Let me check what scutil
says:
scutil --dns | grep 'nameserver\[[0-9]*\]' | sort | uniq
nameserver[0] : 1.1.1.1
nameserver[0] : 100.64.0.2
nameserver[0] : 127.0.0.1
nameserver[1] : 1.0.0.1
nameserver[2] : 2606:4700:4700::1111
nameserver[3] : 2606:4700:4700::1001My machine is using 1.1.1.1 for scoped queries, and 100.64.0.2 is used as
fallback.
;; SERVER: 100.64.0.2#53(100.64.0.2)Now, what is the #53 above? That’s the port number the Internet Assigned
Numbers Authority (IANA) assigned to be used for DNS.
Going back to the original query, because I passed no arguments/flags/options to
dig, it performed a nameserver (NS) query to the root ., which is at the
top of the DNS hierarchy. Basically, if you did a tree of the DNS hierarchy,
this is what it would look like:
. (root)
├── com (TLDs)
│ ├── example
│ │ ├── www (SLDs)
│ │ └── mail
├── ca (CC-TLDs)
│ ├── exampleThe DNS hierarchy is a lot more dense, and I have intentionally left out those
bits. You’ll also notice that each domain in the ANSWER section ends with a
.. That’s (a.root-servers.net.) what you call a Fully Qualified Domain Name
(FQDN). I won’t go any further into this topic, but
you should know that that is not an error.
. 1800 IN NS a.root-servers.net.The 1800 above is the time-to-live (TTL) for the entry, or how long the DNS
resolver will cache the particular entry to optimize subsequent queries.
. 1800 IN NS a.root-servers.net.The IN stands for the Internet class. Other classes include CH (Chaos net),
HS (Hesoid), and NONE for placeholder records. I’ll use CH in an upcoming
section for debugging purposes. The chaos class is a special class used for
querying server-related information.
Personally, I would have liked it if Chaosnet took off instead of the Internet. I like the name more.
Digging past the surface-level
Being able to query the root nameservers is great, but that’s rarely why you
would use dig in day-to-day life. You’re reading this post on
kimchiii.space; but what nameservers does my domain use?
dig +short @1.1.1.1 NS kimchiii.space.
ns1.vercel-dns.com.
ns2.vercel-dns.com.I’m sure Vercel informs its users what DNS service they use (it’s NS1, but I think in the past they use AWS’ Route 53), but it’s cool that you can find out this information from the comfort of your command-line.
Nice. It looks like ns1.vercel-dns.com. is the primary DNS server for my
domain, whereas ns2.vercel-dns.com. is the secondary DNS server.
I want to take it a step further. ns1.vercel-dns.com. and
ns2.vercel-dns.com. are authoritative nameservers for Vercel, but what IPs do
they resolve to?
dig +short @1.1.1.1 A ns1.vercel-dns.com.
198.51.44.13Who does this IP address belong to? Well if I haven’t fallen victim to DNS
spoofing, then whois 198.51.44.13 should return info on NS1.
whois 198.51.44.13
...
NetRange: 198.51.44.0 - 198.51.45.255
CIDR: 198.51.44.0/23
NetName: NSONE-DNS
NetHandle: NET-198-51-44-0-1
Parent: NET198 (NET-198-0-0-0-0)
NetType: Direct Allocation
OriginAS: AS62597
Organization: NSONE Inc (NSONE)
RegDate: 2013-08-07
Updated: 2021-12-14
Comment: http://nsone.net
Ref: https://rdap.arin.net/registry/ip/198.51.44.0
...
Phew. Not seeing NSONE Inc as the Organization would have me tripping right
now. Looks like I can move on with my experiment.
This shit gettin’ deeper and deeper, I dig it
I’m having way too much fun. I don’t want to stop now. I have already confirmed
that they have moved away from AWS for their DNS needs, but what about their
hosting? I want to find out more about the IPs used to resolve my domain
kimchiii.space.
dig +short @1.1.1.1 A kimchiii.space.
76.76.21.22
76.76.21.241Who do these IPs belong to? As of me writing this post, I think Vercel still uses AWS. Let’s confirm that:
...
OrgName: Vercel, Inc
OrgId: ZEITI
Address: 340 S LEMON AVE #4133
City: Walnut
StateProv: CA
PostalCode: 91789
Country: US
RegDate: 2020-03-26
Updated: 2020-06-05
Comment: https://vercel.com
Ref: https://rdap.arin.net/registry/entity/ZEITI
...Hmm… this is not alarming, but it is also not what I expected. Does this
mean Vercel uses its own infrastructure for hosting? I’m not so certain, because
ipinfo.io tells me that the ASN
(AS16509) for the IP 76.76.21.22 is Amazon. It
also confirms that it is allocated for hosting purposes. If you further drill
down information on the allocated IP address ranges, you will find that
76.76.21.0/24 is assigned to Vercel,
which to me, confirms that Vercel still uses AWS for hosting.
If you were to run the same query, you’d most likely get a different set of IPs each time. This is most likely because Vercel (or rather, NS1) is implementing DNS load balancing or DNS round-robin techniques. This is a good thing.
Surfacing back to the root
Going back to the tree structure of the DNS hierarchy, I should be able to trace the path to my domain’s nameserver from the root nameserver by iteratively querying the authoritative nameservers for each level of the DNS hierarchy. Let’s try that:
dig +short @1.1.1.1 NS .
a.root-servers.net.
b.root-servers.net.
c.root-servers.net.
d.root-servers.net.
e.root-servers.net.
f.root-servers.net.
g.root-servers.net.
h.root-servers.net.
i.root-servers.net.
j.root-servers.net.
k.root-servers.net.
l.root-servers.net.
m.root-servers.net.Now, replacing 1.1.1.1 with either one of the 13 root nameservers:
dig @a.root-servers.net. NS kimchiii.space.
...
;; AUTHORITY SECTION:
space. 172800 IN NS b.nic.space.
space. 172800 IN NS e.nic.space.
space. 172800 IN NS f.nic.space.
space. 172800 IN NS a.nic.space.
...Then, replacing one of the root nameservers with the authoritative nameservers above:
dig @a.nic.space. NS kimchiii.space.
...
;; AUTHORITY SECTION:
kimchiii.space. 3600 IN NS ns2.vercel-dns.com.
kimchiii.space. 3600 IN NS ns1.vercel-dns.com.
...Bingo. We have arrived at the same nameservers from the previous section.
FWIW, you can find out this information using
dig +all +trace @server <fqdn>, albeit the output is a bit too verbose
for this use-case.
What if I want to find out which nameserver node responds to my queries
using dig? As I’m writing this post, I’m connected to a NordVPN server hosted
somewhere in London. Because my site is hosted at the edge, I should be
hitting an edge datacenter hosted in London, or closest to it. Let’s confirm
that:
dig +norec +short @ns1.vercel-dns.com. CH TXT hostname.bind
"ns1dns-lhr04-11184-5310"FeelsGoodMan to be right. lhr04 gives it away. LHR is the airport code for
Heathrow Airport. Now I’m going to connect to a server in Canada (Montréal):
dig +norec +short @ns1.vercel-dns.com. CH TXT hostname.bind
"ns1dns-yyz03-11199-5323"Interesting. Looks like NS1 does not have any datacenters in Montréal because YYZ is the airport code for Toronto Pearson International Airport. Or I just don’t happen to be hitting it. This opens up an avenue for another experiment. 😏
Other records
There are several DNS record types that I need to check from time-to-time, so I thought I’d mention them here.
ANY record
This is slowly being
phased out, but a
lot of public DNS servers still respond to this record with results.
Cloudflare’s 1.1.1.1 already returns nothing:
dig +short @1.1.1.1 ANY kimchiii.space.AdGuard and Quad9 DNS return an HINFO in the answer section set to
"RFC8482" "".
dig @9.9.9.9 ANY kimchiii.space.
...
;; ANSWER SECTION:
kimchiii.space. 60 IN HINFO "RFC8482" ""
...dig @94.140.14.14 ANY kimchiii.space.
...
;; ANSWER SECTION:
kimchiii.space. 60 IN HINFO "RFC8482" ""
...You can try other DNS servers. I’ll update this post once I find one that still returns results. But you should probably stop using this record type.
MX record
An MX record directs email to a mail server. In my case, I use
ImprovMX for email forwarding. One can confirm this by
doing:
dig +short @1.1.1.1 MX kimchiii.space.
10 mx1.improvmx.com.
20 mx2.improvmx.com.The highlighted parts above tell you the priority (or preference) with which a mail server, or rather, the Message Transfer Agent (MTA) will try to send emails. Lower number is higher priority. But one can also set the same value for the priority to enable load-balancing.
PTR record
DNS PTR records serve the opposite purpose of A records, which provide the
IP address associated with a domain name. PTR records are used in reverse DNS
lookups. The most common use-case I can think of is a mail server using the
reverse lookup to confirm that an email came from the source it claims to have
come from.
dig +short @1.1.1.1 NS kimchiii.space.
ns1.vercel-dns.com.
ns2.vercel-dns.com.dig +short @1.1.1.1 ns1.vercel-dns.com.
198.51.44.13dig -x 198.51.44.13
...
;; QUESTION SECTION:
;13.44.51.198.in-addr.arpa. IN PTR
;; ANSWER SECTION:
13.44.51.198.in-addr.arpa. 1800 IN PTR dns1.p13.nsone.net.
...This is all well and good, but this is the crucial part: looking up the A
record of dns1.p13.nsone.net. should return the IP address 198.51.44.13:
dig +short @1.1.1.1 A dns1.p13.nsone.net.
198.51.44.13SOA record
No, not Points of Authority, although that be a way cooler name, IMO. The Start of Authority (SOA) record must exist for every DNS zone.
dig +all +multiline @1.1.1.1 SOA kimchiii.space.
...
;; ANSWER SECTION:
kimchiii.space. 3600 IN SOA ns1.vercel-dns.com. hostmaster.nsone.net. (
1655751660 ; serial
43200 ; refresh (12 hours)
7200 ; retry (2 hours)
1209600 ; expire (2 weeks)
60 ; minimum (1 minute)
)
...You can also use the SOA query to find out the primary nameserver for a
domain. Here’s another example:
dig +all +multiline @1.1.1.1 SOA example.com.
...
;; ANSWER SECTION:
example.com. 3600 IN SOA ns.icann.org. noc.dns.icann.org. (
2022091303 ; serial
7200 ; refresh (2 hours)
3600 ; retry (1 hour)
1209600 ; expire (2 weeks)
3600 ; minimum (1 hour)
)
...The FQDN that follows the highlighted ones above are not actually domains, but
rather an email address. For example, in the case of NS1, you would contact
hostmaster@nsone.net for any concerns. This email is different from the email
address used to contact for abuse. That can be found using whois instead.
SPF record
The Sender Policy Framework (SPF) record type doesn’t
actually exist any
more, but it can be set using the TXT record. It exists because the Simple
Mail Transfer Protocol (SMTP), by default, performs no authentication on the
“From” address in an email. Here’s what the SPF record for my domain looks like:
dig +short @1.1.1.1 TXT kimchiii.space.
"v=spf1 include:spf.improvmx.com ~all"Because there is no dedicated SPF record, your TXT record must begin
with v=spf1, or the server querying your domain won’t know of its existence.
You cannot have more than one SPF record per domain, either.
I use an include tag, which tells the server what third-party organizations
are authorized to send emails on behalf of the domain. The domain must be a
valid one, and in my case, it is set to ImprovMX’s domain.
I also made it so that unlisted emails will be marked as insecure or spam but
still accepted, using the ~all option. Other options are -all (reject any
and all emails not listed in the SPF record) and +all (any server can send
emails on your behalf). You probably don’t want to set the last option. If you
don’t set either of the all options, then you must set a redirect: tag which
tells the MTA that the SPF record is hosted by another domain.
Final thoughts
I had a lot of fun with this experiment. I started working on this post at ~0100h EST, and it is now ~0800h EST. Now, I’m going to go make me some coffee.
I left out any and all record types pertaining to DNSSEC because otherwise this post would get too long. I think that DNSSEC deserves its own post, so I’m working towards it as you read this.
Another thing I’d like to mention is that I used dig that came pre-installed
with macOS (9.10.6), which has no support for DoH or DoT. If you have that
requirement, you may want to upgrade dig to 9.16 or higher. You may also
wish to use dnscrypt-proxy
instead.
If you’ve read this far: wow, thank you! Did I get anything wrong? Anything you think I should mention? Feel free to email me.