Fastest way to resolve 100 million A-records in Python

I have a list with 100 million domain names like www.microsoft.com and would like to resol开发者_运维问答ve the IP-number to www.microsoft.com

Running a local pdns server and query localhost using Python adns?

---

I'd probably use Twisted DNS libray to do the DNS resolution from Google's Public DNS (ip address: 8.8.8.8). It'd take some trial and error but I'd guess you could have at least a couple hundred outstanding queries going at once. Google's DNS infrastructure is designed to handle a huge load and Twisted is well suited to handling thousands of simultaneous asychronous operations.

---

I actually had to solve this particular problem quite recently. All public servers I found where quite limited, and answered at a rate of around 2000 qps (queries per second). 2000 qps is clearly too low for millions of FQDNs.

I ended up configuring a recursive resolver to do this in a reasonable amount of time. Use the recursive resolver of you choice. Below I give an example of using unbound.

Using unbound

unbound should be part of most modern distros, and also available at their their page. Using the following configuration, optimized for 12 threaded server without fully killing it, I was able to go way above 2 000 qps, peaking at around 100 000 qps.

# /etc/unbound/unbound.conf
server:
        verbosity: 1
        num-threads: 10
        outgoing-range: 32768
        msg-cache-size: 400m
        msg-cache-slabs: 8
        num-queries-per-thread: 8192
        unknown-server-time-limit: 150
        rrset-cache-size: 400m
        rrset-cache-slabs: 8
        infra-cache-slabs: 8
        infra-cache-numhosts: 1000000
        trust-anchor-file: /etc/unbound/trusted-key.key
        key-cache-size: 400m
        key-cache-slabs: 8
        neg-cache-size: 100m

Testing your recursive resolver

After you have configured your recursive resolver of choice it might be worth it to test it. This dns-stresstester works well for me. You can try the following:

go get github.com/MickaelBergem/dnsstresss
cd ~/go/bin
./dnsstresss -random -concurrency 400 google.com.

This gives me:

dnsstresss - dns stress tool

Testing resolver: 127.0.0.1:53.
Target domains: [google.com.].

Started 400 threads.
Requests sent: 110445r/s        Replies received: 110445r/s (mean=3ms / max=70ms)
Requests sent: 113971r/s        Replies received: 113971r/s (mean=3ms / max=108ms)
Requests sent:  93589r/s        Replies received:  93589r/s (mean=4ms / max=111ms)
Requests sent:  93986r/s        Replies received:  93986r/s (mean=4ms / max=202ms)
Requests sent: 102510r/s        Replies received: 102510r/s (mean=4ms / max=75ms)