Background
On October 31, 2016, Slate published an article titled “Was a Trump Server Communicating With Russia?” This article described the work of Professor L. Jean Camp and her colleagues, all of whom had developed an interest in the activity of mail1.trump-email.com, an email-marketing server associated with the Trump Organization:
The computer scientists posited a logical hypothesis, which they set out to rigorously test: If the Russians were worming their way into the DNC, they might very well be attacking other entities central to the presidential campaign, including Donald Trump’s many servers. “We wanted to help defend both campaigns, because we wanted to preserve the integrity of the election,” says one of the academics, who works at a university that asked him not to speak with reporters because of the sensitive nature of his work.
This professed altruism as a rationale for digital snooping may be hard to swallow given that Camp and her associates spend a great deal of time on Twitter launching unhinged attacks against President Trump, but pushing ahead, enter Tea Leaves:
In late July, one of these scientists—who asked to be referred to as Tea Leaves, a pseudonym that would protect his relationship with the networks and banks that employ him to sift their data—found what looked like malware emanating from Russia. The destination domain had Trump in its name, which of course attracted Tea Leaves’ attention. But his discovery of the data was pure happenstance—a surprising needle in a large haystack of DNS lookups on his screen. “I have an outlier here that connects to Russia in a strange way,” he wrote in his notes. He couldn’t quite figure it out at first. But what he saw was a bank in Moscow that kept irregularly pinging a server registered to the Trump Organization on Fifth Avenue.
Tea Leaves is another piece of work in the “concerned and impartial researcher” department. Check out Tea Leaves on medium.com (@tea.leaves) play-acting as the innocently curious Lea Vestea (roughly a cyclic permutation on “Tea Leaves,” in case you missed it):
If you are a billionaire, do you have lesser people “take the fall” for you? We already saw a loyal Trump employee — who is a woman — used as a scapegoat in the scandal about Melania Trump plagiarizing a Michele Obama speech.
After reading details of an under-reported story that seems to be just emerging — I predict that The Donald will do it again and yet again find a woman amongst his large organization to blame his own extremely risky actions upon.
[…]
Building your business on the backs of little people — whether your own employees or contractors who don’t get paid — is one thing. Using ladies to take the blame for things you yourself have done — now that’s not manly. I’m not seeing the broad shoulders there.
Why is a Russian bank called Alfa Bank hooked up to a Trump Organization email server with only one other user? I mean, three billionaires walk into a bar… help me finish the joke!
Back to the Slate article. We were assured that Tea Leaves is a DNS expert:
(I communicated extensively with Tea Leaves and two of his closest collaborators, who also spoke with me on the condition of anonymity, since they work for firms trusted by corporations and law enforcement to analyze sensitive data. They persuasively demonstrated some of their analytical methods to me—and showed me two white papers, which they had circulated so that colleagues could check their analysis. I also spoke with academics who vouched for Tea Leaves’ integrity and his unusual access to information. “This is someone I know well and is very well-known in the networking community,” said Camp. “When they say something about DNS, you believe them. This person has technical authority and access to data.”)
We were assured that known DNS experts vouched for the authenticity of the DNS logs. We were assured that nine computer scientists considered it nearly impossible for such logs to be forged or manipulated. And thus a virulent narrative was reinforced:
The Slate report was met with much skepticism from tech reporters and bloggers, who correctly remarked that nothing conclusive could be inferred from the DNS logs provided. It seemed everybody was content to accept the DNS logs as a premise of sorts, that only the conclusion was open to debate.
I myself considered the matter a closed case when The Intercept published a couple of marketing emails sent from mail1.trump-email.com to the parties concerned. The Intercept article seemed to validate the DNS logs and debunk the secret-communication theory in one fell swoop. Even so, something kept bothering me about the fact that the emails bore pre-March timestamps (Cendyn claims the server sent its last marketing email in March 2016, and the DNS logs purport to show a May-September window of high-volume activity after an initial May drizzle). Something also kept bothering me about certain aspects of the DNS logs themselves, and after the usual news outlets tried reigniting the story recently, I decided it was time to take another look.
If you want more complete background information before I present my own findings, I refer you to the following:
- http://www.slate.com/articles/news_and_politics/cover_story/2016/10/was_a_server_registered_to_the_trump_organization_communicating_with_russia.html
- http://www.slate.com/articles/news_and_politics/politics/2016/11/the_trump_server_evaluating_new_evidence_and_countertheories.html
- http://blog.erratasec.com/2016/11/debunking-trumps-secret-server.html
- http://blog.erratasec.com/2016/11/in-which-i-have-to-debunk-second-time.html
- https://theintercept.com/2016/11/01/heres-the-problem-with-the-story-connecting-russia-to-donald-trumps-email-server/
- https://medium.com/@randomvariable/on-the-slate-story-about-trumps-private-email-server-9f4391a72e26
- https://www.theverge.com/2016/11/1/13486596/trump-russia-alfabank-server-hotline-connection-debunk
- https://krypt3ia.wordpress.com/2016/10/08/gdd53-a-russian-hosted-i2p-site-that-claims-trumps-email-system-had-ties-to-alfabank-russia/
- https://krypt3ia.wordpress.com/2016/11/01/shits-gone-plaid-gdd53-and-slate/
- https://www.vox.com/policy-and-politics/2016/11/1/13484340/trump-russia-secret-server
- https://www.washingtonpost.com/news/the-fix/wp/2016/11/01/that-secret-trump-russia-email-server-link-is-likely-neither-secret-nor-a-trump-russia-link/
- https://www.nytimes.com/2016/11/01/us/politics/fbi-russia-election-donald-trump.html
- https://edition.cnn.com/2017/03/09/politics/fbi-investigation-continues-into-odd-computer-link-between-russian-bank-and-trump-organization/
Who is Tea Leaves?
If you’ve read the background material, you’ll understand that Tea Leaves, as the supplier of the DNS logs, is the foundation on which this whole saga ultimately rests. While Camp and her colleagues are happy to run their mouths and make an awful lot of noise, Tea Leaves has remained conveniently tucked away from public scrutiny, creating an I2P eepsite for the DNS logs and hounding tech bloggers via Tor. The reason given to The Intercept is that Tea Leaves runs a cybersecurity company and doesn’t want it to be the target of a DDoS attack. I suspect that’s not the real reason — not that I’m any kind of moralist, but I suspect the real reason involves the highly unethical behavior, along with the possible breach of contracts and non-disclosure agreements.
Tea Leaves is DNS and malware researcher April Lorenzen:
Click to enlarge
EXHIBIT A
Examine the archived copy of gdd.i2p.xyz:
http://web.archive.org/web/20161107125853/http://gdd.i2p.xyz/
Notice the clause “Two of the authoritative name server hosts deleted the zone.” The phrase “authoritative name server hosts” is highly idiosyncratic. Most people would just write “authoritative name servers” or “authoritative nameservers.” It’s so idiosyncratic that you can enter it into Google (in quotation marks) and get only five results, three of which come from April Lorenzen in some form, and the phrase additionally appears on one of Lorenzen’s sites:
Click to enlarge
Archived copy of GitHub page: http://archive.is/lWoNm
Archived copy of Dissect Cyber page: http://archive.is/PIBMo
EXHIBIT B
Now, examine the HTML source for http://web.archive.org/web/20161107125853/http://gdd.i2p.xyz/, and you’ll find this sort of thing:
<a href="intranet" target="intranet"> <a href="cendynbank" target="cendynbank"> <a href="/web/20161107125853/https://www.virustotal.com/en/file/05e384f3aef08b7405d0d8c7a381d09406943b86ce3f538540652394ba05fdb2/analysis/" target="virustotal"> <a href="/web/20161107125853/https://krypt3ia.wordpress.com/" target="bigpond"> <a href="a.pdf" target="alfabackground"> <a href="/web/20161107125853/http://docquery.fec.gov/pres/2015/Q3/C00580100.html" target="fec1"> <a href="/web/20161107125853/https://en.wikipedia.org/wiki/Alfa-Bank" target="Alfa-Bank"> <a href="/web/20161107125853/http://www.politico.com/magazine/story/2016/09/the-mystery-of-trumps-man-in-moscow-214283" target="l1"> <a href="/web/20161107125853/http://www.akm.ru/eng/news/2015/november/23/ns5365780.htm" target="insurance"> <a href="/web/20161107125853/http://www.motherjones.com/politics/2014/01/devos-family-foundations-heritage-americans-prosperity-blackwater" target="devos"> <a href="/web/20161107125853/https://mjassets.s3.amazonaws.com/politics/2013/12/Devos_Familytree-960.jpeg" target="founder"> <a href="/web/20161107125853/https://en.wikipedia.org/wiki/Academi" target="blackwater">Blackwater</a>. <a href="/web/20161107125853/https://www.bloomberg.com/news/articles/2016-07-14/russian-billionaires-plan-u-s-health-push-with-d-c-insiders" target="bloomberg">
Notice anything wrong? That’s a highly peculiar use of the “target” attribute, as any Web developer will confirm at once. These aren’t targets like _blank, which has an understood meaning in frame-less documents. These targets are instead named frames, but there are no frames in this document, let alone that many frames, let alone that many frames with those names. Try finding another frame-less site that does this, and you’ll learn how idiosyncratic the coding construct is.
(Aside: I think the “bigpond” target may have been an accidental paste, for it has no apparent connection to the krypt3ia blog. It might be fun to mention that BigPond is one of the many foreign ISPs shipping Technicolor ADSL modems with a TR-069/CWMP service on TCP port 51005 open to the entire Internet — even when the modem is in bridged mode, because the NAT mapping takes priority. Protocol-wise, the service is protected with a secret key, and this is the sort of thing that placates the turnips and zombies, but in the eyes of a world-class adversary, this is attack surface, and 15-30 minutes spent poring over the disassembly of the relevant Linux/MIPS binary will yield a somewhat subtle pre-authentication memory-corruption vulnerability that can be exploited to hijack the modem remotely for traffic monitoring, a primitive bounce point, a DDoS node, an improvised explosive using the lawn mower running off the USB port, whatever.)
Anyway, the same highly peculiar use of the “target” attribute can be found on Lorenzen’s hand-coded sites. I’ll show a small sample using dissectcyber.com, say2.us, and the defunct secure411.org (hence the GitHub account name of secure411dotorg):
<a href="https://vimeo.com/84083759" target="vim84"> <a href="https://vimeo.com/84083759" target="vim84">
This is difficult to preserve in an archiver. The archiver archive.is strips out plenty of attributes as part of its reformatting, and the archiver web.archive.org, though preserving the attributes, allows dissectcyber.com to be delisted via a robots.txt exclusion over which Lorenzen has control. What we’ll do is use archive.is to archive the W3C validation results, with the source listing appearing at the bottom.
Archived copy of W3C validation of Dissect Cyber page: http://archive.is/hD3f8
<a href="https://dissectcyber.com/privacy_policy_dissect_cyber_dot_com.html" target="privacypolicy"> <a href="https://dissectcyber.com/privacy_policy_dissect_cyber_dot_com.html" target="privacypolicy">
Archived copy of W3C validation of Dissect Cyber page: http://archive.is/NcRcc
<a href="https://www.dhs.gov/science-and-technology/csd-imam" target="2016Febslides"> <a href="http://www.pcworld.com/article/2987626/dyreza-malware-steals-it-supply-chain-credentials.html" target="pony1"> <a href="http://www.networkworld.com/article/3095486/security/cybersecurity-is-only-as-strong-as-your-weakest-linkyour-employees.html" target="pony3"> <a href="https://www.research.net/r/say2us" target="letus"> <a href="https://www.fbi.gov/" target="fbigov"> <a href="https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&cad=rja&uact=8&ved=0ahUKEwi9sdT-ltjNAhVIaD4KHaF4B3QQFggnMAI&url=http%3A%2F%2Fwww.wsj.com%2Farticles%2Fhackers-trick-email-systems-into-wiring-them-large-sums-1438209816&usg=AFQjCNEg7VA7IC4knkIPKsCJVwKT9UQgMQ" target="wsj"> <a href="http://www.trendmicro.com/vinfo/us/security/news/cybercrime-and-digital-threats/business-email-compromise-bec-schemes" target="notanendorsementoftrendmicro"> <a href="http://www.abuseat.org/lookup.cgi" target="cbl"> <a href="http://www.abuseat.org/nat.html" target="cblnat"> <a href="https://www.research.net/r/say2us" target="say2us"> <a href="https://www.google.com/webmasters/hacked/" target="help4hacked"> <a href="https://www.google.com/transparencyreport/safebrowsing/diagnostic/" target="goodx"> <a href="https://www.research.net/r/say2us" target="policyfeedback" id="hen" class="button"> <a href="https://www.research.net/r/say2us" target="dissectcyber"> <a href="http://www.cvent.com/events/cyber-security-division-2016-r-d-showcase-and-technical-workshop-february-17-19-2016-the-mayflower-r/custom-35-74921946b25f4e6e81b9add8aa7f24fb.aspx" target="2016Febagenda"> <a href="https://www.dhs.gov/science-and-technology/news/2016/02/10/dhs-st-hosts-cyber-security-research-and-development-showcase" target="dhssandt"> <a href="https://www.fbi.gov/contact-us/field/listing_by_state" target="fbioffices"> <a href="https://www.icann.org/resources/pages/dndr-2012-02-25-en" target="udrp">
Archived copy of W3C validation of Say2.us page: http://archive.is/wEKOJ
<a href="/web/20150125114023/https://codelock.com/" target="codelock"> <a href="/web/20150125114023/https://www.cleverbridge.com/342/cookie?affiliate=2557&product=29945&redirectto=http%3a%2f%2fwww.malwarebytes.org%2fmbam.php" target="mbyte"> <a href="/web/20150125114023/http://sourceforge.net/projects/clamwin/" target="clam"> <a href="/web/20150125114023/http://www.winpatrol.com/download.html" target="winp"> <a href="/web/20150125114023/http://www.mozilla.com/" target="ff"> <a href="/web/20150125114023/http://www.mozilla.com/thunderbird" target="tbird"> <a href="/web/20150125114023/http://www.cypherpunks.ca/otr/" target="otr"> <a href="/web/20150125114023/http://pidgin.im/" target="pidg"> <a href="/web/20150125114023/http://codelock.com/secure802/" target="enig"> <a href="/web/20150125114023/http://codelock.com/" target="ob"> <a href="/web/20150125114023/http://www.networkliquidators.com/article-wireless-router-security-a-step-by-step-guide.asp" target="nl"> <a href="/web/20150125114023/http://www.amazon.com/dp/B001FOQI7O?tag=jamestowncommuni&camp=14573&creative=327641&linkCode=as1&creativeASIN=B001FOQI7O&adid=1Y86G48VKTBJX5C38ADA&" target="ama"> <a href="/web/20150125114023/http://quetek.com/download.htm" target="filesc"> <a href="/web/20150125114023/http://nik.bourbaki.googlepages.com/step-by-stephow-to" target="trk"> <a href="/web/20150125114023/https://www.cleverbridge.com/342/cookie?affiliate=2557&product=29945&redirectto=http%3a%2f%2fwww.malwarebytes.org%2fmbam.php" target="mbyte"> <a href="/web/20150125114023/http://www.amazon.com/gp/subs/primeclub/signup/extmain.html?ref=prime_assoc_bt&tag=jamestowncommuni-20" target="prime"> <a href="/web/20150125114023/http://quetek.com/download.htm" target="filesc"> <a href="/web/20150125114023/http://quetek.com/download.htm" target="filesc"> <a href="/web/20150125114023/http://quetek.com/download.htm" target="filesc"> <a href="/web/20150125114023/http://quetek.com/download.htm" target="filesc"> <a href="/web/20150125114023/http://www.searchlores.org/commonpass1.htm" target="srch"> <a href="/web/20150125114023/http://www.schneier.com/essay-144.html" target="bruce"> <a href="/web/20150125114023/http://www.amazon.com/gp/subs/primeclub/signup/extmain.html?ref=prime_assoc_bt&tag=jamestowncommuni-20" target="prime"> <a href="/web/20150125114023/http://www.networkliquidators.com/article-wireless-router-security-a-step-by-step-guide.asp" target="nl">
Archived copy of Secure411 page: http://web.archive.org/web/20150125114023/http://secure411.org/
Where are the DNS logs?
Tea Leaves originally posted the logs to her I2P eepsite gdd.i2p. They became accessible to clearnet users via the i2p.xyz inproxy as gdd.i2p.xyz. This in turn allowed the logs to be archived:
- http://web.archive.org/web/20161107125853/http://gdd.i2p.xyz/logs/ns1_cdcservices_com.log
- http://web.archive.org/web/20161107125853/http://gdd.i2p.xyz/logs/ns2_cdcservices_com.log
- http://web.archive.org/web/20161107125853/http://gdd.i2p.xyz/logs/ns3_cdcservices_com.log
Copies of the logs also appear on Camp’s personal site:
- Archived copy of Camp’s data page: http://archive.is/9pFX0
- Archived copy of log #1 on Camp’s data page: http://archive.is/VZyCC
- Archived copy of log #2 on Camp’s data page: http://archive.is/gQMQu
- Archived copy of log #3 on Camp’s data page: http://archive.is/Vn6tU
What is wrong with the DNS logs?
The DNS logs are ostensibly BIND9 logs for {ns1,ns2,ns3}.cdcservices.com, formerly the three trump-email.com authoritative nameservers. The problem, however, is that every log line contains four flaws revealing that the logs aren’t authentic BIND9 logs and have been manipulated to appear as such.
For each flaw, I present an easy-to-grasp image and then a rising level of technical analysis. Casual readers ought to be satisfied with the images to get the basic idea and can safely ignore the severe autism.
FLAW #1: Wrong IP address for client destination
As indicated in the image above, the IP address in parentheses is meant to be the IP address of the nameserver that received the query for mail1.trump-email.com, not the IP address of mail1.trump-email.com itself. The person who made these logs had the mistaken assumption that the IP address is the one sent back to the client as an answer.
BIND9 logs for {ns1,ns2,ns3}.cdcservices.com would’ve contained in parentheses the IP address of the relevant nameserver. The IP addresses are as follows:
- ns1.cdcservices.com
- 64.135.26.101 (old)
- 23.23.65.156 (new)
- ns2.cdcservices.com
- 198.91.42.102 (old)
- 52.8.215.65 (new)
- ns3.cdcservices.com
- 64.135.26.103 (old)
- 104.43.163.188 (new)
See the following for confirmation:
- http://research.dnstrails.com/tools/lookup.htm?hostname=ns1.cdcservices.com
- http://research.dnstrails.com/tools/lookup.htm?hostname=ns2.cdcservices.com
- http://research.dnstrails.com/tools/lookup.htm?hostname=ns3.cdcservices.com
This particular query-log element first appeared in BIND9 source on 3 March 2009 with the commit message “Log the destination address the query was sent to.” It has never changed its meaning.
Here’s the code from a recent release:
isc_netaddr_format(&client->destaddr, onbuf, sizeof(onbuf));
if (client->ednsversion >= 0)
snprintf(ednsbuf, sizeof(ednsbuf), "E(%d)",
client->ednsversion);
ns_client_log(client, NS_LOGCATEGORY_QUERIES, NS_LOGMODULE_QUERY,
level, "query: %s %s %s %s%s%s%s%s%s%s (%s)", namebuf,
classname, typename, WANTRECURSION(client) ? "+" : "-",
(client->signer != NULL) ? "S" : "", ednsbuf,
TCP(client) ? "T" : "",
((extflags & DNS_MESSAGEEXTFLAG_DO) != 0) ? "D" : "",
((flags & DNS_MESSAGEFLAG_CD) != 0) ? "C" : "",
HAVECOOKIE(client) ? "V" : WANTCOOKIE(client) ? "K" : "",
onbuf);
FLAW #2: Wrong client format
A genuine client format will, minimally, look like this:
client 1.2.3.4#5678:
The important things to note are the hash character, the port number, and the colon — none of which appears in the Tea Leaves logs. The colon appearing after “query” in the Tea Leaves logs is meant to be the second colon, not the first.
Although the client format has changed somewhat throughout BIND9’s long and troubled history (BIND 9.11 even adds a pointer value to the format) — and although extra details may be inserted between the port number and the colon in particular use cases — the hash character, the port number, and the colon should always appear.
I give you BIND 9.x.0 from the present to the very beginning:
isc_log_write(ns_g_lctx, category, module, level,
"client @%p %s%s%s%s%s%s%s%s: %s",
client, peerbuf, sep1, signer, sep2, qname, sep3,
sep4, viewname, msgbuf);
Notice the colon in the format string above. This colon doesn’t appear in the Tea Leaves logs — only the second colon does. As a quick demonstration of how the second colon arises, observe the %s after the colon. This %s gets replaced by msgbuf, which has been constructed variadically within ns_client_logv() from the following format string and associated arguments:
ns_client_log(client, NS_LOGCATEGORY_QUERIES, NS_LOGMODULE_QUERY,
level, "query: %s %s %s %s%s%s%s%s%s%s (%s)", namebuf,
The hash character and port number are unavoidably part of peerbuf, which has been constructed within isc_sockaddr_totext():
isc_buffer_putmem(target, (const unsigned char *)"#", 1);
isc_buffer_putmem(target, (const unsigned char *)pbuf, plen);
Click to expand earlier versions if you’re so inclined:
isc_log_write(ns_g_lctx, category, module, level,
"client %s%s%s%s%s%s%s%s: %s",
peerbuf, sep1, signer, sep2, qname, sep3,
sep4, viewname, msgbuf);
isc_buffer_putmem(target, (const unsigned char *)"#", 1);
isc_buffer_putmem(target, (const unsigned char *)pbuf, plen);
isc_log_write(ns_g_lctx, category, module, level,
"client %s%s%s%s%s%s%s%s: %s",
peerbuf, sep1, signer, sep2, qname, sep3,
sep4, viewname, msgbuf);
isc_buffer_putmem(target, (const unsigned char *)"#", 1);
isc_buffer_putmem(target, (const unsigned char *)pbuf, plen);
isc_log_write(ns_g_lctx, category, module, level,
"client %s%s%s: %s", peerbuf, sep, name, msgbuf);
isc_buffer_putmem(target, (const unsigned char *)"#", 1);
isc_buffer_putmem(target, (const unsigned char *)pbuf, plen);
isc_log_write(ns_g_lctx, category, module, level,
"client %s%s%s: %s", peerbuf, sep, name, msgbuf);
isc_buffer_putmem(target, (const unsigned char *)"#", 1);
isc_buffer_putmem(target, (const unsigned char *)pbuf, plen);
isc_log_write(ns_g_lctx, category, module, level,
"client %s%s%s: %s", peerbuf, sep, name, msgbuf);
isc_buffer_putmem(target, (const unsigned char *)"#", 1);
isc_buffer_putmem(target, (const unsigned char *)pbuf, plen);
isc_log_write(ns_g_lctx, category, module, level,
"client %s%s%s: %s", peerbuf, sep, name, msgbuf);
isc_buffer_putmem(target, (const unsigned char *)"#", 1);
isc_buffer_putmem(target, (const unsigned char *)pbuf, plen);
isc_log_write(ns_g_lctx, category, module, level,
"client %s%s%s: %s", peerbuf, sep, name, msgbuf);
isc_buffer_putmem(target, (const unsigned char *)"#", 1);
isc_buffer_putmem(target, (const unsigned char *)pbuf, plen);
isc_log_write(ns_g_lctx, category, module, level,
"client %s%s%s: %s", peerbuf, sep, name, msgbuf);
isc_buffer_putmem(target, (const unsigned char *)"#", 1);
isc_buffer_putmem(target, (const unsigned char *)pbuf, plen);
isc_log_write(ns_g_lctx, category, module, level,
"client %s: %s", peerbuf, msgbuf);
isc_buffer_putmem(target, (const unsigned char *)"#", 1);
isc_buffer_putmem(target, (const unsigned char *)pbuf, plen);
isc_log_write(ns_g_lctx, category, module, level,
"client %s: %s", peerbuf, msgbuf);
isc_buffer_putmem(target, (const unsigned char *)"#", 1);
isc_buffer_putmem(target, (const unsigned char *)pbuf, plen);
isc_log_write(ns_g_lctx, category, module, level,
"client %s: %s", peerbuf, msgbuf);
isc_buffer_putmem(target, (const unsigned char *)"#", 1);
isc_buffer_putmem(target, (const unsigned char *)pbuf, plen);
FLAW #3: Wrong time format
A genuine time format will contain milliseconds after the seconds value:
05:25:30.475
The display of milliseconds has always been an inextricable part of the timestamp.
Here again is BIND 9.x.0 from the present to the very beginning:
now = (time_t) t->seconds;
flen = strftime(buf, len, "%d-%b-%Y %X", localtime(&now));
INSIST(flen < len);
if (flen != 0)
snprintf(buf + flen, len - flen,
".%03u", t->nanoseconds / NS_PER_MS);
else {
strncpy(buf, "99-Bad-9999 99:99:99.999", len);
buf[len - 1] = 0;
}
Click to expand earlier versions if you’re so inclined:
now = (time_t) t->seconds;
flen = strftime(buf, len, "%d-%b-%Y %X", localtime(&now));
INSIST(flen < len);
if (flen != 0)
snprintf(buf + flen, len - flen,
".%03u", t->nanoseconds / 1000000);
else
snprintf(buf, len, "99-Bad-9999 99:99:99.999");
now = (time_t) t->seconds;
flen = strftime(buf, len, "%d-%b-%Y %X", localtime(&now));
INSIST(flen < len);
if (flen != 0)
snprintf(buf + flen, len - flen,
".%03u", t->nanoseconds / 1000000);
else
snprintf(buf, len, "99-Bad-9999 99:99:99.999");
now = (time_t) t->seconds;
flen = strftime(buf, len, "%d-%b-%Y %X", localtime(&now));
INSIST(flen < len);
if (flen != 0)
snprintf(buf + flen, len - flen,
".%03u", t->nanoseconds / 1000000);
else
snprintf(buf, len, "99-Bad-9999 99:99:99.999");
now = (time_t) t->seconds;
flen = strftime(buf, len, "%d-%b-%Y %X", localtime(&now));
INSIST(flen < len);
if (flen != 0)
snprintf(buf + flen, len - flen,
".%03u", t->nanoseconds / 1000000);
else
snprintf(buf, len, "99-Bad-9999 99:99:99.999");
now = (time_t) t->seconds;
flen = strftime(buf, len, "%d-%b-%Y %X", localtime(&now));
INSIST(flen < len);
if (flen != 0)
snprintf(buf + flen, len - flen,
".%03u", t->nanoseconds / 1000000);
else
snprintf(buf, len, "99-Bad-9999 99:99:99.999");
now = (time_t) t->seconds;
flen = strftime(buf, len, "%d-%b-%Y %X", localtime(&now));
INSIST(flen < len);
if (flen != 0)
snprintf(buf + flen, len - flen,
".%03u", t->nanoseconds / 1000000);
else
snprintf(buf, len, "99-Bad-9999 99:99:99.999");
now = (time_t) t->seconds;
flen = strftime(buf, len, "%d-%b-%Y %X", localtime(&now));
INSIST(flen < len);
if (flen != 0)
snprintf(buf + flen, len - flen,
".%03u", t->nanoseconds / 1000000);
else
snprintf(buf, len, "99-Bad-9999 99:99:99.999");
now = (time_t) t->seconds;
flen = strftime(buf, len, "%d-%b-%Y %X", localtime(&now));
INSIST(flen < len);
if (flen != 0)
snprintf(buf + flen, len - flen,
".%03u", t->nanoseconds / 1000000);
else
snprintf(buf, len, "99-Bad-9999 99:99:99.999");
now = (time_t) t->seconds;
flen = strftime(buf, len, "%b %d %X", localtime(&now));
INSIST(flen < len);
if (flen != 0)
snprintf(buf + flen, len - flen,
".%03u", t->nanoseconds / 1000000);
else
snprintf(buf, len, "Bad 00 99:99:99.999");
timeptr = localtime(&now);
/*
* Emulate syslog's time format,
* with milliseconds.
*
* It would be nice if the format
* were configurable.
*/
strftime(time_string, sizeof(time_string),
"%b %d %X", timeptr);
len = strlen(time_string);
snprintf(time_string + len,
sizeof(time_string) - len,
".%03u ",
isc_time_nanoseconds(&isctime)
/ 1000000);
timeptr = localtime(&now);
/*
* Emulate syslog's time format,
* with milliseconds.
*
* It would be nice if the format
* were configurable.
*/
strftime(time_string, sizeof(time_string),
"%b %d %X", timeptr);
len = strlen(time_string);
snprintf(time_string + len,
sizeof(time_string) - len,
".%03u ",
isc_time_nanoseconds(&isctime)
/ 1000000);
FLAW #4: Recursion-desired bit set
The ‘+’ character in the logs indicates that the recursion-desired (rd) bit has been set in the incoming DNS queries. This is allowed by the DNS protocol, but it’s not something a typical nameserver will do when communicating with another nameserver (except when a forwarder is involved, which doesn’t apply here). Instead, the nameserver will send an iterative query to the other nameserver (rd bit clear, indicated in BIND9 logs by the ‘-‘ character).
Examine the following image:
Red arrows indicate recursive queries (rd bit set), and blue arrows indicate iterative queries (rd bit clear). The Russian nameserver is one of the two Alfa-Bank nameservers, and the American nameserver is one of the three cdcservices.com nameservers that were authoritative for the trump-email.com domain. (Neither caching nor initial queries to ?.root-servers.net/?.gtld-servers.net are important here.)
In (a), we have the highly typical situation of a stub resolver sending a recursive query to one of its nameservers. The nameserver then issues iterative queries to external nameservers, finding the answer the stub resolver seeks.
In (b), we have the nameserver misconfigured as a so-called open resolver, allowing Internet pests to send their own recursive queries (often with spoofed source addresses for DDoS purposes). What’s important to understand here is that the nameserver will still send iterative queries when communicating with external nameservers. In effect, the Internet pests are simply stub resolvers themselves. I mention this point because of a recurring misunderstanding. Another point worth mentioning is that the Russian nameservers in all three situations are often referred to as recursive nameservers — because they provide recursive service to clients, not because they send recursive queries themselves.
In (c), we have the highly atypical situation of a nameserver sending a recursive query to another nameserver. This is what the Tea Leaves DNS logs indicate with the ‘+’ characters on every line.
We can do better than calling it highly atypical. So far we’ve been examining dodgy BIND9 logs associated in some form with the three American nameservers, without really needing to inspect those American nameservers. For the sake of analyzing rd-bit behavior, though, it’s helpful to know that the two Russian nameservers are (and were) identifiable as BIND9 nameservers. The amount of assurance one can have in this fact depends on what tools one has available and how far one is prepared to go. For a moderate amount of assurance, the fingerprinting capabilities of public tools such as nmap (-sV switch) will suffice. The usual TXT/CH queries (version.bind, etc.) won’t be all that helpful.
Now, it can be shown that BIND9 has never set the rd bit when communicating with external nameservers (except, as mentioned earlier, when a forwarder is involved — not the case here).
Here’s the code from a recent release:
/*
* Set RD if the client has requested that we do a recursive query,
* or if we're sending to a forwarder.
*/
if ((query->options & DNS_FETCHOPT_RECURSIVE) != 0 ||
ISFORWARDER(query->addrinfo))
fctx->qmessage->flags |= DNS_MESSAGEFLAG_RD;
The comment, at first glance, seems to be contradicting what I’ve said, but the client mentioned in the comment is the library client, not the network stub-resolver client we’ve been discussing. The BIND9 nameserver uses this resolver library, but in principle the library can be yanked out of BIND9 and used elsewhere, and that third-party software may wish to set the DNS_FETCHOPT_RECURSIVE option, which will in turn set DNS_MESSAGEFLAG_RD. Nothing in BIND9 sets this option:
$ grep -r DNS_FETCHOPT_RECURSIVE . ./lib/dns/include/dns/resolver.h:#define DNS_FETCHOPT_RECURSIVE 0x004 /*%< Set RD? */ ./lib/dns/resolver.c: if ((query->options & DNS_FETCHOPT_RECURSIVE) != 0 ||
What about DNS_MESSAGEFLAG_RD?
$ grep -r DNS_MESSAGEFLAG_RD . | cat -n
1 ./bin/dig/dig.c: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0)
2 ./bin/dig/dig.c: (msg->flags & DNS_MESSAGEFLAG_RD) != 0 &&
3 ./bin/dig/dighost.c: lookup->sendmsg->flags |= DNS_MESSAGEFLAG_RD;
4 ./bin/dig/host.c: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0) {
5 ./bin/named/client.c: if ((client->message->flags & DNS_MESSAGEFLAG_RD) != 0)
6 ./bin/named/client.c: client->message->flags &= ~DNS_MESSAGEFLAG_RD;
7 ./bin/named/client.c: (client->message->flags & DNS_MESSAGEFLAG_RD) == 0))
8 ./bin/named/client.c: if ((client->message->flags & DNS_MESSAGEFLAG_RD) != 0)
9 ./bin/named/query.c: if ((message->flags & DNS_MESSAGEFLAG_RD) != 0)
10 ./bin/named/query.c: if ((message->flags & DNS_MESSAGEFLAG_RD) != 0)
11 ./bin/named/query.c: (message->flags & DNS_MESSAGEFLAG_RD) == 0) {
12 ./bin/nsupdate/nsupdate.c: soaquery->flags |= DNS_MESSAGEFLAG_RD;
13 ./bin/tests/system/pipelined/pipequeries.c: message->flags |= DNS_MESSAGEFLAG_RD;
14 ./bin/tools/mdig.c: if ((response->flags & DNS_MESSAGEFLAG_RD) != 0)
15 ./bin/tools/mdig.c: if ((response->flags & DNS_MESSAGEFLAG_RD) != 0 &&
16 ./bin/tools/mdig.c: message->flags |= DNS_MESSAGEFLAG_RD;
17 ./contrib/dnsperf-2.1.0.0-1/dns.c: flags = DNS_MESSAGEFLAG_RD;
18 ./lib/dns/client.c: soaquery->flags |= DNS_MESSAGEFLAG_RD;
19 ./lib/dns/include/dns/message.h:#define DNS_MESSAGEFLAG_RD 0x0100U
20 ./lib/dns/include/dns/message.h:#define DNS_MESSAGE_REPLYPRESERVE (DNS_MESSAGEFLAG_RD|DNS_MESSAGEFLAG_CD)
21 ./lib/dns/message.c: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0)
22 ./lib/dns/resolver.c: fctx->qmessage->flags |= DNS_MESSAGEFLAG_RD;
23 ./lib/dns/resolver.c: ((fctx->qmessage->flags & DNS_MESSAGEFLAG_RD) != 0))
24 ./lib/dns/resolver.c: if ((fctx->qmessage->flags & DNS_MESSAGEFLAG_RD) != 0)
25 ./lib/dns/resolver.c: if ((fctx->qmessage->flags & DNS_MESSAGEFLAG_RD) != 0)
- Lines 1-4 are for the dig tool. Irrelevant.
- Lines 5-11 only test or unset the flag.
- Line 12 is for the nsupdate tool. Irrelevant.
- Line 13 is part of the testing kit. Irrelevant.
- Lines 14-16 are for the mdig tool. Irrelevant.
- Line 17 is for the dnsperf tool. Irrelevant.
- Line 18 is for an irrelevant query type. But leaving nothing to chance, I’ll note that it exists in the function request_soa(), which is called by only dns_client_startupdate(), which is called by only dns_client_update(), which is called by only sample code in lib/samples/sample-update.c.
- Line 19 simply defines the flag value.
- Line 20 combines two flags and is used as only a preservation mask for replies (i.e., it won’t set rd itself):
$ grep -r DNS_MESSAGE_REPLYPRESERVE . ./lib/dns/include/dns/message.h:#define DNS_MESSAGE_REPLYPRESERVE (DNS_MESSAGEFLAG_RD|DNS_MESSAGEFLAG_CD) ./lib/dns/message.c: msg->flags &= DNS_MESSAGE_REPLYPRESERVE;
- Line 21 only tests the flag.
- Line 22 is line 2185 in the code we’ve already examined.
- Lines 23-25 only test the flag.
Examine earlier behavior if you’re so inclined:
$ ls
bind-9.0.0 bind-9.10.0 bind-9.2.0 bind-9.4.0 bind-9.6.0 bind-9.8.0
bind-9.1.0 bind-9.11.0 bind-9.3.0 bind-9.5.0 bind-9.7.0 bind-9.9.0
$ grep -r 'Set RD if' . | wc -l
12
$ grep -rn -B1 -A5 'Set RD if' .
./bind-9.0.0/lib/dns/resolver.c-878- /* ./bind-9.0.0/lib/dns/resolver.c:879: * Set RD if the client has requested that we do a recursive query, ./bind-9.0.0/lib/dns/resolver.c-880- * or if we're sending to a forwarder. ./bind-9.0.0/lib/dns/resolver.c-881- */ ./bind-9.0.0/lib/dns/resolver.c-882- if ((query->options & DNS_FETCHOPT_RECURSIVE) != 0 || ./bind-9.0.0/lib/dns/resolver.c-883- ISFORWARDER(query->addrinfo)) ./bind-9.0.0/lib/dns/resolver.c-884- fctx->qmessage->flags |= DNS_MESSAGEFLAG_RD; -- ./bind-9.1.0/lib/dns/resolver.c-896- /* ./bind-9.1.0/lib/dns/resolver.c:897: * Set RD if the client has requested that we do a recursive query, ./bind-9.1.0/lib/dns/resolver.c-898- * or if we're sending to a forwarder. ./bind-9.1.0/lib/dns/resolver.c-899- */ ./bind-9.1.0/lib/dns/resolver.c-900- if ((query->options & DNS_FETCHOPT_RECURSIVE) != 0 || ./bind-9.1.0/lib/dns/resolver.c-901- ISFORWARDER(query->addrinfo)) ./bind-9.1.0/lib/dns/resolver.c-902- fctx->qmessage->flags |= DNS_MESSAGEFLAG_RD; -- ./bind-9.10.0/lib/dns/resolver.c-1902- /* ./bind-9.10.0/lib/dns/resolver.c:1903: * Set RD if the client has requested that we do a recursive query, ./bind-9.10.0/lib/dns/resolver.c-1904- * or if we're sending to a forwarder. ./bind-9.10.0/lib/dns/resolver.c-1905- */ ./bind-9.10.0/lib/dns/resolver.c-1906- if ((query->options & DNS_FETCHOPT_RECURSIVE) != 0 || ./bind-9.10.0/lib/dns/resolver.c-1907- ISFORWARDER(query->addrinfo)) ./bind-9.10.0/lib/dns/resolver.c-1908- fctx->qmessage->flags |= DNS_MESSAGEFLAG_RD; -- ./bind-9.11.0/lib/dns/resolver.c-2177- /* ./bind-9.11.0/lib/dns/resolver.c:2178: * Set RD if the client has requested that we do a recursive query, ./bind-9.11.0/lib/dns/resolver.c-2179- * or if we're sending to a forwarder. ./bind-9.11.0/lib/dns/resolver.c-2180- */ ./bind-9.11.0/lib/dns/resolver.c-2181- if ((query->options & DNS_FETCHOPT_RECURSIVE) != 0 || ./bind-9.11.0/lib/dns/resolver.c-2182- ISFORWARDER(query->addrinfo)) ./bind-9.11.0/lib/dns/resolver.c-2183- fctx->qmessage->flags |= DNS_MESSAGEFLAG_RD; -- ./bind-9.2.0/lib/dns/resolver.c-910- /* ./bind-9.2.0/lib/dns/resolver.c:911: * Set RD if the client has requested that we do a recursive query, ./bind-9.2.0/lib/dns/resolver.c-912- * or if we're sending to a forwarder. ./bind-9.2.0/lib/dns/resolver.c-913- */ ./bind-9.2.0/lib/dns/resolver.c-914- if ((query->options & DNS_FETCHOPT_RECURSIVE) != 0 || ./bind-9.2.0/lib/dns/resolver.c-915- ISFORWARDER(query->addrinfo)) ./bind-9.2.0/lib/dns/resolver.c-916- fctx->qmessage->flags |= DNS_MESSAGEFLAG_RD; -- ./bind-9.3.0/lib/dns/resolver.c-1111- /* ./bind-9.3.0/lib/dns/resolver.c:1112: * Set RD if the client has requested that we do a recursive query, ./bind-9.3.0/lib/dns/resolver.c-1113- * or if we're sending to a forwarder. ./bind-9.3.0/lib/dns/resolver.c-1114- */ ./bind-9.3.0/lib/dns/resolver.c-1115- if ((query->options & DNS_FETCHOPT_RECURSIVE) != 0 || ./bind-9.3.0/lib/dns/resolver.c-1116- ISFORWARDER(query->addrinfo)) ./bind-9.3.0/lib/dns/resolver.c-1117- fctx->qmessage->flags |= DNS_MESSAGEFLAG_RD; -- ./bind-9.4.0/lib/dns/resolver.c-1355- /* ./bind-9.4.0/lib/dns/resolver.c:1356: * Set RD if the client has requested that we do a recursive query, ./bind-9.4.0/lib/dns/resolver.c-1357- * or if we're sending to a forwarder. ./bind-9.4.0/lib/dns/resolver.c-1358- */ ./bind-9.4.0/lib/dns/resolver.c-1359- if ((query->options & DNS_FETCHOPT_RECURSIVE) != 0 || ./bind-9.4.0/lib/dns/resolver.c-1360- ISFORWARDER(query->addrinfo)) ./bind-9.4.0/lib/dns/resolver.c-1361- fctx->qmessage->flags |= DNS_MESSAGEFLAG_RD; -- ./bind-9.5.0/lib/dns/resolver.c-1476- /* ./bind-9.5.0/lib/dns/resolver.c:1477: * Set RD if the client has requested that we do a recursive query, ./bind-9.5.0/lib/dns/resolver.c-1478- * or if we're sending to a forwarder. ./bind-9.5.0/lib/dns/resolver.c-1479- */ ./bind-9.5.0/lib/dns/resolver.c-1480- if ((query->options & DNS_FETCHOPT_RECURSIVE) != 0 || ./bind-9.5.0/lib/dns/resolver.c-1481- ISFORWARDER(query->addrinfo)) ./bind-9.5.0/lib/dns/resolver.c-1482- fctx->qmessage->flags |= DNS_MESSAGEFLAG_RD; -- ./bind-9.6.0/lib/dns/resolver.c-1510- /* ./bind-9.6.0/lib/dns/resolver.c:1511: * Set RD if the client has requested that we do a recursive query, ./bind-9.6.0/lib/dns/resolver.c-1512- * or if we're sending to a forwarder. ./bind-9.6.0/lib/dns/resolver.c-1513- */ ./bind-9.6.0/lib/dns/resolver.c-1514- if ((query->options & DNS_FETCHOPT_RECURSIVE) != 0 || ./bind-9.6.0/lib/dns/resolver.c-1515- ISFORWARDER(query->addrinfo)) ./bind-9.6.0/lib/dns/resolver.c-1516- fctx->qmessage->flags |= DNS_MESSAGEFLAG_RD; -- ./bind-9.7.0/lib/dns/resolver.c-1681- /* ./bind-9.7.0/lib/dns/resolver.c:1682: * Set RD if the client has requested that we do a recursive query, ./bind-9.7.0/lib/dns/resolver.c-1683- * or if we're sending to a forwarder. ./bind-9.7.0/lib/dns/resolver.c-1684- */ ./bind-9.7.0/lib/dns/resolver.c-1685- if ((query->options & DNS_FETCHOPT_RECURSIVE) != 0 || ./bind-9.7.0/lib/dns/resolver.c-1686- ISFORWARDER(query->addrinfo)) ./bind-9.7.0/lib/dns/resolver.c-1687- fctx->qmessage->flags |= DNS_MESSAGEFLAG_RD; -- ./bind-9.8.0/lib/dns/resolver.c-1741- /* ./bind-9.8.0/lib/dns/resolver.c:1742: * Set RD if the client has requested that we do a recursive query, ./bind-9.8.0/lib/dns/resolver.c-1743- * or if we're sending to a forwarder. ./bind-9.8.0/lib/dns/resolver.c-1744- */ ./bind-9.8.0/lib/dns/resolver.c-1745- if ((query->options & DNS_FETCHOPT_RECURSIVE) != 0 || ./bind-9.8.0/lib/dns/resolver.c-1746- ISFORWARDER(query->addrinfo)) ./bind-9.8.0/lib/dns/resolver.c-1747- fctx->qmessage->flags |= DNS_MESSAGEFLAG_RD; -- ./bind-9.9.0/lib/dns/resolver.c-1750- /* ./bind-9.9.0/lib/dns/resolver.c:1751: * Set RD if the client has requested that we do a recursive query, ./bind-9.9.0/lib/dns/resolver.c-1752- * or if we're sending to a forwarder. ./bind-9.9.0/lib/dns/resolver.c-1753- */ ./bind-9.9.0/lib/dns/resolver.c-1754- if ((query->options & DNS_FETCHOPT_RECURSIVE) != 0 || ./bind-9.9.0/lib/dns/resolver.c-1755- ISFORWARDER(query->addrinfo)) ./bind-9.9.0/lib/dns/resolver.c-1756- fctx->qmessage->flags |= DNS_MESSAGEFLAG_RD;
$ grep -rn DNS_FETCHOPT_RECURSIVE .
./bind-9.0.0/lib/dns/include/dns/resolver.h:89:#define DNS_FETCHOPT_RECURSIVE 0x04 /* Set RD? */ ./bind-9.0.0/lib/dns/resolver.c:882: if ((query->options & DNS_FETCHOPT_RECURSIVE) != 0 || ./bind-9.1.0/lib/dns/include/dns/resolver.h:89:#define DNS_FETCHOPT_RECURSIVE 0x04 /* Set RD? */ ./bind-9.1.0/lib/dns/resolver.c:900: if ((query->options & DNS_FETCHOPT_RECURSIVE) != 0 || ./bind-9.10.0/lib/dns/include/dns/resolver.h:92:#define DNS_FETCHOPT_RECURSIVE 0x004 /*%< Set RD? */ ./bind-9.10.0/lib/dns/resolver.c:1906: if ((query->options & DNS_FETCHOPT_RECURSIVE) != 0 || ./bind-9.11.0/lib/dns/include/dns/resolver.h:90:#define DNS_FETCHOPT_RECURSIVE 0x004 /*%< Set RD? */ ./bind-9.11.0/lib/dns/resolver.c:2181: if ((query->options & DNS_FETCHOPT_RECURSIVE) != 0 || ./bind-9.2.0/lib/dns/include/dns/resolver.h:89:#define DNS_FETCHOPT_RECURSIVE 0x04 /* Set RD? */ ./bind-9.2.0/lib/dns/resolver.c:914: if ((query->options & DNS_FETCHOPT_RECURSIVE) != 0 || ./bind-9.3.0/lib/dns/include/dns/resolver.h:89:#define DNS_FETCHOPT_RECURSIVE 0x04 /* Set RD? */ ./bind-9.3.0/lib/dns/resolver.c:1115: if ((query->options & DNS_FETCHOPT_RECURSIVE) != 0 || ./bind-9.4.0/lib/dns/include/dns/resolver.h:91:#define DNS_FETCHOPT_RECURSIVE 0x04 /*%< Set RD? */ ./bind-9.4.0/lib/dns/resolver.c:1359: if ((query->options & DNS_FETCHOPT_RECURSIVE) != 0 || ./bind-9.5.0/lib/dns/include/dns/resolver.h:91:#define DNS_FETCHOPT_RECURSIVE 0x04 /*%< Set RD? */ ./bind-9.5.0/lib/dns/resolver.c:1480: if ((query->options & DNS_FETCHOPT_RECURSIVE) != 0 || ./bind-9.6.0/lib/dns/include/dns/resolver.h:91:#define DNS_FETCHOPT_RECURSIVE 0x04 /*%< Set RD? */ ./bind-9.6.0/lib/dns/resolver.c:1514: if ((query->options & DNS_FETCHOPT_RECURSIVE) != 0 || ./bind-9.7.0/lib/dns/include/dns/resolver.h:92:#define DNS_FETCHOPT_RECURSIVE 0x04 /*%< Set RD? */ ./bind-9.7.0/lib/dns/resolver.c:1685: if ((query->options & DNS_FETCHOPT_RECURSIVE) != 0 || ./bind-9.8.0/lib/dns/include/dns/resolver.h:92:#define DNS_FETCHOPT_RECURSIVE 0x04 /*%< Set RD? */ ./bind-9.8.0/lib/dns/resolver.c:1745: if ((query->options & DNS_FETCHOPT_RECURSIVE) != 0 || ./bind-9.9.0/lib/dns/include/dns/resolver.h:92:#define DNS_FETCHOPT_RECURSIVE 0x04 /*%< Set RD? */ ./bind-9.9.0/lib/dns/resolver.c:1754: if ((query->options & DNS_FETCHOPT_RECURSIVE) != 0 ||
$ grep -rn DNS_MESSAGEFLAG_RD .
./bind-9.0.0/bin/dig/dig.c:369: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0) {
./bind-9.0.0/bin/dig/dighost.c:1050: lookup->sendmsg->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.0.0/bin/dig/host.c:464: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0) {
./bind-9.0.0/bin/named/client.c:1086: rd = ISC_TF((client->message->flags & DNS_MESSAGEFLAG_RD) != 0);
./bind-9.0.0/bin/named/query.c:2933: (message->flags & DNS_MESSAGEFLAG_RD) == 0) {
./bind-9.0.0/bin/nsupdate/nsupdate.c:1355: soaquery->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.0.0/bin/tests/dispatch_test.c:233: msg->flags = DNS_MESSAGEFLAG_RD;
./bind-9.0.0/bin/tests/printmsg.c:185: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0) {
./bind-9.0.0/lib/dns/include/dns/message.h:95:#define DNS_MESSAGEFLAG_RD 0x0100U
./bind-9.0.0/lib/dns/include/dns/message.h:100:#define DNS_MESSAGE_REPLYPRESERVE (DNS_MESSAGEFLAG_RD)
./bind-9.0.0/lib/dns/message.c:2836: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.0.0/lib/dns/resolver.c:884: fctx->qmessage->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.1.0/bin/dig/dig.c:388: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0) {
./bind-9.1.0/bin/dig/dighost.c:1386: lookup->sendmsg->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.1.0/bin/dig/host.c:463: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0) {
./bind-9.1.0/bin/named/query.c:3299: if ((message->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.1.0/bin/named/query.c:3315: (message->flags & DNS_MESSAGEFLAG_RD) == 0) {
./bind-9.1.0/bin/nsupdate/nsupdate.c:1524: soaquery->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.1.0/bin/tests/dispatch_test.c:233: msg->flags = DNS_MESSAGEFLAG_RD;
./bind-9.1.0/bin/tests/printmsg.c:185: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0) {
./bind-9.1.0/lib/dns/include/dns/message.h:93:#define DNS_MESSAGEFLAG_RD 0x0100U
./bind-9.1.0/lib/dns/include/dns/message.h:100:#define DNS_MESSAGE_REPLYPRESERVE (DNS_MESSAGEFLAG_RD)
./bind-9.1.0/lib/dns/message.c:2891: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.1.0/lib/dns/resolver.c:902: fctx->qmessage->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.10.0/bin/dig/dig.c:561: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.10.0/bin/dig/dig.c:580: (msg->flags & DNS_MESSAGEFLAG_RD) != 0 &&
./bind-9.10.0/bin/dig/dighost.c:2286: lookup->sendmsg->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.10.0/bin/dig/host.c:516: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0) {
./bind-9.10.0/bin/named/client.c:2027: client->message->flags &= ~DNS_MESSAGEFLAG_RD;
./bind-9.10.0/bin/named/client.c:2129: !((client->message->flags & DNS_MESSAGEFLAG_RD)
./bind-9.10.0/bin/named/query.c:8196: if ((message->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.10.0/bin/named/query.c:8215: (message->flags & DNS_MESSAGEFLAG_RD) == 0) {
./bind-9.10.0/bin/nsupdate/nsupdate.c:2909: soaquery->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.10.0/bin/tests/printmsg.c:182: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.10.0/lib/dns/client.c:2428: soaquery->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.10.0/lib/dns/include/dns/message.h:98:#define DNS_MESSAGEFLAG_RD 0x0100U
./bind-9.10.0/lib/dns/include/dns/message.h:116:#define DNS_MESSAGE_REPLYPRESERVE (DNS_MESSAGEFLAG_RD|DNS_MESSAGEFLAG_CD)
./bind-9.10.0/lib/dns/message.c:3430: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.10.0/lib/dns/resolver.c:1908: fctx->qmessage->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.10.0/lib/dns/resolver.c:1921: ((fctx->qmessage->flags & DNS_MESSAGEFLAG_RD) != 0)) {
./bind-9.11.0/bin/dig/dig.c:559: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.11.0/bin/dig/dig.c:578: (msg->flags & DNS_MESSAGEFLAG_RD) != 0 &&
./bind-9.11.0/bin/dig/dighost.c:2422: lookup->sendmsg->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.11.0/bin/dig/host.c:508: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0) {
./bind-9.11.0/bin/named/client.c:1174: if ((client->message->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.11.0/bin/named/client.c:2459: client->message->flags &= ~DNS_MESSAGEFLAG_RD;
./bind-9.11.0/bin/named/client.c:2591: (client->message->flags & DNS_MESSAGEFLAG_RD) == 0))
./bind-9.11.0/bin/named/client.c:2779: if ((client->message->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.11.0/bin/named/query.c:9045: if ((message->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.11.0/bin/named/query.c:9062: if ((message->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.11.0/bin/named/query.c:9077: (message->flags & DNS_MESSAGEFLAG_RD) == 0) {
./bind-9.11.0/bin/nsupdate/nsupdate.c:3085: soaquery->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.11.0/bin/tests/system/pipelined/pipequeries.c:152: message->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.11.0/bin/tools/mdig.c:306: if ((response->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.11.0/bin/tools/mdig.c:324: if ((response->flags & DNS_MESSAGEFLAG_RD) != 0 &&
./bind-9.11.0/bin/tools/mdig.c:534: message->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.11.0/contrib/dnsperf-2.1.0.0-1/dns.c:817: flags = DNS_MESSAGEFLAG_RD;
./bind-9.11.0/lib/dns/client.c:2455: soaquery->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.11.0/lib/dns/include/dns/message.h:87:#define DNS_MESSAGEFLAG_RD 0x0100U
./bind-9.11.0/lib/dns/include/dns/message.h:107:#define DNS_MESSAGE_REPLYPRESERVE (DNS_MESSAGEFLAG_RD|DNS_MESSAGEFLAG_CD)
./bind-9.11.0/lib/dns/message.c:3552: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.11.0/lib/dns/resolver.c:2183: fctx->qmessage->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.11.0/lib/dns/resolver.c:2196: ((fctx->qmessage->flags & DNS_MESSAGEFLAG_RD) != 0))
./bind-9.11.0/lib/dns/resolver.c:2577: if ((fctx->qmessage->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.11.0/lib/dns/resolver.c:7876: if ((fctx->qmessage->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.2.0/bin/dig/dig.c:402: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.2.0/bin/dig/dighost.c:1292: lookup->sendmsg->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.2.0/bin/dig/host.c:459: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0) {
./bind-9.2.0/bin/named/client.c:1313: !((flags & DNS_MESSAGEFLAG_RD) == 0 &&
./bind-9.2.0/bin/named/query.c:3293: if ((message->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.2.0/bin/named/query.c:3313: (message->flags & DNS_MESSAGEFLAG_RD) == 0) {
./bind-9.2.0/bin/nsupdate/nsupdate.c:1781: soaquery->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.2.0/bin/tests/printmsg.c:185: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0) {
./bind-9.2.0/lib/dns/include/dns/message.h:95:#define DNS_MESSAGEFLAG_RD 0x0100U
./bind-9.2.0/lib/dns/include/dns/message.h:102:#define DNS_MESSAGE_REPLYPRESERVE (DNS_MESSAGEFLAG_RD)
./bind-9.2.0/lib/dns/message.c:2965: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.2.0/lib/dns/resolver.c:916: fctx->qmessage->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.3.0/bin/dig/dig.c:488: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.3.0/bin/dig/dighost.c:1692: lookup->sendmsg->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.3.0/bin/dig/host.c:452: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0) {
./bind-9.3.0/bin/named/client.c:1297: client->message->flags &= ~DNS_MESSAGEFLAG_RD;
./bind-9.3.0/bin/named/client.c:1426: !((client->message->flags & DNS_MESSAGEFLAG_RD)
./bind-9.3.0/bin/named/query.c:3398: if ((message->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.3.0/bin/named/query.c:3417: (message->flags & DNS_MESSAGEFLAG_RD) == 0) {
./bind-9.3.0/bin/nsupdate/nsupdate.c:1877: soaquery->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.3.0/bin/tests/printmsg.c:185: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0) {
./bind-9.3.0/lib/dns/include/dns/message.h:95:#define DNS_MESSAGEFLAG_RD 0x0100U
./bind-9.3.0/lib/dns/include/dns/message.h:102:#define DNS_MESSAGE_REPLYPRESERVE (DNS_MESSAGEFLAG_RD|DNS_MESSAGEFLAG_CD)
./bind-9.3.0/lib/dns/message.c:3048: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.3.0/lib/dns/resolver.c:1117: fctx->qmessage->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.4.0/bin/dig/dig.c:479: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.4.0/bin/dig/dig.c:496: (msg->flags & DNS_MESSAGEFLAG_RD) != 0 &&
./bind-9.4.0/bin/dig/dighost.c:1917: lookup->sendmsg->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.4.0/bin/dig/host.c:475: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0) {
./bind-9.4.0/bin/named/client.c:1471: client->message->flags &= ~DNS_MESSAGEFLAG_RD;
./bind-9.4.0/bin/named/client.c:1600: !((client->message->flags & DNS_MESSAGEFLAG_RD)
./bind-9.4.0/bin/named/query.c:4439: if ((message->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.4.0/bin/named/query.c:4458: (message->flags & DNS_MESSAGEFLAG_RD) == 0) {
./bind-9.4.0/bin/nsupdate/nsupdate.c:2046: soaquery->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.4.0/bin/tests/printmsg.c:185: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0) {
./bind-9.4.0/lib/dns/include/dns/message.h:99:#define DNS_MESSAGEFLAG_RD 0x0100U
./bind-9.4.0/lib/dns/include/dns/message.h:106:#define DNS_MESSAGE_REPLYPRESERVE (DNS_MESSAGEFLAG_RD|DNS_MESSAGEFLAG_CD)
./bind-9.4.0/lib/dns/message.c:3144: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.4.0/lib/dns/resolver.c:1361: fctx->qmessage->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.5.0/bin/dig/dig.c:480: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.5.0/bin/dig/dig.c:497: (msg->flags & DNS_MESSAGEFLAG_RD) != 0 &&
./bind-9.5.0/bin/dig/dighost.c:1940: lookup->sendmsg->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.5.0/bin/dig/host.c:489: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0) {
./bind-9.5.0/bin/named/client.c:1555: client->message->flags &= ~DNS_MESSAGEFLAG_RD;
./bind-9.5.0/bin/named/client.c:1709: !((client->message->flags & DNS_MESSAGEFLAG_RD)
./bind-9.5.0/bin/named/query.c:4495: if ((message->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.5.0/bin/named/query.c:4514: (message->flags & DNS_MESSAGEFLAG_RD) == 0) {
./bind-9.5.0/bin/nsupdate/nsupdate.c:2530: soaquery->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.5.0/bin/tests/printmsg.c:185: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0) {
./bind-9.5.0/lib/dns/include/dns/message.h:99:#define DNS_MESSAGEFLAG_RD 0x0100U
./bind-9.5.0/lib/dns/include/dns/message.h:110:#define DNS_MESSAGE_REPLYPRESERVE (DNS_MESSAGEFLAG_RD|DNS_MESSAGEFLAG_CD)
./bind-9.5.0/lib/dns/message.c:3258: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.5.0/lib/dns/resolver.c:1482: fctx->qmessage->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.6.0/bin/dig/dig.c:481: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.6.0/bin/dig/dig.c:498: (msg->flags & DNS_MESSAGEFLAG_RD) != 0 &&
./bind-9.6.0/bin/dig/dighost.c:1940: lookup->sendmsg->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.6.0/bin/dig/host.c:489: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0) {
./bind-9.6.0/bin/named/client.c:1558: client->message->flags &= ~DNS_MESSAGEFLAG_RD;
./bind-9.6.0/bin/named/client.c:1712: !((client->message->flags & DNS_MESSAGEFLAG_RD)
./bind-9.6.0/bin/named/query.c:4934: if ((message->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.6.0/bin/named/query.c:4953: (message->flags & DNS_MESSAGEFLAG_RD) == 0) {
./bind-9.6.0/bin/nsupdate/nsupdate.c:2573: soaquery->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.6.0/bin/tests/printmsg.c:185: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0) {
./bind-9.6.0/lib/dns/include/dns/message.h:99:#define DNS_MESSAGEFLAG_RD 0x0100U
./bind-9.6.0/lib/dns/include/dns/message.h:110:#define DNS_MESSAGE_REPLYPRESERVE (DNS_MESSAGEFLAG_RD|DNS_MESSAGEFLAG_CD)
./bind-9.6.0/lib/dns/message.c:3266: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.6.0/lib/dns/resolver.c:1516: fctx->qmessage->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.7.0/bin/dig/dig.c:503: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.7.0/bin/dig/dig.c:520: (msg->flags & DNS_MESSAGEFLAG_RD) != 0 &&
./bind-9.7.0/bin/dig/dighost.c:2139: lookup->sendmsg->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.7.0/bin/dig/host.c:509: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0) {
./bind-9.7.0/bin/named/client.c:1568: client->message->flags &= ~DNS_MESSAGEFLAG_RD;
./bind-9.7.0/bin/named/client.c:1718: !((client->message->flags & DNS_MESSAGEFLAG_RD)
./bind-9.7.0/bin/named/query.c:5167: if ((message->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.7.0/bin/named/query.c:5186: (message->flags & DNS_MESSAGEFLAG_RD) == 0) {
./bind-9.7.0/bin/nsupdate/nsupdate.c:2683: soaquery->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.7.0/bin/tests/printmsg.c:185: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0) {
./bind-9.7.0/lib/dns/client.c:2290: soaquery->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.7.0/lib/dns/include/dns/message.h:98:#define DNS_MESSAGEFLAG_RD 0x0100U
./bind-9.7.0/lib/dns/include/dns/message.h:109:#define DNS_MESSAGE_REPLYPRESERVE (DNS_MESSAGEFLAG_RD|DNS_MESSAGEFLAG_CD)
./bind-9.7.0/lib/dns/message.c:3315: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.7.0/lib/dns/resolver.c:1687: fctx->qmessage->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.8.0/bin/dig/dig.c:508: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.8.0/bin/dig/dig.c:527: (msg->flags & DNS_MESSAGEFLAG_RD) != 0 &&
./bind-9.8.0/bin/dig/dighost.c:2139: lookup->sendmsg->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.8.0/bin/dig/host.c:509: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0) {
./bind-9.8.0/bin/named/client.c:1568: client->message->flags &= ~DNS_MESSAGEFLAG_RD;
./bind-9.8.0/bin/named/client.c:1718: !((client->message->flags & DNS_MESSAGEFLAG_RD)
./bind-9.8.0/bin/named/query.c:6908: if ((message->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.8.0/bin/named/query.c:6927: (message->flags & DNS_MESSAGEFLAG_RD) == 0) {
./bind-9.8.0/bin/nsupdate/nsupdate.c:2797: soaquery->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.8.0/bin/tests/printmsg.c:185: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0) {
./bind-9.8.0/lib/dns/client.c:2300: soaquery->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.8.0/lib/dns/include/dns/message.h:98:#define DNS_MESSAGEFLAG_RD 0x0100U
./bind-9.8.0/lib/dns/include/dns/message.h:109:#define DNS_MESSAGE_REPLYPRESERVE (DNS_MESSAGEFLAG_RD|DNS_MESSAGEFLAG_CD)
./bind-9.8.0/lib/dns/message.c:3334: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.8.0/lib/dns/resolver.c:1747: fctx->qmessage->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.9.0/bin/dig/dig.c:522: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.9.0/bin/dig/dig.c:541: (msg->flags & DNS_MESSAGEFLAG_RD) != 0 &&
./bind-9.9.0/bin/dig/dighost.c:2166: lookup->sendmsg->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.9.0/bin/dig/host.c:509: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0) {
./bind-9.9.0/bin/named/client.c:1569: client->message->flags &= ~DNS_MESSAGEFLAG_RD;
./bind-9.9.0/bin/named/client.c:1719: !((client->message->flags & DNS_MESSAGEFLAG_RD)
./bind-9.9.0/bin/named/query.c:7230: if ((message->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.9.0/bin/named/query.c:7249: (message->flags & DNS_MESSAGEFLAG_RD) == 0) {
./bind-9.9.0/bin/nsupdate/nsupdate.c:2840: soaquery->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.9.0/bin/tests/printmsg.c:182: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.9.0/lib/dns/client.c:2304: soaquery->flags |= DNS_MESSAGEFLAG_RD;
./bind-9.9.0/lib/dns/include/dns/message.h:98:#define DNS_MESSAGEFLAG_RD 0x0100U
./bind-9.9.0/lib/dns/include/dns/message.h:109:#define DNS_MESSAGE_REPLYPRESERVE (DNS_MESSAGEFLAG_RD|DNS_MESSAGEFLAG_CD)
./bind-9.9.0/lib/dns/message.c:3334: if ((msg->flags & DNS_MESSAGEFLAG_RD) != 0)
./bind-9.9.0/lib/dns/resolver.c:1756: fctx->qmessage->flags |= DNS_MESSAGEFLAG_RD;
Possible objections
OBJECTION #1: What about Camp’s own DNS log on her site?
Camp’s DNS log has almost certainly been derived from the Tea Leaves logs and is therefore not a corroborative log.
Camp’s log appears in a different format. Here’s a sample:
As you can see, the dates and times are terribly out of order, defeating casual correlation analysis on the part of the typical reporter, who may assume the log comes from an independent sensor. This is very odd because one of the main advantages of the ISO 8601 date and time format is its lexicographical sortability, meaning this log could have been presented in sorted order with one simple command:
$ sort Log-Of-DNS-Lookups-For-mail1.trump-email.com-851.txt 2016-05-04T10:48:06.000Z|mail1.trump-email.com|217.12.97.15 2016-05-06T11:46:32.000Z|mail1.trump-email.com|217.12.97.15 2016-05-06T20:27:30.000Z|mail1.trump-email.com|217.12.96.15 2016-05-10T02:31:32.000Z|mail1.trump-email.com|217.12.96.15 2016-05-10T04:50:20.000Z|mail1.trump-email.com|167.73.110.8 2016-05-10T20:12:48.000Z|mail1.trump-email.com|167.73.110.8 2016-05-11T19:38:40.000Z|mail1.trump-email.com|217.12.96.15 2016-05-12T11:02:14.000Z|mail1.trump-email.com|217.12.97.15 2016-05-14T06:06:17.000Z|mail1.trump-email.com|217.12.97.15 2016-05-14T08:08:05.000Z|mail1.trump-email.com|217.12.97.15 [...]
99% of the lines in the Camp log are for mail1.trump-email.com. We’ll grab those lines from the sorted log and store them into camp.txt:
$ sort Log-Of-DNS-Lookups-For-mail1.trump-email.com-851.txt | fgrep mail1.trump-email.com > camp.txt
With a similar sequence for ease of understanding, we’ll run the three Tea Leaves logs through the Python conversion script I’ve given below, sort the output, grab the mail1.trump-email.com lines, and store them into tealeaves.txt:
$ ./script.py ns[1-3]_cdcservices_com.log | sort | fgrep mail1.trump-email.com > tealeaves.txt
#!/usr/bin/env python3
from datetime import datetime, timedelta
import fileinput
for line in fileinput.input():
L = line.split()
ts, ip = L[0]+L[1], L[3]
ts = datetime.strptime(ts, '%d-%b-%Y%X') # default 'C' locale
ts = (ts + timedelta(hours=4)).isoformat() + '.000Z'
print('|'.join((ts, 'mail1.trump-email.com', ip)))
The principal operation is converting the Tea Leaves timestamps from UTC−04:00 to UTC and then presenting them in the relevant ISO 8601 format. (Incidentally, EDT is UTC−04:00, a zone applying both to where Camp’s crew is in Indiana and to the three American nameservers’ former location of southern Florida.)
The next problem is that the Camp log may have been produced using a broken de-duplication algorithm for identical lines. If the Tea Leaves logs have 2n identical lines, the Camp log will, except in a few stray instances, reduce that to n identical lines. Observe how, on a typical day, the Camp counts are half the Tea Leaves counts:
$ uniq -c camp.txt > camp-unique-count.txt $ uniq -c tealeaves.txt > tealeaves-unique-count.txt $ grep 2016-07-11 tealeaves-unique-count.txt camp-unique-count.txt tealeaves-unique-count.txt: 4 2016-07-11T00:46:11.000Z|mail1.trump-email.com|217.12.96.15 tealeaves-unique-count.txt: 2 2016-07-11T01:54:12.000Z|mail1.trump-email.com|217.12.96.15 tealeaves-unique-count.txt: 4 2016-07-11T02:15:51.000Z|mail1.trump-email.com|167.73.110.8 tealeaves-unique-count.txt: 2 2016-07-11T03:12:34.000Z|mail1.trump-email.com|217.12.97.15 tealeaves-unique-count.txt: 2 2016-07-11T03:16:53.000Z|mail1.trump-email.com|167.73.110.8 tealeaves-unique-count.txt: 2 2016-07-11T04:15:37.000Z|mail1.trump-email.com|167.73.110.8 tealeaves-unique-count.txt: 4 2016-07-11T07:36:13.000Z|mail1.trump-email.com|217.12.96.15 tealeaves-unique-count.txt: 14 2016-07-11T11:22:05.000Z|mail1.trump-email.com|167.73.110.8 tealeaves-unique-count.txt: 8 2016-07-11T12:08:24.000Z|mail1.trump-email.com|217.12.97.15 tealeaves-unique-count.txt: 2 2016-07-11T14:27:58.000Z|mail1.trump-email.com|167.73.110.8 tealeaves-unique-count.txt: 2 2016-07-11T15:23:55.000Z|mail1.trump-email.com|217.12.96.15 tealeaves-unique-count.txt: 2 2016-07-11T17:39:20.000Z|mail1.trump-email.com|217.12.96.15 tealeaves-unique-count.txt: 2 2016-07-11T19:29:05.000Z|mail1.trump-email.com|217.12.97.15 tealeaves-unique-count.txt: 4 2016-07-11T22:22:49.000Z|mail1.trump-email.com|217.12.97.15 tealeaves-unique-count.txt: 2 2016-07-11T23:37:03.000Z|mail1.trump-email.com|167.73.110.8 camp-unique-count.txt: 2 2016-07-11T00:46:11.000Z|mail1.trump-email.com|217.12.96.15 camp-unique-count.txt: 1 2016-07-11T01:54:12.000Z|mail1.trump-email.com|217.12.96.15 camp-unique-count.txt: 2 2016-07-11T02:15:51.000Z|mail1.trump-email.com|167.73.110.8 camp-unique-count.txt: 1 2016-07-11T03:12:34.000Z|mail1.trump-email.com|217.12.97.15 camp-unique-count.txt: 1 2016-07-11T03:16:53.000Z|mail1.trump-email.com|167.73.110.8 camp-unique-count.txt: 1 2016-07-11T04:15:37.000Z|mail1.trump-email.com|167.73.110.8 camp-unique-count.txt: 2 2016-07-11T07:36:13.000Z|mail1.trump-email.com|217.12.96.15 camp-unique-count.txt: 7 2016-07-11T11:22:05.000Z|mail1.trump-email.com|167.73.110.8 camp-unique-count.txt: 4 2016-07-11T12:08:24.000Z|mail1.trump-email.com|217.12.97.15 camp-unique-count.txt: 1 2016-07-11T14:27:58.000Z|mail1.trump-email.com|167.73.110.8 camp-unique-count.txt: 1 2016-07-11T15:23:55.000Z|mail1.trump-email.com|217.12.96.15 camp-unique-count.txt: 1 2016-07-11T17:39:20.000Z|mail1.trump-email.com|217.12.96.15 camp-unique-count.txt: 1 2016-07-11T19:29:05.000Z|mail1.trump-email.com|217.12.97.15 camp-unique-count.txt: 2 2016-07-11T22:22:49.000Z|mail1.trump-email.com|217.12.97.15 camp-unique-count.txt: 1 2016-07-11T23:37:03.000Z|mail1.trump-email.com|167.73.110.
Along with the fact that the Camp log starts at the exact same time in May and cuts off early, the one-way loss of information (does a 1-count line in the Camp log correspond to a 2-count line in the Tea Leaves logs, or does it correspond to another 1-count line?) strongly suggests that the Camp log is merely derivative. In any event, to show that this log adds nothing to the discussion, observe the perfect (timestamp, qname, IP) correlation we obtain when we strip out duplicates (except for a stray final line probably resulting from broken processing):
$ uniq camp.txt > camp-uniques.txt
$ uniq tealeaves.txt > tealeaves-uniques.txt
$ wc -l camp-uniques.txt
704 camp-uniques.txt
$ head -n 704 tealeaves-uniques.txt > tealeaves-uniques-first704.txt
$ diff camp-uniques.txt tealeaves-uniques-first704.txt
704c704
< 2016-07-29T15:33:41.000Z|mail1.trump-email.com|217.12.96.15
---
> 2016-07-29T15:33:40.000Z|mail1.trump-email.com|217.12.96.15
$ grep -A1 2016-07-29T15:33:40 tealeaves-uniques.txt
2016-07-29T15:33:40.000Z|mail1.trump-email.com|217.12.96.15
2016-07-29T15:33:41.000Z|mail1.trump-email.com|217.12.96.15
OBJECTION #2: If they were going to fabricate DNS logs, why not go the full distance and fabricate something more damning?
Unlike with hacker dumps, researcher dumps enjoy free passes for numerous smell tests. It’s easy to imagine some hacker fabricating textual data, but we tend to give researchers the benefit of the doubt. How’s this relevant to the objection? Well, the cost of going the researcher route is that there’s not actually a whole lot a researcher can release with a plausible background story. DNS logs are pretty much already pushing the limit.
OBJECTION #3: You’re not seriously suggesting they’d risk their reputations by fabricating logs?
I’m not asserting it outright (yet), but as should be obvious by now, I’m definitely not ruling out the possibility. I present this post in the form of a question. It provides some fresh data points and analysis for people to bear in mind as events unfold in the near future.
We must remember that the 2016 election was one in which countless hysterical types thought they were trying to stop the next Hitler (and couldn’t count on the Russians, for this time around the Russians were working with the next Hitler). We saw numerous advocacy groups and numerous left-wing icons (such as Bernie Sanders, Elizabeth Warren, Noam Chomsky, and Michael Moore) pissing away their credibility for the Clinton machine, probably figuring they’d pick up the pieces later (later never came). We saw numerous incidents blamed on Trump supporters that turned out to be false flags in hindsight (unfortunately, hindsight exists outside the latest news cycle). We saw social-justice and identity-politics hypocrisy at stratospheric levels, with the Clintons getting a free pass on everything — one popular feminist outlet went so far as to release an article saying that Bill Clinton “could very well” have been a rapist but that it “doesn’t even necessarily make him a bad feminist” once he stops raping women.
When I see the same 24/7 hysteria and painful lack of self-awareness in these researchers’ tweets and posts, I can’t help but consider the possibility they’re doing all they can (literally) to stop (literally) the next Hitler (literally). Can you really blame me?
OBJECTION #4: What about all those experts and computer scientists who vouched for the logs?
Well, except in the Paul Vixie case, it’s not even clear they examined any logs whatsoever. Paul Vixie mentioned only two words: “interpacket gaps.” He’s probably referring to the fact that the timestamps in the logs are patterned around a 3600 TTL, which I can confirm was the TTL for mail1.trump-email.com’s ‘A’ record (it’s a very common TTL for such records). I can also add that the putative Russian nameservers do not “lock on” to any of the three putative American nameservers (each being selected roughly 1/3 of the time), which is consistent with sRTT-based server selection for nameservers with similar RTTs (as was the case when the three nameservers were in southern Florida). To be sure, there are some oddities, but they’re too easily explained away as the result of unsynchronized clocks, packet delay, cache prefetching (BIND 9.10+), cache evictions, multiple stub resolvers, stub-resolver caching, etc., which is why I scrapped the discussion of two potential flaws discovered via interpacket analysis.
The trouble is, first, that some of the experts are now known not to be independent and, second, that the experts didn’t adjust their thinking to accommodate the fact they’re dealing with researchers who’ve written extensively on network deception. Botched DNS logs aside, these aren’t clueless laypeople posting DNS logs to some hand-holding tech forum. It’s very easy for researchers to repurpose DNS logs from passive collections or to set up a few nameserver instances in the lab with a network emulator in the middle (it happens in DNS research all the time because of the highly representative data flows obtained). At the extreme end, they could even start spoofing packets on the live Internet to trigger some passive collectors for corroboration. There’s absolutely no need to craft the logs by hand one line at a time, and the fact that forgery/manipulation was discussed in such a narrow and unimaginative context makes me wonder whether some folks are being willfully obtuse.
OBJECTION #5: What about the marketing email to Alfa-Bank?
I wrote this earlier:
I myself considered the matter a closed case when The Intercept published a couple of marketing emails sent from mail1.trump-email.com to the parties concerned. The Intercept article seemed to validate the DNS logs and debunk the secret-communication theory in one fell swoop. Even so, something kept bothering me about the fact that the emails bore pre-March timestamps (Cendyn claims the server sent its last marketing email in March 2016, and the DNS logs purport to show a May-September window of high-volume activity after an initial May drizzle). Something also kept bothering me about certain aspects of the DNS logs themselves, and after the usual news outlets tried reigniting the story recently, I decided it was time to take another look.
Although we shouldn’t assume the quotations in the CNN article are free of an agenda, and although they don’t rule out the high-volume DNS activity, I think they provide food for thought:
Alfa Bank and Mandiant could not point to marketing emails from the time period in question. “Mandiant has found evidence of an old marketing campaign, which … is too old to be relevant,” Alfa Bank said in a statement.
But Cendyn acknowledged that the last marketing email it delivered for Trump’s corporation was sent in March 2016, “well before the date range in question.”
Spectrum Health told CNN it “did find a small number of incoming spam marketing emails” from “Cendyn, advertising Trump Hotels.” But it pointed to emails sent in 2015, long before the May-through-September 2016 time period examined by scientists.
But Alfa Bank starkly denies “any dealings with Cendyn.” And, it says, it’s unlikely that it received any emails from that server. “Mandiant investigated 12 months of email archives and it found no emails to or from any of the IP addresses given to us by the media.”
It wouldn’t surprise me to learn that the average email-marketing server eventually sends an email to Alfa-Bank, even if only once or twice a year. Nor would it surprise me to learn that some hysterical individual with access to passive-DNS data has decided to piggyback off a legitimate but rare passive-collection association for extra cover when drumming up a narrative. I’ve certainly seen stranger things, and I have a bunch of manipulated logs in front of me, so what am I supposed to think?
OBJECTION #6: What if the logs were manipulated for benign reasons rather than as a fabrication effort?
This is ultimately the only objection I consider relevant now. The other objections don’t address the elephant in the room: the logs that have been clearly manipulated to look like BIND9 logs.
Conceivably, raw passive-DNS data could’ve been sloppily presented in a BIND9 pseudo-format for presentation, analysis, or whatever, making this a tremendously difficult objection to address using only public technical information. (Note, however, that the benign explanation that they had legitimate BIND9 logs and simplified them a little by removing some cruft doesn’t work, given the nature of the first and fourth flaws.)
I’ll point out that the DNS logs weren’t the only things manipulated to appear like something they’re not. Consider this information from the Tea Leaves WordPress site:
Click to enlarge
The directives ($ORIGIN, $TTL) and semicolons (comment tokens) show that this information is in the form of a DNS zone file that might exist on a master nameserver’s filesystem — the textual representation of the wire-format response to either an ‘ANY’ query or an ‘AXFR’ zone transfer will not appear in this configuration-style form. Moreover, the inconsistent use and spacing of the unnecessary semicolons suggest that this was crafted by hand rather than by a software tool.
Why is this important? Consider the dig tool, which Lorenzen has used extensively in her shell scripts and which we know she is therefore highly familiar with. Using the dig tool (or a related tool) for a couple of ‘ANY’ queries, I can pull such information from the {ns1, ns2, ns3}.cdcservices.com nameservers (information for mjh-email.com, a domain still served by those nameservers):
mjh-email.com. 3600 IN NS ns3.cdcservices.com. mjh-email.com. 3600 IN NS ns1.cdcservices.com. mjh-email.com. 3600 IN NS ns2.cdcservices.com. mjh-email.com. 3600 IN SOA ns1.cdcservices.com. postmaster.centralservices.local. 2012062512 1200 120 1209600 3600 mjh-email.com. 3600 IN MX 10 incoming.cdcservices.com. mjh-email.com. 600 IN TXT "v=spf1 ip4:198.91.42.0/23 ip4:64.135.26.0/24 ip4:64.95.241.0/24 ip4:206.191.130.0/24 ip4:63.251.151.0/24 ip4:69.25.15.0/24 mx ~all" mjh-email.com. 600 IN TXT "Internet Solution from Cendyn.com." mjh-email.com. 600 IN RP epresence.support\@cendyn.com. mjh-email.com. [...] mail1.mjh-email.com. 3600 IN A 66.216.133.36
It took me mere seconds to pull that information, copy it, and paste it. Why, then, would I go through the trouble of manually reformatting it to look like a DNS zone file for the sake of posting to WordPress? It’s not as if I’d be saving on the character count with the $ORIGIN directive (it might even sacrifice clarity), and it’s not as if the $TTL directive would be correct. Is there a conceivable reason that Tea Leaves did this other than making people believe that she has direct access to the master nameserver and that therefore her DNS logs are the real deal, totally unmanipulated? Did she perhaps do it to make people think her view of the DNS traffic hitting the nameservers is absolute rather than restricted to x number of passive feeds?
I’ll leave it to you fucking normies to judge whether fabrication has occurred.
Weaponized Autism 