The Hardest Software Problem I have ever Solved

September 13, 2026

Sometimes it is only known in how many percent of the cases software fails, but not on which inputs.

Emails, DKIM, DMARC and Aggregate Reports

When emails are sent back and forth there is this thing called DKIM. The sender of the email publishes in DNS a public key. With the private key is signed the content of the email and the result is put in the DKIM-Signature email header. The recipient of the email can then validate that the one who announced the public key and the one who signed the email are the same entity.

DMARC goes one step further. It allows a domain owner to publish in DNS information, that all emails from that domain must have (valid) DKIM-Signature header. It prohibits misusing a domain for sending emails, without the consent of the domain owner. But before DMARC is utilized, first must be ensured that the DKIM processes work correctly and for this there are the aggregate reports.

The aggregate reports are emails sent in machine readable format from the receiving to the sending SMTP server. The reports contain information how many emails were received in the last 24 hours and how many of these emails contained valid DKIM-Signature.

The Problem

After I utilized the OpenDKIM software, the reports I received stated that almost all sent emails had correct DKIM-Signature. But not all messages! So I wanted to figure out for which messages adding DKIM-Signature failed.

I looked for memory corruption and mutli-threading anomalies at run-time, but could not find anything. I switched from OpenSSL to GnuTLS, nothing changed. I asked providers, which send the aggregate reports, to give me a sample, where the signature verification has failed, I received no answers.

I had no idea what to do and in fact over several years I have spent now and then time on this problem without success. I have also not filed a bug report, as the only relevant information I could provide was “in rare cases it does not work” - there were no reproducers.

The Solution

OpenDKIM is a milter - plugin for sendmail and postfix. These plugins receive all emails running over the MTA, and can register to obtain the full emails or parts of them (e.g. only the recipients).

At some moment I started logging the data, which the milter receives from the MTA⇔postfix⇔sendmail, and in addition logging the data, which the MTA sends to the internet. This happened some years ago, so I cannot say from today’s perspective why I created this double-logging. Then I compared what was received by the milter to what was sent to the internet as signed emails. And this approach turned out to be the key!

OpenDKIM can be used to validate in incoming emails if the DKIM-Signature headers are valid. It can also be used to append a new DKIM-Signature from the current host. Of course, it can do both actions at the same time. The behaviour is controlled by the Mode directive in opendkim.conf: signer, verifier or sv for both.

As the system was receiving emails from the internet to internal aliases, and redirecting the emails to external email hosts, I had Mode: sv. When OpenDKIM received an email without DKIM-Signature, and OpenDKIM was told to validate the signatures of the incoming emails, after the first 64kb transferred, OpenDKIM communicated, that it does not want to receive from the MTA the remaining parts of the email. But then, without having the complete email, the created DKIM-Signature header was invalid!

The fix at github.com/trusteddomainproject/OpenDKIM/issues/15 was in signer mode to accept all content of the email:

diff --git a/opendkim/opendkim.c b/opendkim/opendkim.c
@@ -13394,13 +13394,10 @@ mlfi_body(SMFICTX *ctx, u_char *bodyp, size_t bodylen)
                return dkimf_libstatus(ctx, last, "dkim_body()", status);
 
 #ifdef SMFIS_SKIP
-       if (dfc->mctx_srhead != NULL && cc->cctx_milterv2 &&
-           dkimf_msr_minbody(dfc->mctx_srhead) == 0)
-                       return SMFIS_SKIP;
-
-       if (dfc->mctx_dkimv != NULL && cc->cctx_milterv2 &&
-           dkim_minbody(dfc->mctx_dkimv) == 0)
-                       return SMFIS_SKIP;
+       if (cc->cctx_milterv2 &&
+           (dfc->mctx_srhead == NULL || dkimf_msr_minbody(dfc->mctx_srhead) == 0) &&
+           (dfc->mctx_dkimv == NULL || dkim_minbody(dfc->mctx_dkimv) == 0))
+               return SMFIS_SKIP;
 #endif /* SMFIS_SKIP */
 
        return SMFIS_CONTINUE;

And that’s it. It sounds no soo complicated, but in reality it took soo much time!

Unfortunately after I published the change in 2018, until now (2026) there is no tagged release of the OpenDKIM software, which includes this fix.