Tuning EDNS0 retries?

On a busy unbound 1.6.2 server I observed the following sequence of events,
in which an initial query socket is closed quickly (for a retry with a
smaller EDNS0 buffer size) and ICMP unreachable is returned by the time the
answer arrives, with the retry answer finally accepted at the retry socket
60ms after the first answer, which was dropped.