How does lnd make sure it does not reuse the same route twice for a retry after the first payment failed?

In fact, finding a route in LND can be described as the process of finding the "shortest path" on a weighted graph. In LND, the weight of each channel is composed of three parts, fee, CLTV_DELTA and the probability. You'll understand the first two factors if you are familiar with the Lightning Network, and the last factor is the probability that this channel can afford the payments from a local perspective.

If a route fails, then the node with insufficient capacity will actively include this information in the return message, which can only be unlocked and seen by the payer. For example, Then the payer will reduce the probability of this channel in the local view and find the route again (you can treat it as setting the weight of this edge to very large).

You can find detailed information here.

So I went ahead and looked at the LND implementation here: https://github.com/lightningnetwork/lnd/blob/master/routing/pathfind.go and this is the comment block above the function findPath:

findPath attempts to find a path from the source node within the ChannelGraph to the target node that's capable of supporting a payment of amt value. The current approach implemented is modified version of Dijkstra's algorithm to find a single shortest path between the source node and the destination. The distance metric used for edges is related to the time-lock+fee costs along a particular edge. If a path is found, this function returns a slice of ChannelHop structs which encoded the chosen path from the target to the source. The search is performed backwards from destination node back to source. This is to properly accumulate fees that need to be paid along the path and accurately check the amount to forward at every node against the available bandwidth.

If this is the case then I don't think your scenario would play out. It appears LND wouldn't return the path A --> B --> C --> D because it does not have sufficient balance/bandwidth.