I won't get too far into the details, but a mixed-strategy Nash equilibrium in game theory is essentially the set of best strategies for each player in a game, given the other player's strategies. For instance, in soccer, the shooter shouldn't always kick to the right side of the goal, even if it's his strongest side, because the goalie would always dive to the same side and stop most of the shots. It may be the case, for example, that the shooter should kick right 5/7 of the time and left 2/7 of the time, while the goalie should dive right 3/4 of the time and left 1/4 of the time. If either player deviates from this strategy, he is worse off.
Dawkins applies this same idea to the evolution of genes for aggression:
An evolutionary stable strategy or ESS is defined as a strategy which, if most of the members of a population adopt it, cannot be bettered by an alternative strategy. It is a subtle and important idea. Another way of putting it is to say that the best strategy for an individual depends on what the majority of the population are doing. Since the rest of the population consists of individuals, each one trying to maximize his own success, the only strategy that persists will be one which, once employed, cannot be bettered by any deviant individual. Following a major environmental change there may be a brief period of evolutionary instability, perhaps even oscillation in the population. But once an ESS is achieved it will stay: selection will penalize deviation from it.Dawkins imagines a species in which individuals could act as either "doves" or "hawks" (not the actual birds, but the attitudes toward aggression). Doves never pick fights and always run away when threatened. Hawks always fight as hard as they can, only surrendering when seriously injured. Dawkins assigns some arbitrary payouts to each strategy and concludes that neither can completely dominate: a hawk has a huge advantage in a population full of doves, and vice versa. However, a fixed proportion of the two strategies will eventually emerge and hold stable, when the payoffs to being a dove or being a hawk are equal. In Dawkins' numerical example, equilibrium is reached when the population consists of 5/12 doves and 7/12 hawks.