If a nuclear power plant can buy 86% enriched fuel for $21000/lb and can buy cheaper less enriched uranium that is 80% pure and for $14000/lb still using as a mixer the 57% pure uranium at $5000/lb which fuel is going to be more cost effective for the plant?

If it's the pure content that counts, then the proportionate cost for the pure compound is:

for 86% : 21000/0.86=$24418.60 / lb.
for 80% : 14000/0.80=$17500.00 / lb.

The 80% material is more effective.