Let, the roots of (m2+1)x2−3x+(m2+1)2=0 are α and β
We know that the sum and product of roots of a quadratic equation ax2+bx+c=0,a=0 are respectively −ab and ac.
∴α+β=m2+13 and αβ=(m2+1)(m2+1)2=(m2+1)
∵α+β is maximum, ∴m=0
∴ the equation is x2−3x+1=0
And α+β=3,αβ=1
Now, α3−β3=(α−β)(α2+β2+αβ)
=(α+β)2−4αβ((α+β)2−αβ)
=9−4(9−1)
=85.