Equation x3+ax2+bx+c=0 has roots α,β,γ. Therefore,
α+β+γ=−a
αβ+βγ+γα=b
Since the given system of equations has non-trivial solutions,
we have
∣αβγβγαγαβ∣=0
or α3+β3+γ3−3αβγ=0
or (α+β+γ)[α2+β2+γ2−αβ−βγ−γα]=0
or (α+β+γ)[(α+β+γ)2−3(αβ+βγ+γα)]=0
⇒−a[a2−3b]=0
or a2/b=3