Given: sin−1α+sin−1β+sin−1γ=π
Let, sin−1α=A,sin−1β=B,sin−1γ=C
⇒A+B+C=π
Also, α=sinA,β=sinB,γ=sinC...(i)
It is given that, (α+β)2−γ2=3αβ.
⇒α2+β2−γ2=αβ
⇒sin2A+sin2B−sin2C=sinAsinB
⇒sin2A+sin(B+C)sin(B−C)=sinAsinB
⇒sin2A+sin(π−A)sin(B−C)=sinAsinB
⇒sin2A+sinAsin(B−C)=sinAsinB
⇒sinA[sinA+sin(B−C)]=sinAsinB
⇒sinA[sinA+sin(B−C)]−sinAsinB=0
⇒sinA[sinA+sin(B−C)−sinB]=0
⇒sinA[sin(B+C)+sin(B−C)−sinB]=0
⇒sinA[2sinBcosC−sinB]=0
⇒sinAsinB[2cosC−1]=0
⇒sinA=0orsinB=0orcosC=21
It is given that, α,β,γ=0
⇒cosC=21
⇒sinC=23
⇒γ=23