Let P=[ … ] and A=[ … ] where ω = -1+i√3 2, and I_3 be the identity matrix of order 3. If the determinant of the matrix (P^-1AP-I_3)^2 is α ω ^2,…

Let $M={({P}^{-1}AP-I)}^{2}$M=(P−1AP−I)2

$$={({P}^{-1}AP)}^{2}-2{P}^{-1}AP+I$$=(P−1AP)2−2P−1AP+I

$$={P}^{-1}{A}^{2}P-2{P}^{-1}AP+I(\because {({P}^{-1}AP)}^{2}=({P}^{-1}AP)({P}^{-1}AP))$$=P−1A2P−2P−1AP+I(∵(P−1AP)2=(P−1AP)(P−1AP))

Multiplying with $P$P on both sides

$$\Rightarrow PM={A}^{2}P-2AP+P$$⇒PM=A2P−2AP+P

$$=({A}^{2}-2A.I+{I}^{2})P$$=(A2−2A.I+I2)P

$$\Rightarrow Det(PM)=Det({(A-I)}^{2}\times P)$$⇒Det(PM)=Det((A−I)2×P)

$$\Rightarrow Det(PM)=Det{(A-I)}^{2}\times Det(P)$$⇒Det(PM)=Det(A−I)2×Det(P)

$$\Rightarrow \mathrm{Det}M={(\mathrm{Det}(A-I))}^{2}$$⇒DetM=(Det(A−I))2

Now $A-I=[\begin{matrix}1 & 7 & {w}^{2} \\ -1 & -w-1 & 1 \\ 0 & -w & -w\end{matrix}]$A−I=[1−10​7−w−1−w​w21−w​]

$$Det(A-I)=({w}^{2}+w+w)+7(-w)+{w}^{3}=-6w$$Det(A−I)=(w2+w+w)+7(−w)+w3=−6w

$$Det{((A-I))}^{2}=36{w}^{2}$$Det((A−I))2=36w2

$$\Rightarrow \alpha =36$$⇒α=36