Let,
A=[prqs]
A2=[prqs][prqs]=[p2+qrrp+rspq+qsqr+s2]
Now given, A2=I
⇒[p2+qrrp+rspq+qsqr+s2]=[1001]
Now on comparing both side we get,
⇒p2+qr=1,q(p+s)=0
And r(p+s)=0,qr+s2=1
Now on solving above relation we get,
q=0⇒p+s=0⇒a=0
And b=∣A∣=ps−qr=−p2−qr=−1(∵s=−p)
∴3a2+4b2=4