If two vectors a1i^+b1j^+c1k^ and a2i^+b2j^+c2k^ are perpendicular, then a1a2+b1b2+c1c2=0.
Given OP⊥OQ
⇒−x+2y−3x=0
⇒y=2x...(i)
Also, PQ=OQ−OP
⇒PQ=(−1−x)i^+(2−y)j^+(3x+1)k^
And ∣PQ∣=20
⇒(−1−x)2+(2−y)2+(3x+1)2=20
⇒(x+1)2+(y−2)2+(1+3x)2=20
Put the value from equation (i), to get
(x+1)2+(2x−2)2+(1+3x)2=20
⇒x2+2x+1+4x2−8x+4+1+6x+9x2=20
⇒14x2+6=20
⇒14x2=14
⇒x=±1, but given x>0
⇒x=1 and y=2.
Now OP,OQ,OR are coplanar and we know that the three vectors a1i^+b1j^+c1k^,a2i^+b2j^+c2k^ and a3i^+b3j^+c3k^ are coplanar, then ∣a1a2a3b1b2b3c1c2c3∣=0
⇒∣x−13y2z−13x−7∣=0
⇒∣1−1322z−13−7∣=0
⇒1(−14−3z)−2(7−9)−1(−z−6)=0
⇒z=−2
∴x2+y2+z2=1+4+4=9.