In the G.P., let the first three terms be ra,a,ar.
From the product condition: ra⋅a⋅ar=a3=27, so a=3.
The sum of first three terms is S=r3+3+3r=3(r1+1+r).
For r>0, by AM-GM: r1+r≥2, so S≥9.
For r<0: r1+r≤−2.
So S≤−3.
Thus the possible values of S are (−∞,−3]∪[9,∞)=R−(−3,9).
Therefore (a,b)=(−3,9) and a2+b2=9+81=90.