For no solution D=∣11a1ab111∣=0
Expanding the determinant along the first row, we get D=−(a−1)2=0
⇒a=1
For a=1, first two planes are coincident.
Now, we have x+y+z=1&ax+by+z=0
For no solution, these two planes are parallel.
⇒a1=b1=11
\Rightarrow a=1&b=1.
∴b=1⇒it is a singleton set