The given equation is
2x+3y+6z=8
x+2y+az=5
3x+5y+9z=b
So, D=∣2133256a9∣
=2(18−5a)−3(9−3a)+6(5−6)
=36−10a−27+9a+30−36
=3−a
Dz=∣21332585b∣
=2(2b−10)−3(b−15)+8(5−6)
=4b−20−3b+15+40−48
=b−13
The system of equation has no solution if D=0&{D}_{z}\neq 0
So, If a=3,b=13, then the system of equation has no solution.