(1+ax+bx2)(1−3x)15=(1+ax+bx2)(C015−15C13x+15C2(3x)2−15C3(3x)3+…−15C15(3x)15)
∴ Coefficient of x2 is: b.15C0−3.a15C1+32..15C2=0
(Given)
⇒b−45a+945=0......(i)
Coefficient of x3 is:
−b.3.15C1+9a.15C2−27.15C3=0
⇒−45b+9a×215×114−27×315×214×113=0
⇒−b+21a−273=0
⇒b−21a+273=0.......(ii)
From (i) and (ii)
a=28 and b=315