The coefficient of x7 in (ax2+2bx1)11 and (ax−3bx21)11 are equal, then
Tr+1=11Cr(ax2)11−r(2bx1)r
=11Cr(a)11−r(2b1)rx22−3r
Now solving 22−3r=7⇒r=5
Coefficient of x−7 in (ax−3bx21)11
Tr+1=11Cr(ax)11−r(−3bx21)r
=11Cra11−r(−3b1)rx11−3r
Now solving 11−3r=−7⇒r=6
Now equating coefficient of x7 and x−7 we get,
∴11C5(a)6(2b1)5=11C6a5(−3b1)6
⇒36ab=32
⇒729ab=32
Hence this is the correct option.