The general term in the expansion of (a+b)n is Tr+1=Crnan−rbr.
Given, in the expansion of (1+xlog2x)5, third term is 2560
⇒T3=5C2(xlog2x)2=2560
Using, Crn=r!⋅(n−r)!n!, we get
⇒2!⋅3!5!(xlog2x)2=2560
⇒2×1⋅3!5⋅4⋅3!(x2log2x)=2560
⇒(x2log2x)=256
Taking logarithm to the base 2 on both sides
⇒log2(x2log2x)=log2256
Now, using {\mathrm{log}}_{a}{m}^{n}=n{\mathrm{log}}_{a}m&{\mathrm{log}}_{a}a=1,
⇒2log2x(log2x)=log228
⇒2(log2x)2=8
⇒(log2x)=±2
Using, logax=b,⇒x=ab
x=22,2−2
⇒x=4,41
Here, only x=41 is given in the options.