The general term in the binomial expansion (a+b)n is Tr+1=Crnan−rbr.
Given, the fourth term in the expansion of (x2+xlog8x)6 is
T4=20×87
⇒C36(x2)3(xlog8x)3=20×87
⇒3!⋅3!6!(x2)3(xlog8x)3=20×(23)7
⇒(3×2×1⋅3!6×5×4×3!)(x2)3(xlog8x)3=20×221
⇒(20)(x2)3(xlog8x)3=20×221
⇒[(x2)(xlog8x)]3=(27)3
⇒(x2)(xlog8x)=27
⇒(x1)(xlog8x)=26
⇒xlog8x−1=82
Taking log both side to the base 8
⇒log8(xlog8x−1)=log8(82)
Using logamn=nlogam, we get
⇒(log8x−1)(log8x)=2
⇒(t−1)(t)=2 (Let log8(x)=t)
⇒t2−t−2=0
⇒t=2 or −1
⇒log8x=2 or −1
If logax=b,⇒x=ab,
⇒x=82 or 8−1
⇒x=64 or 81.