Given,
cos(sin−1(xcot(tan−1(cos(sin−1x)))))=k
Now simplifying cos(sin−1x)=cos(cos−11−x2)=1−x2
So, cos(sin−1(xcot(tan−1(cos(sin−1x)))))=k
becomes cos(sin−1(xcot(tan−11−x2)))=k
And now solving cot(tan−11−x2)=cotcot−1(1−x21)=1−x21
So, cos(sin−1(xcot(tan−11−x2)))=k becomes
cos(sin−1(1−x2x))=k
Now solving cos(sin−1(1−x2x))=1−x21−2x2
So, 1−x21−2x2=k
⇒1−2x2=k2(1−x2)
⇒(k2−2)x2=k2−1
⇒x2=k2−2k2−1
So, roots are α=k2−2k2−1⇒α2=k2−2k2−1
And β=−k2−2k2−1⇒β2=k2−2k2−1
Now finding α21+β21=2(k2−1k2−2) and βα=−1
So, sum of roots of x2−bx−5=0 will be =α21+β21+βα=b
⇒k2−12(k2−2)−1=b⋯(1)
Product of roots of x2−bx−5=0 will be =(α21+β21)βα=−5
⇒k2−12(k2−2)(−1)=−5
⇒2k2−4=5k2−5
⇒3k2=1⇒k2=31⋯Putin(1)
⇒b=k2−12(k2−2)−1=5−1=4
k2b=314=12