Given that the sum of all the diagonal elements of ATA is 6, means sum of all diagonal elements is equal to sum of square each element of the matrix.
i.e., A=[a1b1c1a2b2c2a3b3c3] and AT=[a1a2a3b1b2b3c1c2c3]
Then tr(A⋅AT)=a12+a22+a32+b12+b22+b32+c12+c22+c32
Given ai,bi,ci∈−1,0,1 for i=1,2,3
Now sum to be 6 exactly three of them should be zero and rest should be 1 or −1.
So, total number of possible matrices C39×26
=69×8×7×64
=5376