In the very first sentence of the passage, the author sets up a strict binary choice presented to him in his youth: dealing with "only men or only books."
He explicitly states that his answer "would certainly have been in favour of books." Therefore, his preference was specifically to have dealings only with books when given that choice. Option 2 captures this exact phrasing and absolute constraint ("only") used in the passage