The primary source is the Mishna in Sanhedrin "הקורא בספרים חיצוניים..." , (see Tiferes Yisroel there). I am convinced however that this refers (as I believe Radal writes) to the apocryphal books, and this out of fear that they would make their way into the canon. R' MM Kasher has a lengthy addendum on the subject somwhere in his Torah Sheleimah.
If the Yerushalmi on that Mishna is in fact permitting the reading of Homer (ספר הומירוס as explained by S. Lieberman, Hellenism) one wonders what can possibly be forbidden?
See here on the Yaavetz's vast reading. R' Yaakov Kamenetsky is said to have commented that he "read a whole lot, perhaps more then was permitted" (MOAG) but in this he refers to the issue of Bittul Torah, while the issue under discussion is if there are books with forbidden content (obviously excluding דברי חשק).
I know of SA OC 307:17 (and see Mor U' Ketziah there) but even there the matter requires further study. For instance, I am inclined to wonder if the ban against the despised Sefer Immanuel is indeed because of questionable content (as argued by Mekor Baruch) or is it rather a reaction to the author's opposition to Kabbalah?
Comments welcome (For Hungarian posekim, as is obvious, the question doesn't start.)