Academia is a toxic environment in many ways. You can love the work (as I do), teaching and research, without loving the environment itself. Ironically, one thing that makes it toxic is the presence of academics, people who should be devoted to the work itself. Even those devoted to the work, and with good intentions, have insecurities and neuroses to such a degree that the presence of so many in one place creates a kind of toxic overload.
One source of problems is the presence of overt political agendas, whether conservative or radical. A political agenda makes an atmosphere toxic by confusing real politics with symbolic, merely academic politics.
Another source is institutional structure and ethos. A wealthy institution like Princeton can be toxic (see the Calvo case), as much a resource-poor state institution.
My institution and department is relatively free of these toxins.