Scholarship is just one dimension of academic life, of life in general. A really big name in my field has no family, few close friends who aren't academics, no romantic relationship. All she does is her work. I'm not suggesting that as a model by any means, but when you look at the great things this person has accomplished, that's the context.
If you are being judged for tenure on research, at a research university, you have to realize that people expect a certain dedication to developing a research program. The problem is that you might be pulled in many directions at once. If you are involved in five or six things that are as demanding as your research, guess what's going to suffer? Better to do two or three things well than struggle to keep up with twice that many commitments. You can have family, work, and one serious hobby. You can't have small children, work, one serious hobby, run a community organization, be involved in university governance, spend hours devoted to religious activities, and take care of three dogs.