I'm gonna suggest that, if you truly feel that way, you may be doing it wrong.
The wonderful thing about reading good fiction is that the book merely presents a framework for a dream; the reader must fill in the blanks, consciously or subconsciously. So it's not so much having someone else's dream as allowing someone else guide to you in your own dreams. And switching books, authors, or genres makes for endlessly changing guidance requiring varying degrees of imagination by the reader.
Whereas visual entertainment mostly lacks this quality. Watching TV is much closer to having someone else's dream than reading fiction books or stories (or poems).
Not that anybody should or shouldn't like any form of art. There are no rules, despite what we're taught. We should all like whatever we like and to hell with what anybody else says.
I use imagination with non-fiction.
If someone writes non-fiction about Gettysburg I imagine many things rooted in reality.
Imagination is ok when it an extension of the mind.
When the mind becomes an extension of the imagination that's the border into all sorts of issues.
Heightened degrees of fantasy proneness are associated with greater suggestibility.
That's a reason hypnotists use imagery/concentration to alter a subject's mental state.
When people focus tightly to one thing while losing track of others that can be being transfixed.
People can be told tight focusing actually helps a person be more aware but they are actually less aware
People get married to the wrong person, elect the wrong person, hate the wrong person etc because of imaginary perceptions they think are real.
A guy wrote a Pulitzer book about Vietnam called "A Bright Shinning Lie"
We can all fall for shiny lies if we get lost in them uncritically.
Nazis gassing people in the day and enjoying fine wines, foods and Wagner at night had altered states
So did the kamikaze pilot dying for the emperor.
A saying about jazz is that "music is a journey and jazz is getting lost."
Jazz started to suffer when people started studying and dissecting it too much
A person cant know what its like to drive cross-country from studying a map.
But people try just that when the trees become more important than the Forrest.
Watching a video of a CC drive isn't a substitute either but its more informative than a map or Google Earth.
Joyce is a jazz type who burrows deeper into terms and crevasses and thoughts and feelings and most are just evocative nothings - he admitted to trying to lead a reader into an altered state. It seems like a lot is going on when there isn't really. Reminds me of old public TV ads when they played discordant music suggesting bizarre sates of mind that that were supposed to suggest deep knowledge/mystery but a scrambled mind isn't deep in anything - it can just think it is.
A reason academics are often bad at practical things is because they they studied and absorbed, and got deeper and deeper into less and less thinking they were expanding into more and more. They get prone to bright shinning lies. Joyce himself had a fairly broken life and thats no surprise
I'll take Hugo's reality based fiction over Joyce's jazzy nonsense that "seems" smart. A lot of artists get lost because they are self-conditioned to be suggestible and fantasy prone.
Fantasy-prone personality
en.wikipedia.org
Exploring the relationship between fantasy proneness and delusional beliefs in psychosis and non-clinical individuals