I think a lot about why some people are more affected by games than others. I think an interesting way to look at it is as a parallel is to suggestibility. "Suggestibility is the quality of being inclined to accept and act on the suggestions of others." What I think affects the impact that games have on people is their systemic suggestibility – how inclined people are to accept and act on the suggestions of systems. For people who are systemically suggestible, they want to follow the rules of the system, and that system therefore has a weight in their mind. This is important in games, since one of the strongest ways the medium has to impart meaning is through systems. Most of the time this meaning is a feeling of power, but it can also be more subtle emotions.
Dys4ia, for example, often relies on an interest in "succeeding" at a task to convey its affect/meaning. For example, the first task the player is given in the game is to move past this wall.
However, the inability to do so, and therefore the feeling of helplessness conveyed by this system is directly tied to how strongly the player desires to "accept and act" on the system, i.e. how suggestible they are.
Certain people seem to have little to no systemic suggestibility. I've showed Dys4ia to someone who after playing it said "I don't know, I would've enjoyed that more had they just written about their experiences." For these people the systems in a game carry no weight. If Dys4ia tells them to try an impossible task, they just see it as something they can't do, not something that they are being prevented from doing.
The fact that some people will never enjoy games doesn't seem exclusive to the medium of games. I have met people who say they just can't read comic books (and a search on google reveals that there are a lot of these people). What does this mean for games? I'm not sure. Perhaps it means games are something that is learned, just like reading. Abstracting ideas and concepts into systems and placing weight on those abstractions perhaps just doesn't come naturally to some people.