I don't personally care for the S... it will play all the same games but with limitations to resolution and/or framerates. But I have a really nice setup and play games often so I want the best. If money really isn't an issue for you and you happen to have a really nice OLED tv, I'd say get the X and enjoy.
But, when you say casual and you only mention NCAA, I lean towards saying just get the S. My son has the S in his room and plays on a 1440p computer monitor. He loves it because he doesn't really know or care about the bells and whistles he's missing. And honestly, the games I've seen him play all still look great.
One thing you could do... wait until NCAA releases. There will no doubt be Youtube videos showing the game's Series S performance. I bet a game like that would run just fine on the S, but it might be worth waiting.