It's not as bad as people who don't live there say. That said, check out some of the towns around Beaumont. Nederland is a very short drive from the Lamar campus. Also, Lumberton is growing, and they have some nice new neighborhoods. The west end of Beaumont proper is a good area if you're looking in town.

I live in the south end of Silsbee. My wife works in Beaumont, and I drive there to the gym every day. The only bad part is going through Lumberton when the school zones are in effect. I went to Lamar a couple of years before transferring to A&M. I commuted (I lived halfway between Silsbee and Kountze back then), and it wasn't bad at all.