My feeling about Rayong is simple. If you are a Farang Please dont go there. This is a nice city, but not on the "Tourist" track and it is a Thai city for Thais only. They are many cities in a lot of countries like this: keep out foreigner, we like our city as it is and dont need you here! It kind of goes along wth the Thai saying of "Are you man enough to eat our food and leave our women alone?"
The nearest resort to Rayong is Wang Kao and it is a Thai only resort also, even though it isnt "Posted" or anything such as that you should be able to see it clearly if you go there, especially is you are travelling with a BG.
Another city to avoid with a BG is BanPong (on the road to Kanjanaburi) and I only tell you this because I saw a young man get an unwanted haircut by a Policeman there once and I knew then how it was.
Having been associated with the Embassy for a few years I speak from knowledge that is rooted in 35 years of experience with Thais and I love them dearly, but often am in awe at the insensitive actions of my countrymen when visiting Thailand. Chok Dee in your travels.