It must more pleasant than with a normal boat : 1 hour car tll the seaport, then 3 hours on a boat if I remember well (good luck if the sea is bad), 2 or 3 hours on the island with a little lunch and it's already time to come back. The island is beautiful (a little dirty around the camping), but the trip is too long for me.
A hydrofoil from Patong should considerabily reduce the trip time and allow you a longer time on the island.