Most towns in Iceland have a public pool, usually outdoors, heated by nearby hot springs and open year-round. Admission is a inexpensive and towels can be rented. Some charge an additional amount for use of the sauna. 

There is a protocol that you should follow before entering any pool in Iceland. In the locker room (separate for men and women), remove your clothes and take your towel and swimsuit to the showers. Leave your towel in the towel compartment near the showers. The showers are open and you will be showering in the nude with others. Enter the shower and wash completely - hair and body (soap and shampoo are provided). Put your suit on under the shower, then go to the pool.

Reykjavík, the spa city, has seven geothermally heated pools with hot pots (outdoor hot tubs), steam baths and saunas. Ocean swimming is also available in Reykjavík at a hot spring-fed beach during the summer months.

Wherever you go, you should take a bathing suit along in case you want to enjoy a pool or natural warm spring (Be sure to ask if it is safe - many are too hot to soak in).