Don't get me wrong; I don't think everybody needs to get all bike-specific with their commuting clothes (although bike clothes do dry more easily while you work). Keeping your work clothes dry and clean on the way in or leaving enough clothes at work on the weekend or whatever will prevent the aforementioned nasty experiences and make you more likely to keep it up.
Another bonus of commute clothes vs. work clothes is that you are more likely to ride harder and/or have more fun. Yes, you may sweat more, but some deodorant and a towel are enough to be fresh and pretty after you change for lots of people, at least the ones who shower every day. I don't know about you, but I commute by bike mainly for the sheer fun of it. Different clothes = more fun. That means that even fair weather only riders could be having more fun too.
I know some people are into this cycle chic thing, but I just don't get it. I mean, many of their bicycle choices are driven by the fact that they will be wearing "lovely shoes and trousers/skirt" (see the fenders section)... are these chic people really going to remain regular bike commuters in the long term? Not after the rain and sweat and dirt of commuting makes them look un-chic at work a few times. C'mon, chic people, instead of reducing the fun of cycling in order to increase the fun of wearing stuff, including your bicycle, just drop the high style and let your bike sing! A dutch city bike just looks like a depressed pack mule to me. That is what happens when you make a beautiful machine a mere servant of your clothes. Even if you really like that style of bike, why not do it some justice and ride the hell out of it instead of using it as an accessory?
You know what? Who am I to tell you what to do? Get your kicks however you like. Might I suggest, however, that a set of clothes separate from those you intend to wear at work all day would allow you to 1) ride more days of the year and 2) offer the opportunity to wear TWO fabulous outfits every day! The healthy glow and relaxed good nature that really commuting for fun will give you, combined with your awesome clothes, will make you even more attractive, I promise!
PPS - Counter-point comments of reasonable depth have been coming in... I invite you to read them, because cycling starts with balance.