Sounds like you may have gone a little overboard. If you are discreet you should get very few if any complaints. Be selective. Only hit sites that are related to your sites theme/niche and no more than one hit per site per day.
That way you will get a boost from a few more backlinks without the refspam backlinks being the majority of your total backlinks which will get you banned sooner rather than later.
The traffic from the actual links isn't really anything of consequence as it will just be webmasters checking stats or surfers who just happened to spot your ref link, although if you stick to similarly themed sites some of these surfers may buy/join something on your site.
