Eliminating sweets is likely more important. Complex carbs in pasta / bread or even simple sugars in fruits are absorbed more slowly which supposedly is much better for you. I think the jury is still out on how good or bad carbs are for you. My instinct tells me they are probably completely fine if they don't make you feel bad personally.