I tried on the sweater, and the difference in color was really obvious. So, in the dye bath it goes! I used RIT Lemon Yellow.
The color difference in the yarn is less noticeable. There's nothing I can do about the texture, though. I washed and dried it and it seems to have improved a little.
I still need a white sweater!