Quoting Mmmmbop.:" I'm not sure i can do almond since we also found out he has a peanut allergy and I'm afraid to try other ... [snip!] ... formulas taste good enough for your little one? Mines still breast feeding but the doctor suggested I stop or change my diet."
I would just change your diet and not give him any of the milks.
I had to completely cut dairy out of my diet after my son was born because he had a milk protein allergy. When he was 1 we tried cheese and he did good with it so I slowly reintroduced dairy but it really wasnt too hard cutting dairy out of my diet for a year.