Alfalfa as part of a forage is great. I personally do not like it for sole hay for my dairy goats, though many use it successfully. I have not noticed alfalfa level to affect flavor, however - mostly production.
Alfalfa (high calcium) IS implicated in the formation of Calcium Oxalate crystals in male animals. Crystal formation is multifactorial, however, and so anecdotal stories about one herd doing fine does not imply that it is safe in all situations to feed a very high amount of calcium to male animals. The calcium to phosphorous ratio is critical, as is promotion of adequate or increased water consumption, urethral size (castrates vs intact males may play a role), Local forage/soil types and growing environment, daily variation in individual physiology, and a genetic predisposition are all very important in stone formation.
PREVENTION by feeding a proper Ca to P ratio without excess Ca or P than requirements is probably the most important thing you can actively do as a producer (along with water access and quality of course), as ammonium chloride has recently been researched and shown to cause only TRANSIENT urine acidification. Prolonged use shows decreased efficacy and urine pH will turn back to 'normal' alkalotic with extended use. This can be curbed somewhat by dosing intermittently with AC to help dissolve any forming stones.
It is important to note that Calcium Oxalate crystals and many other stones CAN NOT be dissolved and are a surgical emergency if the cannot pass. Struvite stones are dissolvable but are still a medical and/or a surgical emergency.
Water deprivation prevention seems obvious but not completely considered in all cases it seems. It can include deterrence such as soiled or medicated water, iced over water in the winter time in addition to the usual first assumed of 'running out of water'. Decreased water consumption causes a concentration of urine and waste products including calcium, as well as a potential altered pH of urine causing precipitation of certain solid crystals).
As for milk taste, that can vary greatly depending on many, many factors. Diet is one. Instead of switching back and forth rapidly, allow adjustment time whenever you change diet.
Personally, I hate leafy alfalfa as I see the leaves precipitating from the stems right into the dirt. Sure, the does sometimes will lick them up so they're not wasted, but if they hit the ground and are licked up I may be watching a very nice, rapid route for parasite infection occurring.
The majority of the weight of pure alfalfa hay is stem, which is poorly digestible and of low nutritive value. Usually waste is quite high. As stated by someone previously, the bossy does get access to fresh hay while the less dominant does are left with the stemmy leftovers - further predisposing them to 'culling' with lower intake, lower milk production, poorer reproduction, poorer growth, increased susceptibility to parasites etc. This can be remedied with increased bunk space/feeder availability however, but this means increased labor usually.
Personally here, I feed a nice high quality, soft green grass or grass/alfalfa mix hay and supplement with alfalfa pellets in the feed. My goats generally HATE alfalfa pellets but eat enough of them when I mix with their grain on the milkstand for lack of options.
Hay quality can certainly vary year to year due to growing conditions. I have one hay seller that I buy from because we're most impressed through the years with his hay - and he provides a steady supply without us having to store a bulk amount through the year. Sometimes that means I end up with a lower quality hay than I'd like, but never mouldy or disease-causing - unlike other hay I've had.
My bucks generally don't get grain but if they do, they receive the kid feed which is actually an 18% lamb grower with a coccidistat and ammonium chloride. The reason I use this feed is because it's balanced in Calcium and Phosphorous. I use it to feed my kids (including wethers), but feed it to the bucks so that my father does not have to bother with buying a separate feed for the bucks. It's ease of management more than anything. All goats receive the same hay for the most part, with the bucks usually receiving less quality hay than the does for the majority of the year. If the hay has alfalfa in it, so be it - If I can't conveniently avoid it, it is a risk I assume at that point.