This is normal for goats. It's not only yicky, it is also a good way for them to get worms. And, they will spoil a lot of feed, because any feed that they've walked on or soiled in any way, they won't eat. You need to build a feeder that they can't get into. There are several ways to do this. I've built two different types for my goats. The does have what is called a 'keyhole' feeder. I built a manger, but the front of it is a solid sheet of heavy plywood with 'keyhole' shapes cut into it. I nailed a board several inches up from the bottom of the plywood. They stand their front feet on this board in order to insert their heads into the upper portion, the larger part, of the keyhole slot. I think the lower part of the keyhole is three or four inches wide, while the upper part is about seven inches wide. It's big enough that they can stick their heads through it, anyway. Then they step down to eat, and when they are done they have to step back up so they can get their head out the larger part of the keyhole. This keeps them from backing out of the manger with a big wad of hay, most of which will get dropped on the floor and wasted.
The other feeder, in the buck pen, is simply a hog panel upside down so the larger holes are at the bottom. I have Kinder goats, which are smaller than the standard breeds, and they are able to stick their heads through the holes in the hog panel. It doesn't necessarily keep them from pulling hay out and wasting it, but it does keep them from getting into the manger and walking on the hay there.
There are other types of feeders that will work. Which type you decide to use may depend on whether any of your goats have horns or not, as some feeders, including both of mine, won't work with horned goats.
Look for the Fiasco Farms website. She has a lot of good goatkeeping information there, and I think she may have plans for a goat feeder. If not, I know she has links to other sites, and one of them ought to have plans. Goats are clever animals and it takes time to get to know all their ways and learn to outsmart them!
Kathleen