Things to consider;
Rabbits can easily dig warrens 3' down and 8' across. They can chew through chicken wire, and wire degrades underground.
Rabbits like to build warrens near "something", like a structure. A tree in the middle of the pen would be good for example. Otherwise the "structure" they dig by will be your fence line.
Rabbits have been raised individually in cages for 1000 years. some 2000+ generations of rabbits have been raised in cage-based environments. Some lines of rabbits have really lost their social instincts to 'play well with others'. Most I'd wager. It may take a few years of selective breeding for temperament to get rabbits that really do well in a colony setting. Choose the rabbits you put into a colony with care. You may want to seek out people raising in colonies for your brood stock.
Your rabbits WILL need hiding spots to get away from each other, without being backed into a corner. Tubes, boxes with multiple exits, etc. You will need multiple feeding and watering spots so they don't fight over food and water. They need to be able to get away from eachother.
You will have to watch for and break up fights.
You will never know which kit comes from which parents unless you construct your genetics carefully so the liters always look different based on the parents. (For example, a Charlie buck will always have broken kits. A REW buck will only have REW kits with a REW carrier or a REW doe.)
You may have trouble controlling when litters happen. A rabbit CAN have 8-9 litters a year, but this wears out your does very fast.
Contact with ground can spread parasites.
Keeping an eye on herd health will be paramount. One sick rabbit could mean they're ALL sick rabbits, and rabbits feeling stress (environmental or social) get sicker easier.
My experience;
Rabbits fight. Every caged rabbit has been bred and raised to think it is the "queen" rabbit, the rabbit that leads the colony, is in charge of the territory, and therefore the rabbit that gets to breed a lot. I've tried a colony before. 30X60 garage, with lots of places to run and hide, infinite food from huge hay bales, etc. It was a disaster. My rabbits come from show and meat lines, raised in cages. The setup was with 2 moms (one was colony raised as a kit) and their kits. By the time they were 12 weeks old (ready to butcher as they grew slower than their caged counterparts), they were fighting so viciously that I couldn't keep some of the pelts from terribly patchy scabbing. Mom1 wouldn't let any kits near the feeders of pellets while she was eating the water dishes, etc. She'd even camp out by them and attack even her own kits that got close. Yikes! She doesn't behave this way with her kits in cages... And in a cage situation I pull kits to their own cage at 6-8 weeks. She's an angel in a cage, a terror in a colony. And the other mom (colony raised until she was 4 months old) was only slightly better. The aggressive mom actually got into such a fight with the other mom's buck kit that she now has a hole in her ear and a scarred up nose. After that I decided never again, or at least never again until I can take the time to raise and breed rabbits of the right temperament. Even then I'd be very careful.
My solution;
3'x4' or 4'x4' cages 2' high with solid or very tight wire mesh floors and hay bedding. They hop, run, binky, they have toys and they can dig in the hay. They're very low stress animals, and NO fighting. I breed when I want, and give breaks when I want. I can wean away from mom at 6 weeks and re breed at 8 weeks giving them a nice big recovery time. They're happy, active, and healthy. I never have a litter lost "on the wire" because they can always nest even without a nest box and they have hay bedding. Hay is cheap compared to pellets and they love it. It works for me!