Hey.
Carpenter ants(the big black ones) will do what you said, but they usually do it in a damaged area of the tree. It would be easy to see if they're coming out of the holes. It may be a beetle also. Check with your local DNR to see what borers are around.
Borers that attack sugar maple include the carpenterworm (Prionoxystus robiniae), sugar maple borer (Glycobius speciosus), maple callus borer (Synanthedon acerni), and occasionally horntajls (Xiphydria abdominalis and X. maculata)
Among the more important borers attacking red maple are the gallmaking maple borer (Xylotreehus aceris), the maple callus borer (Synanthedon acerni), and the Columbian timber beetle (Corthylus columbianus).
Asian long horned beetles bore into all types of maples...3/8 inch hole and these guys are bad news because they will kill the tree and go after other varieties in the area.
Lookup here to see if you have any matchups:
http://www.forestryimages.org/browse...ject.cfm?cat=7
The size of the hole(s) and the look of the area around the holes may be clues. Obviously, look for beetle type insects. The larvae(worms) you found should be analyzed.
RF