It sounds like you have a good relation with your neighbor, and this is a just 'what if' type of question?
Each state has taken a few centuries to develop laws around this issue, so what applies in one state might be different in yours.
The common usual way in many states: your neighbor owns the tree, you cannot do something to kill it or you are liable for the loss.
You can trim anything hanging over your property line, as you control your space. But the wood likely could belong to your neighbor, even what you cut down - you'd need to stack for him to take. But if it has no value, it is yours to dispose of, you can't just toss debris over the fence.
The fruit of the tree would belong to the neighbor (as it is of value, like the wood of the tree). But if it falls on your ground, it is yours to do something with - abandoned by the owner, more or less.
Yes - that leaves a _lot_ open to figure out yet - if you & your neighbor hate each other to the end of time, how would the neighbor harvest the fruit hanging over your property without tresspassing on your land? And so on.
I donno.
It gets kinda odd if you look into the laws deeply.
Hopefully this is just one of those idle questions, wonder what if?
--->Paul