Ivy can soon smother a small tree and add a lot of weight to the branches, also being evergreen it can hide any defects in the tree.
The 'wind-sail effect' can also play a factor in the tree's demise.
If its a productive tree that you want to keep in good condition, then I'd be inclined to remove the ivy, in spite of its wildlife value.
Cut through the stems near the base, cutting a section out of at leas 3 inches, to stop it rejoining. the small rootlets what it uses to cling to the tree are merely anchor roots and do not take any nutrients from the tree.
I removed a lot of ivy from a dead damson tree and a brick out-building 2 years ago and found minimal re-growth to deal with.
Not a job you'd want to wear your Sunday best for, I'd also recommend eye protection too, as it can be very dusty and dirty!