Better to move rocks away from tree?

I have 2 large boulders fairly close to this tree. Would it be better for the long term health of the tree to get these moved away from before they inhibit growth? I’m also slightly concerned the tree or roots will get damaged while trying to move them.