Ceanothus and Olive tree too close?

Planted these a few years ago, didn't expect the ceanothus to get this big! Should I move the olive tree maybe? It doesn't seem to be loving this spot anyway. Gets full sun in summer but partial shade the rest of the year.