Hard to tell what is actually hitting, is it the compressor housing or exhaust on the booster? You may get a bit of clearance by clocking the compressor housing so the outlet volute is positioned differently.
Depending on what you are doing for an intercooler, like a front mount, you could have an elbow welded to the compressor to make that outlet point where you need.
The center section of the turbo might need to be clocked more also so you have your oil return positioned properly. I don't know how much angle they ok in the installation guidelines.