Brick patio question

I’m building a brick patio in this spot I’m digging out, next to it is a shed. I’m wondering if I should avoid digging out right up to where the shed is to avoid any kind of cave in, or if it’s ok to dig right up to the pavers the shed is sitting on