Wash orange thoroughly; dry. Using a vegetable peeler, remove zest from orange, taking care to peel as little of the white pith as possible; roughly chop zest and transfer to a medium bowl. Juice orange and add juice to zest along with olives, rosemary, and pepper; toss to coat. Let sit at room temperature for 1 hour to marinate before serving.