Depends on what theme you use.
For my themes Zukitwo and Zukiwi.
The first you'll see when you open that file is this:
xthickness = 4 # The spacing (padding) inside and between applets. Add a lower value for a more compact look.
Change 4 to a higher number. In other themes it may be located in the gtkrc file. Style classes can vary between themes. But this will fix it in my themes anyways.
Unfortunately there's a design miss by the XFCE devs. When that number is higher the icons in the panel will become smaller.