Gather all the stone fruit cake ingredients together.
Mix the milk and lemon juice together. Cover and set aside to curdle slightly.
Line a square 20 cm by 20 cm cake pan with baking paper.
Preheat the oven to 180°C/350°F.
Cream the butter and sugar together until fluffy and light.
Whisk in the eggs, one at a time with a tablespoon of flour.
Add the flour, baking powder, xanthan gum, ground almonds, yoghurt and milk mixture. Fold everything together gently.
Add the orange juice, zest, and vanilla extract. Gently mix to combine.
Gently spoon the cake mixture into the prepared cake pan.
Arrange the stone fruit slices and slivered almonds in rows.
Complete the four rows with alternating stone fruit slices.
Bake for 40 minutes, or when a toothpick inserted into the centre of the cake comes out clean.