Step 1Heat the oven to 325 degrees. Grease a 9-by-5-inch loaf pan (8-cup capacity) with olive oil and line the bottom and two long sides with parchment paper. Grease the paper, then dust the inside of the pan with all-purpose flour to coat, knocking out any excess.
Step 2In a small bowl, sift or whisk together both flours, the baking powder, baking soda and salt.
Step 3Place a microplane grater over the bowl of a stand mixer or large bowl and grate the zest from the oranges and lemon into the bowl. Halve one orange and juice it; you should get at least ¼ cup juice (drink any extra). Using a paring knife, trim the ends from the remaining two oranges, then cut off their white pith. Cut in between the membranes to release their segments (discard the membranes), then cut each segment in half crosswise. Reserve the segments (about ½ cup) in a small bowl.
Step 4Pour the orange juice into the bowl along with the brown sugar and attach to the stand mixer (or use a hand mixer); beat on medium speed to combine. Add the eggs one at a time, beating well after each addition. Reduce the mixer speed to low and slowly pour in the olive oil and vanilla. With the mixer still running on low, spoon in half the dry ingredients, then slowly pour in the buttermilk. Spoon in the remaining dry ingredients and continue mixing until the batter just comes together. Remove the bowl from the mixer and scrape the bottom and side with a large rubber spatula to make sure the batter is completely mixed.
Step 5Pour the batter into the prepared loaf pan, then scatter the orange segments evenly over the top. Use your finger or a small spatula to gently press the segments into the batter so they’re just covered. Sprinkle the top of the batter with the granulated sugar and bake until a toothpick inserted in the center comes out with no crumbs attached, 60 to 70 minutes. Remove the loaf cake from the oven and let cool completely in the pan before unmolding and serving.