Step 1Whisk the yolks and sugar with the whisk attachment of an electric mixer until the batter reaches the ribbon-forming stage, about 1 minute.
Step 2Place the cream and milk in a large saucepan and bring to a boil, then turn off heat. Stir one-half cup of the hot milk mixture into the egg mixture to temper the eggs. Continue to stir in the remainder of the milk mixture until thoroughly combined.
Step 3Reduce the heat to low and cook until the mixture lightly coats the back of a spatula or wooden spoon, stirring constantly.
Step 4Strain the mixture into a metal bowl and place the bowl in a larger bowl filled with ice water. When the mixture is chilled, stir in the olive oil. Pour the mixture, in batches if necessary, into an ice cream maker and process according to the manufacturer's directions. Place in sealed containers and freeze.
Step 1Heat the oven to 350 degrees. Sift the flour, sugar, baking soda and baking powder into a large bowl. Make a deep well in the center. Combine the eggs, milk and 1 1/2 cups olive oil in the well. Whisk the wet ingredients, then gradually start whisking in the dry ingredients. Each time, beat until smooth, then pull in more dry ingredients. When the batter is smooth without any lumps of flour, stir in the lemon zest.
Step 2Grease one-fourth cup size brioche molds or substitute muffin tins. Spoon 1 teaspoon of the remaining olive oil into each mold or cup. Brush all of the sides and let the residual oil pool in the bottom of each mold or cup.
Step 3Fill each mold three-fourths full. You will see olive oil sitting around the batter. This is OK. Sprinkle each top with one-fourth teaspoon rosemary. Place the individual molds on a baking sheet or put the muffin tin on a rack in the center of the oven. Bake until dark golden, about 35 minutes.
Step 4The cakes are easier to unmold when warm. Let the molds cool a little. Give the bottoms a hard bump with a knuckle protected with a towel. Use a small offset spatula or knife to loosen the cakes.
Step 5For each serving, allow 1 1/2 cakes. Place 1 cake on the plate with rounded side up. Cut another cake in half, then in half again, making quarters. Place 2 quarters beside the whole cake, lying on their side. Place one scoop of ice cream on top of the 2 quarters. Sprinkle a touch of fleur de sel -- less than a pinch -- on top and serve.