Heat the oven to 350°F and grease an 8- or 9-inch square or round pan.
Put 2 tablespoons of the olive oil in a large skillet over medium heat. When it’s hot, add the plums and cook, stirring occasionally, until they’re tender and swimming in juice, 5 to 7 minutes. Add 2 tablespoons of the brown sugar and cook until it dissolves, 1 to 2 minutes. Remove from the heat.
Put the yogurt, the remaining ⅓ cup olive oil, the egg, the lemon zest, the vanilla, and the almond extract in a medium bowl. Whisk to combine, then whisk in the remaining ⅔ cup brown sugar, followed by the baking powder and salt. Add the flour and stir just until combined.
Transfer the plum mixture to the greased pan and spread into an even layer. Spread the batter evenly over the fruit, and bake until the cake is golden brown and a knife inserted into the center of the cake comes out mostly clean, 50 to 60 minutes. Cool thoroughly, then run a knife around the edge of the cake to loosen it from the pan, place a large plate over the pan, and invert the cake. Serve at room temperature. (Store leftover cake in an airtight container in the refrigerator for up to a few days.)
