Place a rack in middle of oven; preheat to 350°. Lightly butter a 9"-diameter cake pan and dust with all-purpose flour; tap out excess. Line bottom of pan with a parchment paper round. Arrange 4-5 large red plums, cut into ½"-thick wedges (about 8 oz.), in a circular pattern in pan, starting from center and working outward and overlapping if needed.

Whisk 1 tsp. baking powder, 1½ cups (188 g) all-purpose flour, and 1 tsp. Diamond Crystal or ½ tsp. Morton kosher salt in a large bowl. Combine 2 Tbsp. finely grated lime zest and 1½ cups (300 g) granulated sugar in a large bowl and rub together with your fingers until evenly distributed, sugar starts to clump, and mixture is very fragrant. Add ¾ cup (1½ sticks) unsalted butter, room temperature, and beat with an electric mixer on medium-high until light and fluffy, about 3 minutes. Add 2 large eggs, room temperature, one at a time, beating to combine between each addition, then add 1 tsp. vanilla extract and beat 10 seconds. Reduce speed to low and gradually beat in dry ingredients, scraping down sides of bowl as needed. Add ½ cup whole milk, room temperature, and mix just until combined. Set batter aside.

Bring remaining ½ cup (100 g) granulated sugar, remaining 1 tsp. Diamond Crystal or ½ tsp. Morton kosher salt, and 3 Tbsp. water to a boil in a small saucepan over medium heat. Carefully add ¼ cup fresh lime juice, watching out for spatter, and cook undisturbed until liquid is thickened and beginning to turn a deep amber color, 5–7 minutes. Gently stir, remove from heat, and immediately pour lime caramel evenly over plums in pan. Scrape reserved batter over and smooth surface if needed.

Bake cake until golden and a tester inserted into the center comes out clean, about 1 hour. Transfer pan to a wire rack and let cake cool in pan.

Run an offset spatula along edges of cooled cake to loosen. Set a platter upside down over pan and flip over to release cake. Carefully remove pan and peel away parchment paper; discard. Serve warm or room temperature, topped with scoops of vanilla ice cream if desired.