Preheat oven to 350 degrees Fahrenheit. Remove eggs from the refrigerator (room temp eggs beat up better and ensure lightness of the cake). Spray a 9 x 2-inch round cake pan with cooking spray.
Melt the butter in a small saucepan over medium heat. Add brown sugar, lemon juice and allspice; cook, stirring, for about 2 minutes, until sugar has dissolved and the mixture is thick and bubbling.
Pour mixture into the bottom of the cake pan. Arrange the peach slices over the sugar mixture in an overlapping spiral design. If desired, scatter the raspberries over and around the peaches; set aside.
Whisk the flour, baking powder, baking soda and salt together in a medium bowl.
In another large bowl, beat the butter with an electric mixer at medium speed for 1 minute until smooth. Add granulated sugar and almond extract; beat for 3 minutes until light and fluffy. The mixture should be almost white in color.
Add the eggs, one at a time, beating after each addition.
Reduce the speed of the mixer to low and beat in the flour mixture in three additions, alternating with the buttermilk, beginning and ending with the flour mixture.
Spoon the batter over the fruit in the pan, gently spreading to cover.
Bake the cake for about 45 minutes, until a wooden skewer inserted into the cake center comes out clean, and the top is deep golden brown and springs back when gently pressed. The center of the cake needs to be firm to support the weight of the fruit when inverted.
Cool cake in pan on a rack for 15 minutes. Run a thin knife around the sides of the cake. Place a serving platter upside down on top of the cake pan. Using potholders, grasp the side lips of the pan with your fingers and the platter with your thumbs. Invert! If the cake resists releasing give the inverted pan a gently up-and-down shake and/or knock gently on its underside to give it an assist.
Cool cake for 20-30 minutes and serve warm or at room temperature. For extra decadence, serve with ice cream, whipped cream, or creme fraiche.