Heat the oven to 300ºF. Place 1 (1-pound) loaf cubed challah or brioche in an even layer on a rimmed baking sheet.

Bake until the bread is dry but not browned, about 20 minutes. Set aside until mostly cool. Meanwhile, make the apple filling.

Melt 3 tablespoons unsalted butter in a large skillet over medium-high heat. Stir in ½ cup packed light brown sugar, 2 teaspoons ground cinnamon, ¾ teaspoon ground nutmeg, and 1 pinch kosher salt. Continue to stir until the sugar is dissolved.

Add 3 peeled and diced medium apples and cook, stirring occasionally, until the apples are beginning to soften and the sauce is thickened, about 3 minutes. Turn off the heat and set aside until mostly cool. Meanwhile, increase the oven temperature to 350℉. Lightly coat a 9x13-inch baking dish with unsalted butter.

Whisk 4 large eggs and 2 large egg yolks in a large bowl to break up. Whisk in 3 cups whole milk, 2 tablespoons vanilla extract, and ½ teaspoon kosher salt until combined.

Gently stir in the bread and press the bread down slightly to soak up the custard. Stir in the apple mixture. Let sit for 15 minutes, gently stirring once halfway through, to allow the bread to soak up the custard. Transfer to the baking dish, arrange into an even layer, and pour any excess custard over the top.

Bake until puffed, golden brown, the bread pulls away from the edges of the pan, and the custard is set, 45 to 50 minutes. Let cool for at least 5 minutes before serving. Serve topped with vanilla ice cream and caramel sauce if desired.