Arrange a rack in the middle of the oven and heat the oven to 300℉. Place the bread cubes in an even layer on a rimmed baking sheet.

Bake until the bread is dry but not browned, about 20 minutes. Set aside until mostly cool.

Meanwhile, increase the oven temperature to 350℉. Lightly coat a 9x13-inch baking dish with unsalted butter. Melt 3 tablespoons unsalted butter in a small microwave-safe bowl in 20-second bursts until mostly melted, 30 to 40 seconds total. (Alternatively, melt the butter in a small saucepan over medium heat.) Set aside to cool slightly.

Place 4 large eggs and 2 large egg yolks in a large bowl and whisk to break up. Add orange or lemon zest if using, 4 cups whole milk or half-and-half, 1 cup granulated sugar, 2 tablespoons vanilla extract, 1 teaspoon ground cinnamon, ½ teaspoon ground nutmeg, and ¼ teaspoon kosher salt. Whisk until combined.

Add the bread and ½ cup dried fruit or chocolate chips if using. Gently stir to coat the bread and press the bread down slightly to soak up the custard.

Transfer to the baking dish and arrange into an even layer. Pour any remaining liquid over the top. Drizzle the melted butter over the top.

Bake until puffed, golden brown, the bread pulls away from the edges of the pan, and the custard is set, 45 to 50 minutes. Let cool for 5 minutes before serving. Serve topped with powdered sugar or vanilla ice cream if desired.