Beat warm milk and yeast in bowl of a stand mixer fitted with the paddle attachment on low speed, about 30 seconds. Let stand until yeast dissolves and small bubbles appear, about 5 minutes.

Add flour and sugar; beat on low speed until a shaggy dough forms, about 30 seconds.

Separate egg yolk and egg white from 1 egg; set aside. With mixer running on low speed, add remaining 2 whole eggs and reserved egg yolk, 1 at a time, beating until each is incorporated, about 45 seconds total.

Add vanilla, orange blossom water (if using), and salt; increase mixer speed to medium, and beat until dough is thickened, about 2 minutes. (Small lumps are OK.)

Increase mixer speed to medium-high; add butter cubes, a few at a time, beating until cubes are incorporated, about 1 minute. (Dough will be very loose at this point.)

Continue beating on medium-high speed until dough is very smooth, thick, stretchy, and lightened in color, about 5 minutes.

Scrape down and clean sides of bowl using a rubber spatula; gather dough into a round shape. Cover bowl tightly with plastic wrap; set aside in a warm, draft-free place (80°F to 85°F) until almost doubled in volume, 1 hour to 1 hour and 15 minutes.

Lightly grease a 9-inch springform pan with butter. Transfer dough to pan, and spread evenly. Cover pan tightly with plastic wrap; set aside, and let rise again in a warm, draft-free place until bubbles appear on surface, about 30 minutes.

Preheat oven to 350°F. Whisk reserved egg white. Lightly brush egg white on top of dough. Sprinkle top evenly with pearl sugar.

Bake in preheated oven until cake is golden brown and a wooden skewer inserted in center comes out clean, 20 to 25 minutes.

Transfer cake to a wire rack, and let cool 20 minutes. Remove sides of springform pan; let cake cool completely, about 1 hour.

Cut vanilla bean pod in half lengthwise; reserve 1 half for another use. Scrape seeds from remaining pod half; add vanilla bean seeds and pod to a small saucepan. Stir in milk and salt; bring just to a boil over medium high. Remove from heat; let steep 20 minutes. Remove and discard vanilla bean pod.

Whisk together sugar, cornstarch, and egg yolks in a medium bowl. Slowly add about ⅓ cup warm milk mixture to egg mixture, whisking constantly until well combined. Add another ⅓ cup milk mixture, whisking to combine; whisk in remaining milk mixture.

Return milk-egg mixture to saucepan. Cook over medium, whisking constantly, until thickened, 4 to 5 minutes. Continue cooking, whisking constantly, 1 minute. Remove from heat.

Immediately transfer hot pastry cream to a clean medium bowl. Let cool 3 minutes, stirring occasionally. Whisk in cold butter, 1 or 2 pieces at a time, until all butter is incorporated. Cover pastry cream with plastic wrap, pressing directly onto surface. Chill in refrigerator at least 1 hour or up to 3 days.

Whisk heavy cream in a large bowl by hand until firm peaks form, about 3 minutes. Remove pastry cream from refrigerator, and whisk to loosen and smooth any lumps. Stir about half of whipped cream into pastry cream using a spatula. Add remaining whipped cream, and gently fold until just combined.

Slice cake in half horizontally using a serrated knife. Place bottom layer of cake on a platter, reserving top layer. Spread pastry cream mixture over bottom cake layer, leaving a ¼-inch border. Place top cake layer over pastry cream mixture, pressing gently. Chill, loosely covered in plastic wrap, in refrigerator at least 30 minutes or up to 8 hours before serving.