In a small saucepan over medium-high heat, bring cranberries, sugar, and ½ c. cold water to a simmer. Cook, stirring occasionally, until liquid has reduced into a thickened saucy consistency and cranberries have burst, 12 to 14 minutes.
Add brandy and cook, stirring occasionally, until most of the liquid is evaporated, about 2 minutes more. Transfer to a medium heatproof container or bowl and let cool.
Meanwhile, in a large bowl, using a handheld mixer on low speed (or in the large bowl of a stand mixer fitted with the paddle attachment), beat cream cheese, heavy cream, lemon zest, and salt, increasing speed to high, until smooth and fluffy, about 1 minute.
Dollop cream cheese mixture into a serving bowl. Spoon cranberry sauce over. Sprinkle with almonds and scallions; season with pepper (if using). Drizzle with oil. Serve with crackers alongside.
