Combine the butter, cream cheese, and vanilla; beat until light and fluffy.
Add the sugar gradually, beating to combine.
Stir in the nuts and/or ginger. Add milk, a little at a time, until the frosting is a spreadable consistency.
Serve cake at room temperature. Frost just before serving.