Preheat your oven to 350°F (175°C). Grease and flour a 9×13-inch baking pan.
In a large bowl, whisk together the mashed sweet potatoes, granulated sugar, brown sugar, vegetable oil, eggs, buttermilk, and vanilla extract until smooth and well combined.
In a separate bowl, sift together the flour, baking powder, baking soda, salt, cinnamon, nutmeg, and ginger.
Gradually add the dry ingredients to the wet ingredients, mixing until just combined.
In a small bowl, mix together the granulated sugar and ground cinnamon.
Pour half of the cake batter into the prepared baking pan. Sprinkle half of the cinnamon-sugar mixture evenly over the batter.
Spoon the remaining batter over the cinnamon-sugar layer and gently swirl with a knife or toothpick to create a marble effect. Sprinkle the remaining cinnamon-sugar mixture on top.
Bake in the preheated oven for 35-40 minutes, or until a toothpick inserted into the center comes out clean.
Allow the cake to cool completely in the pan on a wire rack.
In a small bowl, whisk together the powdered sugar, milk, and vanilla extract until smooth.
Drizzle the glaze over the cooled cake.