Prepare and bake gingerbread mix according to package directions for cake. Cool completely.
Break cake into crumbles; reserve ¼ cup crumbs. In a large bowl, whisk milk, pudding mixes and spices until mixture is thickened, about 2 minutes. Stir in pumpkin.
In a 3-½ qt. trifle or glass bowl, layer one-fourth of the cake crumbs, half of the pumpkin mixture, one-fourth of the cake crumbs and half of the whipped topping; repeat layers. Top with reserved crumbs. Refrigerate until serving.
