Preheat oven to 350 degrees F (175 degrees C). Line muffin cups with paper liners.

Beat cake mix, chai tea, vegetable oil, and eggs together in a bowl using an electric mixer on low speed for 30 seconds. Increase speed to medium and beat until batter is smooth, about 2 minutes more. Spoon batter into the prepared muffin cups, filling 2/3-full.

Bake in the preheated oven until a toothpick inserted in the center comes out clean, 22 to 24 minutes. Cool cupcakes in the tin for 5 minutes before transferring to a wire rack to cool completely.

Beat butter and cream cheese together in a bowl using an electric mixer on medium speed until smooth, 20 to 30 seconds. Add confectioners' sugar, vanilla extract, and nutmeg and beat on medium-high speed until smooth, about 1 minute. Scrape sides of bowl as needed. Add eggnog and beat until fully incorporated into the frosting; beat in rum extract on medium-low speed.

Transfer frosting to a pastry bag or plastic bag with a corner snipped and pipe frosting onto cooled cupcakes.