Preheat the oven to 350 degrees F (175 degrees C). Line two 12-cup muffin tins with paper liners.

Whisk flour, baking powder, baking soda, and salt together in a bowl. Mix mashed bananas and sour cream in a separate bowl.

Beat sugar and oil in a large mixing bowl. Add eggs and mix until light and fluffy. Mix in liqueur, then flour mixture. Add banana mixture and mix just until combined, scraping sides of the bowl quickly; don't overmix. Pour batter into the prepared muffin cups, filling 3/4 full.

Bake in the preheated oven until tops are golden brown and a toothpick inserted into the center comes out clean, 25 to 30 minutes. Cool in the tins for 5 minutes. Transfer to a wire rack and let cool, about 20 minutes.

Meanwhile, dissolve instant coffee in hot water in a medium bowl. Stir in confectioners' sugar, oil, and liqueur. Drizzle over cooled muffins.