Preheat oven to 350 degrees F (175 degrees C). Grease and lightly flour 2 loaf pans.
Combine flour, white sugar, orange zest, baking soda, nutmeg, cinnamon, salt, and ginger in a bowl.
Beat eggs in a separate bowl using an electric mixer until soft peaks form. Add pumpkin, oil, milk, and 1/4 cup orange juice; beat on medium speed until combined. Add the flour mixture in 2 batches, blending well after each addition. Pour batter into the prepared pans.
Bake in the preheated oven until a toothpick inserted into the center comes out clean, about 1 hour. Cool slightly, about 5 minutes. Turn loaves onto a rack to finish cooling, about 20 minutes more.
Mix confectioners' sugar, lemon juice, and 2 teaspoons orange juice until desired consistency is reached. Drizzle over the bread. Top with freshly grated cinnamon.