Preheat oven to 350 degrees F (175 degrees C). Grease and flour two 9-inch round cake pans; line bottoms with greased parchment paper.
Combine milk and 2 tablespoons orange juice in a bowl; set aside at room temperature for at least 10 minutes.
Mix orange zest, flour, baking powder, baking soda, allspice, cinnamon, and sea salt together in a bowl.
Beat white sugar, butter, and brown sugar together in a large bowl using an electric mixer on high speed until creamy. Reduce speed to low; slowly add 1/3 of the beaten eggs and mix until well incorporated. Repeat process until all eggs are added. Add sweet potatoes and almond extract; mix until smooth.
Stir flour mixture into butter mixture 1/3 at a time, alternating with milk-juice mixture. Beat on low speed until just incorporated. Divide batter evenly between prepared pans.
Bake in the preheated oven until a toothpick inserted in the center comes out clean, 35 to 45 minutes. Cool in pans for 10 minutes; invert onto rack to cool completely, about 15 minutes.