Preheat oven to 350 degrees F (175 degrees C). Grease two 9-inch round cake pans.

Sift together cake flour, baking powder, soda, salt, cinnamon, allspice, and ginger.

Mix together buttermilk, sweet potatoes, and raisins in a medium bowl and set aside.

Beat butter, brown sugar, and white sugar with an electric mixer in a large bowl until light and fluffy. The mixture should be noticeably lighter in color. Add room-temperature eggs one at a time, allowing each egg to blend into the butter mixture before adding the next. Pour in the flour mixture alternately with the milk and sweet potato mixture, mixing until just incorporated. Divide batter between prepared pans and spread evenly.

Bake in preheated oven until a tester inserted in the center of each layer comes out clean, about 30 minutes. Cool cake layers on wire rack. Remove from pans and frost as desired.