Beat butter, brown sugar, and white sugar with an electric mixer in a large bowl until light and fluffy. The mixture should be noticeably lighter in color. Add room-temperature eggs one at a time, allowing each egg to blend into the butter mixture before adding the next. Pour in the flour mixture alternately with the milk and sweet potato mixture, mixing until just incorporated. Divide batter between prepared pans and spread evenly.