Preheat the oven to 350 degrees F (375 degrees C). Lightly butter an 8-inch baking pan.
Arrange enough apple wedges to form a single layer over the bottom and sides of the prepared pan.
Heat a saucepan over medium heat. Add almonds; cook and stir until toasted, 4 to 5 minutes. Remove from heat.
Beat 3 1/2 tablespoons butter, white sugar, and 1/4 cup brown sugar together in a bowl using an electric mixer until smooth. Add egg, applesauce, and honey; beat until smooth. Stir buttermilk, almond extract, and vanilla extract into the batter.
Combine flour, baking soda, and 1/2 teaspoon salt in a bowl; stir mixture into the batter. Mix until well combined. Fold in 3/4 of the remaining apple wedges. Pour batter into the prepared pan. Top with 2 tablespoons toasted almonds.
Bake the preheated oven for 35 minutes.
Heat oats in a saucepan over medium heat; cook and stir until slightly toasted, about 5 minutes.
Stir 3 tablespoons butter and 1/3 cup brown sugar together in a bowl until combined. Add remaining almond slices, oats, milk, and 1 pinch salt.
Top the cake with remaining apple slices and oat mixture. Return to the oven and continue baking until a toothpick inserted into the center of the cake comes out clean, 10 to 15 minutes more. Let cool to room temperature, at least 25 minutes.