Preheat oven to 350 degrees F (175 degrees C). Lightly grease and flour two 9 inch square cake pans.
In the top of a double boiler, combine the cut up candy bars, water, and peanut butter. Heat and a stir until well melted and blended. Set mixture aside to cool.
Mix together the flour, baking soda, and salt. Set aside.
Separate the egg whites from the yolks. Beat the egg whites until stiff peaks are formed. Set aside.
In a large bowl, cream butter or margarine thoroughly. Gradually add the sugar and beat in the 4 unbeaten egg yolks, vanilla, and cooled candy bar mixture. Mix until smooth and add 1/4 cup of the buttermilk. Stir in the flour mixture alternately with the remaining 1 cup of buttermilk, mix only until just blended. Gently fold in the stiffly beaten egg whites. Pour batter into prepared pans.
Bake at 350 degrees F (175 degrees C) for 45 minutes. Let cakes cool in pans for 5 minutes then turn out onto a cake rack to finish cooling. Serve with ice cream or whipped cream, if desired.