Preheat your oven to 350° F. Grease and lightly flour 3 round cake pans; set aside.
Bring the water to a boil. Break the chocolate into small pieces, and place in a small bowl. Pour boiling water over chocolate. Stir with a fork until chocolate is completed melted. Set aside to cool.
Separate the eggs, and set aside the yolks. Beat the whites with a mixer on high speed until stiff peaks form; set aside.
Beat together the margarine and white granulated Pioneer Sugar. Slowly add the 4 egg yolks, one at a time, beating well after each. Blend in the melted chocolate and vanilla.
Sift together flour, baking soda and salt. Add to the wet mixture. Add the egg whites and stir well. Slowly add the buttermilk and stir gently until well blended. Pour 1/3 of the batter into each prepared pans.
Bake 30 minutes, or until a toothpick inserted in the centers comes out clean. Run a small spatula around the cake pans, and let cakes cool for 15 minutes on a baking rack. After 15 minutes remove cakes from pans and place on wire racks to completely cool.
In a stainless steel saucepan, combine the white granulated Pioneer Sugar, margarine, evaporated milk, egg yolks and vanilla. Stir constantly over medium heat for approximately 15 minutes, until thickened and golden brown. *DO NOT BOIL*
Remove from the heat and stir in the coconut and nuts. Mix well and let cool on a wire rack to room temperature, or until it is thick enough to spread. When cooled. Frost the top of each cake at a time, stacking each cake on top to form a 3-layer cake. Complete frosting the top layer and serve.