To make the cake:
Preheat the oven to 350 degrees. Lightly grease 2 8” or 9” round cake pans and line the bottoms with a round of parchment paper.
In a large bowl or the bowl of a stand mixer, cream the butter and sugar on medium speed for 5 minutes. Scrape the sides of the bowl and add the egg and egg yolks one at a time, mixing until combined after each addition. Add the vanilla. Scrape the sides of the bowl and stir in half of the dry ingredients on low speed. Add half of the milk and scrape the bowl again. Repeat this process with the remaining dry ingredients and milk. Fold in any unincorporated bits and divide the batter between the two pans. Bake in the preheated oven for about 35-37 minutes or until a cake tester inserted into the center comes out clean. Allow to cool completely.
To prepare the almond toffee:
Prepare a large rimmed sheet pan with a sheet of aluminum foil that is greased with butter. Set aside. In a medium-sized heavy bottomed saucepan melt the butter over medium-high heat. Add the sugars and stir until dissolved. Reduce the heat to medium low and cook, stirring constantly, until a candy thermometer reads a temperature of 300 degrees. Immediately remove the pan from the heat and stir in the almonds. Spread the mixture into a layer on the buttered foil and allow to cool completely.
To prepare the frosting:
Pour the chopped chocolate/chips into a large mixing bowl. In a saucepan over medium heat, warm the cream and brown sugar together until the milk is almost starting to bubble around the edges and the sugar is dissolved. Pour the cream over top of the chocolate and cover the bowl with a sheet of plastic wrap. Allow to rest 5 minutes and then whisk to combine into a smooth and thick chocolate mixture. Stir in the butter and allow the mixture to set. I like to put the whole bowl into the fridge and stir it every 10 minutes until the mixture is a thick, fudge-like consistency. Once it’s thick enough to spread, remove from the fridge and prepare to frost your cake! If you happen to over-chill your ganache, just heat it barely in the microwave until it’s loosened up. Re-whisk and allow it to thicken up again if needed.
To assemble the cake:
Level the tops of the cakes with a serrated knife. Line a cake stand or serving plate with a cardboard cake round and spread a little of the ganache frosting into the center of the board. Place your first cake layer in the center and spread about 1/3-1/2 cup ganache on top. Smooth and sprinkle the crumbled toffee on top. Dollop a few small dots of ganache on top of the toffee and press the next cake layer down on top. Smooth more ganache on top and spread it all over the cake. You can place your cake in the fridge to set up. Sprinkle the top of the cake with more almond toffee and serve with additional chopped almonds. Store the cake covered in plastic wrap and keep the toffee tightly sealed. Cake is best enjoyed within two to three days.