Preheat the oven to 160°C/ gas mark 3. Grease and line the base of a 17cm cake tin with non-stick baking paper.
Beat the butter and sugar until creamy and pale, then beat in the eggs one at a time. Fold in the ground almonds, polenta, lemon zest and juice, baking powder, salt and vanilla extract and half of the caramel fudge pieces.
Pour the batter into the prepared tin and bake in the oven for 45 minutes, until golden and just firm in the middle.
Meanwhile, to make the butterscotch sauce, melt the butter in a large saucepan, then add the brown sugar. Allow this mixture to come to the boil, then pour in the cream and whisk continuously. Bring to the boil again, then reduce the heat and simmer for 5 minutes, stirring frequently. Remove from the heat and leave to cool.
When cooled, drizzle the sauce over the polenta cake and top with some chopped fudge and a few physalis. Serve with extra physalis and a few pieces of fudge.