Preheat oven to 350ºF. Place whole apples on a baking sheet and bake until brown and tender, about 25 min. Gently crush the allspice berries and break up the cinnamon stick. Place the allspice, cinnamon, cloves, nutmeg and ginger in a cheesecloth bag. Pour 2 cups of the ginger ale into a large sauce pan and add the spice bag. Bring to a boil, reduce heat and simmer for 10 min. Remove spice bag and add the remaining ginger ale, sherry and brandy. Heat until hot but do not boil. Whisk in milk and pour into a punch bowl. Garnish with slices of baked apple.