Scrub the sweet potatoes, cut in half and place in a large pot of boiling salted water, until fork tender, about 40-50 minutes. Drain, and allow to slightly cool, in order to peel.
While potatoes are boiling, heat a cup of water in a small sauce pan and add the apples. Lightly boil for 10-15 minutes.
Drain the water and add 3 tablespoons of butter, half the brown sugar, and saut the apples for 5 minutes. Add a dash of cinnamon.
Pre-heat oven to 350 degrees.
Peel the potatoes and place in a large bowl. Mash them using a potato masher. Add the apples and continue to mash.
Add the rest of the butter, heated milk, and brandy (optional) to the bowl. Blend well with a mixer. While mixing, add the sugar and a dash of cinnamon and all-spice.
Scoop this mixture into a well-buttered casserole dish, and bake for 30 minutes. Remove from oven and allow this to cool for a few minutes. Top with crushed walnuts or slivered almonds.
