In a 2 quart saucepan, combine 1/2 cup butter, brown sugar, corn syrup and sweetened condensed milk. Bring to a boil, stirring constantly. Continue to boil for 5 minutes. Remove from heat and beat vigorously with a wooden spoon for 3 minutes. Pour over warm or cool baked crust. Cool until it begins to firm.