Melt butter in a medium saucepan over medium heat. Stir in brown sugar and milk. Bring to a boil, boil for 2 minutes, stirring often. Remove from heat. Stir in peanut butter and vanilla. Pour into confectioners' sugar in a large bowl. Mix until smooth. Pour into an 8x8 inch pan. Chill until firm (about an hour) and cut into squares.
To make more fudge, double the recipe and use a 9x13 pan.