To prepare filling, combine milk and pumpkin in a mixing bowl. Mix until smooth. Add pudding mixes and pumpkin pie spice. Mix well. Fold in whipped topping. Spoon into crust. Refrigerate for four hours, or until set. Meanwhile, to prepare topping, combine walnuts, brown sugar, and margarine in a small bowl. Mix until crumbly. Sprinkle over top chilled pie.