In a medium bowl, combine graham cracker crumbs, sugar, butter and nutmeg. Blend until crumbly. Save out 1/3 of the crumbs to sprinkle on top of the pie if desired. Press remaining crumbs evenly on the bottom and sides of a 9-inch pie plate, making a small rim.