Beat white sugar and butter together in a bowl using an electric mixer until smooth and creamy. Add eggs and vanilla extract and beat until well mixed. Stir flour, baking powder, and salt into creamed butter mixture until dough is evenly mixed. Chill dough in refrigerator, 3 hours to overnight.
Preheat oven to 350 degrees F (175 degrees C). Line a baking sheet with parchment paper.
Roll cookie dough on a floured work surface. Cut dough into strips about 2 1/2 inches long and about as thick as a pencil, forming one end into a point. Arrange "pencils" on the prepared baking sheet.
Bake in the preheated oven until lightly browned, 8 to 10 minutes. Transfer parchment paper with cookies to a wire rack to cool completely.
Mix enough confectioners' sugar and lemon juice together in a bowl to form a thick icing. Divide icing among several small bowls and mix different colors of food coloring into each bowl.
Decorate "pencil" cookies with colored icing so they look like colored pencils. Allow icing to dry completely.