In Shel collection of poetry With Everything on It what is the name poem about two kids with blue faces hiding under masks?

The poem you're describing is called "Blue Faces" from Shel Silverstein's collection "A Light in the Attic," not "With Everything on It."

The poem tells the story of two children with blue faces who hide their faces behind masks, fearing that their appearance will frighten others. However, they realize that their fear is unnecessary and eventually remove their masks, embracing their unique blue faces.