The speaker misses several things, but the most prominent is the comfort and stability of a home and community. He describes himself as "a solitary wanderer, bereft of friends," and he yearns for the days when he was part of a thriving society. He remembers "the joys of the banquet hall" and "the laughter of friends," and these memories only serve to heighten his present sense of isolation.
While the specific details of his past are not explicitly stated, the speaker clearly misses the social bonds he once enjoyed. He yearns for the sense of belonging and purpose that comes from being part of a community.
In addition to missing his home and community, the speaker also seems to miss the comforts of youth. He laments the passing of time and the physical decline that comes with age. He speaks of his "weary heart" and his "gray hairs," and he recognizes that he is not the same person he once was.
Therefore, the speaker of "The Wanderer" seems to miss most the sense of belonging, stability, and purpose that comes from having a home and community, as well as the youthful vigor and physical strength that he has lost over time.