In this poem, the "scholar" is a recluse who spends his days in solitary study. The scholar's own eyes are the ones that are "cast" in the poem, meaning they are fixed on his books and studies.
The relevant lines are:
> "To muse and brood and live again in dreams,
> With those old faces of the days gone by,
> And evermore upon the silent stream
> To cast his eyes."
This suggests that the scholar's eyes are constantly looking inward, reflecting on the past and lost moments. He's absorbed in his thoughts and studies, casting his eyes on the "silent stream" of his own mind.