In this sentence, "my brother" and "I" are both objects of the preposition "to." When there are two or more objects of a preposition, they should be joined by the conjunction "and." Therefore, the correct sentence should be "He gave advice to my brother and me."