For example, if a check is written "Pay to the order of John Smith," it means that the check should be paid to John Smith and no one else. If someone other than John Smith tries to cash the check, the bank will not honor it.
The phrase "pay to the order of" is often used in conjunction with the phrase "for [purpose]." For example, a check might be written "Pay to the order of John Smith for rent." This indicates that the payment is being made to John Smith for the purpose of paying rent.