See how he reacts when you ask him what he’s thinking, feeling, wondering about. If he immediately backs off, then you have a problem.
Even if it’s not literally all you do, if you feel that way, then you have to follow your gut.
Sure, he may tell you that he has to wake up wicked early. But then why did he wait until 1 a. m. to come over to your place?
Check out his level of affection. Does he ever want to snuggle, cuddle, or hug just because, without any sexual connection? If not, then it may be because he wants one thing only.
Guess what? There’s plenty more where he came from. Move on and locate your self-esteem; put it back where it belongs, reclaiming your self-respect and keeping your sexuality, physical health, emotional sanity, and sense intact.