Am I the asshole?

I live in an apartment complex. We have a poster on the door about this cat that lives outside mostly, but his owners are in the building. The poster says not to let him on the stairwell, so that he doesn't pee there. I like cats, I used to always pat Fox (poster cat) when I was outside. He started to come with me, I tried to kick him out, but he wouldn't go. I couldn't let him stay on that stairwell, so I let him inside. Gave him milk. He would drink it and then leave - id let him down the elevator and outside. We have this ritual - he would eat and leave. Then he would come, eat and sleep. But now he doesn't want to leave. I treat him with respect, he goes whenever he pleases, I never keep him inside, but he seems to prefer it in the apartment. I feel like a complete ass. I like this cat, I have formed a bond with him, but I also feel like I’m doing something wrong, even though he always has his freedom to go