They do have the authority to do that. They ask you to leave. If you say no then you're trespassing and can be physically removed.

How do you think bouncers work?