Should I avoid tour groups?

Here's the situation: I paid my fee to get into the palace in Madrid. I am
walking around, bothering nobody, when suddenly a tour guide says to me
"Why are you listening to my presentation?" It's not as if I was in her
group or hanging on to the group and following them from room to room. I was
going at my own pace. She acted as if I should immediately leave the area if
she started to talk. A bit later, I wandered into the room with the musical
instruments and was looking at the violins. She ushered her group into the
room and gave me the evil stare. I didn't leave. I liked the grouping of
instruments that were made to look identical with the ivory inlays and saw
no reason to leave the room. I was there first!

I have noticed this territorial attempt to dominate everyone from other tour
guides. So, should I leave the room if one of them moves her group into the
same room?

Pat