The first night I didn't care -- it's more if it's going to help your family feel better to see you. They can maybe sleep easier (depending on their personality) if they lay eyes on you and see that you are not crying out in pain, are resting peacefully, etc. My aunt did that and eventually went back to Seton to sleep that night. As for during the surgery, in the age of cell phones, they don't need to be "right there" (you won't be awake anyway obviously). Since I was out of town it wasn't really an issue, but I didn't feel a need for a full waiting room!
(The next two nights, when I was out of ICU but still pretty high maintenance, my aunt slept in the room with me and that was really nice. I was a little fevered and felt almost panicky sometimes when I woke up, and knowing I could ask someone for ice, or to help me get a nurse -- to just be able to close the outer door for some quiet, yet not be alone, was priceless. But I'm a first-time patient -- once I had confidence that the nurses were prompt, and I felt sentient enough to ask for what I needed, I was fine alone.)