I have seen two Joseph Losey works 'The Assassination of Trotsky' and 'Monsieur Klein', so compared to them 'The Servant' looks like a small production. In the beginning of the movie, it even has an atmosphere of a stage production. I guess it's because of the sound system, which I think is poor.
Dirk Bogarde plays Barrett, who in the beginning of the movie arrives at Tony's big, empty house in London, to become his servant. Tony is a rich aristocrat who does nothing in life and Barrett becomes a perfect servant for him. Barrett decorates the house, cooks, cleans, ... everything. However, Tony's fiancée, Susan, doesn't like Barrett. When Barrett brings his sister Vera to the house, she seduces Tony and they have an affair. One night, Tony and Susan drops by the house without warning and they find Barrett & Vera on Tony's bed. Then Barrett tells Tony that Vera is his fiancée. Tony sacks Barrett & Vera, but Susan also leaves him. Tony leads a troubled life until he meets Barrett again in a bar when Barrett asks him for a second chance. Tony hires him again, but this time, realizing that Tony cannot live without the servant, the servant takes over the control. Tony may own the house, but Barrett takes care of it. In the end, they live not as master-servant, but like two equals who can fight sometimes. It's even like that weak Tony succumbs to his servant.
Both main male actors, Dirk Bogarde and James Fox (who plays Tony), are very good. Barrett is polite and competent, a perfect English butler; but he becomes a threat when possible - and it's very possible with a master like useless Tony. The evil inside Barrett is depicted well. Wendy Craig who plays Susan is okay, but Sarah Miles's Vera is often annoying. At times Vera does look very beautiful, but in wild scenes with Barrett, her laugh and attitude are very annoying. I believe she is not that uneducated if she is clever enough to seduce her master and make him believe that Barrett doesn't know about it. In scenes with Tony, it seems she calculates well her moves; but with Barrett, she doesn't have manners.
It isn't clear to me why Barrett asks Vera to seduce Tony. At first I think they are going to blackmail him, but it only succeeds to break Tony's engagement with Susan and, of course, corrupts him more.