If you have this situation, most likely it means the Enable Layered PopUp Style option is activated.
To check it go to Design tab -> Popup Location.
Layered style option allows you to show your PopUp – on selected position: top, bottom, etc. and not over your site – but right near your content. So you can work with your site and popup will be opened all time, like part of it.
For this reason Background overlay opacity will not work.
If you want to show popup in the center position of your site – not necessarily to enable this option. By default – your popup will be opened in the center.