Securing --rpcaddr endpoint on Go Ethereum node

I have been connecting nodes to a PoA blockchain based server and the only way I could get it to work was to specify the server address in the geth connection string as follows –

 --rpcaddr xx.xxx.xx.x

I understand this allows anyone to access the node. It is a private network and it seems to me I have now created a security hole.

Is there a way to actually allow access from my own server but deny access to others? So I want in effect a server IP list and to allow the approving server for new nodes to restrict access to just those IPs.