Is it possible to have open source software such that binaries can only be distributed by the official builder?

Not according to the definition of open source we use here, which is the OSI's. That requires that

The license shall not restrict any party from selling or giving away the software

and that

The license must allow modifications and derived works, and must allow them to be distributed under the same terms as the license of the original software

So if I can't modify the program to make my own "unofficial build", and then share it, we don't consider the software to be open source. That said, the term "open source" isn't a restricted one, so anyone can distribute their software under any old hideous proprietary licence and yet claim it's "open source". They can even start DMCA takedowns for violation of their proprietary licence, and still claim with a straight face that their software is open source. To my mind, they're wrong, and their claim is a lie, but they're not forbidden from making it.

However, the idea that you can restrict the use of a trademarked name is much more normal, and acceptable within the free software community. So a site that said, eg, "we distribute the official build of FooBarOS^TM for the Google BazBar Developer Phone; if you distribute your own build for this phone, you must not call it FooBarOS" is fine.