Training neural network to generate realistic terrain for video games

I’ve recently had an idea to create a tool that makes it easier for environment artists to generate highly realistic terrain for video games. I’ve seen approaches using GANs and I’m familiar with the publication by Guérin et al "Interactive Example-Based Terrain Authoring with Conditional Generative Adversarial Networks".

However I don’t like the limited level of control the proposed solution gives and the fact that they had to artificially generate terrain sketches later used for training. What I’d like to have would be the possibility to use different brushes to place different objects – lakes, valleys, roads etc and a separate texture which roughly represents the desired height and erosion. For that I’d need a dataset of high resolution aerial maps of terrain and correlated elevation maps. I’d be very thankful for insights (if my approach makes sense at all) and some help finding such dataset – so far I’ve been failing to find it and either there’s no such dataset or I’m looking in the wrong place.