New! Sign up for our free email newsletter.
Science News
from research organizations

Images of simulated cities help artificial intelligence to understand real streetscapes

September 14, 2023
Osaka University
To address the lack of suitable training data for deep-learning semantic segmentation models in urban landscaping, researchers developed a method that generates a training dataset without the need for real images or a model of an existing city. The method, which is based on procedural modelling and image-to-image techniques, enables segmentation models to achieve comparable performance under some conditions at a fraction of the cost of real dataset generation.

Recent advances in artificial intelligence and deep learning have revolutionized many industries, and might soon help recreate your neighborhood as well. Given images of a landscape, the analysis of deep-learning models can help urban landscapers visualize plans for redevelopment, thereby improving scenery or preventing costly mistakes.

To accomplish this, however, models must be able to correctly identify and categorize each element in a given image. This step, called instance segmentation, remains challenging for machines owing to a lack of suitable training data. Although it is relatively easy to collect images of a city, generating the 'ground truth', that is, the labels that tell the model if its segmentation is correct, involves painstakingly segmenting each image, often by hand.

Now, to address this problem, researchers at Osaka University have developed a way to train these data-hungry models using computer simulation. First, a realistic 3D city model is used to generate the segmentation ground truth. Then, an image-to-image model generates photorealistic images from the ground truth images. The result is a dataset of realistic images similar to those of an actual city, complete with precisely generated ground-truth labels that do not require manual segmentation.

"Synthetic data have been used in deep learning before," says lead author Takuya Kikuchi. "But most landscape systems rely on 3D models of existing cities, which remain hard to build. We also simulate the city structure, but we do it in a way that still generates effective training data for models in the real world."

After the 3D model of a realistic city is generated procedurally, segmentation images of the city are created with a game engine. Finally, a generative adversarial network, which is a neural network that uses game theory to learn how to generate realistic-looking images, is trained to convert images of shapes into images with realistic city textures This image-to-image model creates the corresponding street-view images.

"This removes the need for datasets of real buildings, which are not publicly available. Moreover, several individual objects can be separated, even if they overlap in the image," explains corresponding author Tomohiro Fukuda. "But most importantly, this approach saves human effort, and the costs associated with that, while still generating good training data."

To prove this, a segmentation model called a 'mask region-based convolutional neural network' was trained on the simulated data and another was trained on real data. The models performed similarly on instances of large, distinct buildings, even though the time to produce the dataset was reduced by 98%.

The researchers plan to see if improvements to the image-to-image model increase performance under more conditions. For now, this approach generates large amounts of data with an impressively low amount of effort. The researchers' achievement will address current and upcoming shortages of training data, reduce costs associated with dataset preparation and help to usher in a new era of deep learning-assisted urban landscaping.

Story Source:

Materials provided by Osaka University. Note: Content may be edited for style and length.

Journal Reference:

  1. Takuya Kikuchi, Tomohiro Fukuda, Nobuyoshi Yabuki. Development of a synthetic dataset generation method for deep learning of real urban landscapes using a 3D model of a non-existing realistic city. Advanced Engineering Informatics, 2023; 58: 102154 DOI: 10.1016/j.aei.2023.102154

Cite This Page:

Osaka University. "Images of simulated cities help artificial intelligence to understand real streetscapes." ScienceDaily. ScienceDaily, 14 September 2023. <>.
Osaka University. (2023, September 14). Images of simulated cities help artificial intelligence to understand real streetscapes. ScienceDaily. Retrieved November 28, 2023 from
Osaka University. "Images of simulated cities help artificial intelligence to understand real streetscapes." ScienceDaily. (accessed November 28, 2023).

Explore More
from ScienceDaily