Venice2020 Building Heights Detection

Introduction

The main goal of our project is to obtain the height information of buildings in Venice. In order to achieve this goal, we construct a point cloud model of Venice with drone video from youtube and google earth with the help of a photogrammetry tool developed at DHLAB.

Motivation

Building height is one of the most important information to consider for urban planning, economic analysis, digital twin implementation, and so on. For Time Machine implementation, it is important to compare different historical data and current data based on different times. Drones and Google Maps provide 3D views of buildings and surroundings in detail which are useful resources to keep track of changes in a target place. In this project, we aim to detect the building heights of Venice. Many geographical areas, which are either not famous or not accessible easily are accessible by Google Earth 3D views. We aim to make a 3D model of Venice with every detail of the city and calculate point clouds to detect the heights of the city. This information can be used to understand current details of the city and also will be useful in the future as historical data to compare with.

Methodology

Photogrammetry

Once the images of different parts of Venice are prepared, we use a photogrammetric reconstruction tool to generate point cloud models. Photogrammetry is the art and science of extracting 3D information from photographs.

Images Aquisition

Point cloud denotes a 3D content representation which is commonly preferred due to its high efficiency and relatively low complexity in acquisition, storage and rendering of 3D models. In order to generate Venice dense point cloud models, sequential images from different angles in different locations need to be collected first based on the following two sources.

Youtube Video Method

Our initial plan was to download drone videos of Venice on Youtube platform, used FFmpeg to extract images of 1fps, and generate point cloud models based on the images that we obtained. However, the videos we could get from drone videos are very limited: only a few of the videos are of high quality.Among those qualified videos, most of the videos only focused on monumental architecture such as St Mark's Square, St. Mark's Basilica and Doge's Palace. Therefore, a large part of Venice will be omitted if we only adopt Youtube Video Method.

Google Earth Method

In order to access more area of Venice, we decided to use Google Earth as a complementary tool: we could obtain images of whatever building we want, from whatever angle we want. With Google Earth, we have much more access to aerial images of Venice, not only will these images serve as an images set to generate point cloud to calculate building heights, we can also using the overlapping part in different point cloud models to evaluate point cloud models.

Plane Generation

To calculate the height of building, we have to find the plane in the very beginning, which could be realised by a tool provided by Open3D. Open3D supports segmententation of geometric primitives from point clouds using 'RANdom SAmple Consensus'. To find the plane with the largest support in the point cloud, we can use segment_plane by specify 'distance_threshold'.

Point-to-Plane Based Height Calculation

After we obtained the plane equation in the previous step, we could calculate the height of buildings.

In the first step, we calculate the relative height of the buildings based on the equation of distance from point to plane. The coordinate information of each point could be accessed by the 'points' property of ‘PointCloud Class’, and we already have the plane equation, using this formula, we could obtain the distance between each point and the plane.

In the second step, we use mapping to calculate the real height of the buildings. For instance, we know the real height of St. Mark Basilica, and we already calculated the relative height of St. Mark Basilica, therefore, we could obtain the scale factor of this point cloud model. For all the other buildings in the model, the real height = the relative height * scale factor.

Milestones

Milestone 1

• Get familiar with OpenMVG, Open3D, OpenGM, Point Cloud Library, Agisoft Metashape, Blender, CloudCompare and QGIS
• Collect high resolution venice drone videos on youtube and record Google Earth birdview videos as supplementary materials

Milestone 2

• Align photos of the same location in Venice, derive sparse point clouds made up of only high-quality tie points and repeated optimize the camara model by reconstruction uncertainty filtering, projection accuracy filtering and reprojection error filtering.
• Build dense point clouds using the estimated camera positions generated during sparse point cloud based matching and the depth map for each camera
• Evaluate point clouds objective quality and select high quality models by point-based approaches which can assess both geometry and color distortions

Milestone 3

• Align and compose dense point clouds of different spots to generate an integrated Venice dense cloud model (implement registration with partially overlapping terrestrial point clouds)
• Build Venice 3D model (mesh) and tilted model according to dense point cloud data
• Generate plane of the surface ground

Milestone 4

• Build the digital elevation model of Venice and align the model with open street map and cadaster in the 16th century to obtain building heights
• Assess the accuracy of the building heights detection

Planning

Week	Tasks	Completion
week 5	Brainstorm and present initial ideas for the project	✓
week 6-7	Configure the environment and master the related softwares and libraries Find suitable drone videos on YouTube and obtain photos for further photogrammetry work	✓
week 8-9	Align photos, generate and denoise dense point clouds Assess the quality of point clouds both subjectively and objectively	✓
week 10	Implement ground segmentation Collect screenshots of buildings from Venice on Google Earth Prepare for the midterm presentation	✓
week 11	Calculate the height of buildings and generate map layers with building height information Evaluate the models based on the overlapped parts in the map layer with building height information
week 12	Build 3D model, tiled model of Venice Align the height information layers with OpenStreetMap
week 13	Write final report, refine the map layers with building height information and the final models
week 14	Refine the final report and code Prepare for final presentation