1.2.1 Convert Camera Normals to World Normals

This is an optional step, but for some use cases, it may be better to use world normals instead of camera normals, which are provided by default in normals_sfm_manual202204. To convert, we provide camera rotation matrices in raw_data/camera_matrics.tar.gz for all 8 scenes that also have SfM.

In order to convert, use this function (for each npz file):

def convert_camera_to_world(normals: np.ndarray, rotation_matrix: np.ndarray) -> np.ndarray:
  normals = (normals.copy() - 0.5) * 2 # [-1:1] -> [0:1]
  camera_normals = camera_normals @ np.linalg.inv(rotation_matrix)
  camera_normals = (camera_normals / 2) + 0.5 # [0:1] => [-1:1]
  return np.clip(camera_normals, 0.0, 1.0)

1.2.2 Raw camera location

The GPS location (lat/long/height) is in the raw_data/raw_camera_info.tar.gz. Each file over there is an archive over the original videos, so the train/val/test splits files from raw_data/txt_files must be used to map