- Document Number:
20240386528
- Appl. No:
18/666438
- Application Filed:
May 16, 2024
- نبذة مختصرة :
This disclosure relates generally to image processing and specifically to processing panorama images using neural networks to generate depth maps, layouts, semantic maps or the like with reduced distortion and improved continuity. Methods and systems are described for generating such maps by leveraging several essential properties of these panorama images and by using a panorama panel representation and a neural network framework. A panel geometry embedding network is incorporated for encoding both the local and global geometric features of the panels in order to reduce negative impact of panoramic distortion. A local-to-global transformer network is also incorporated for capturing geometric context and aggregating local information within a panel and panel-wise global context.
- Assignees:
TENCENT AMERICA LLC (Palo Alto, CA, US)
- Claim:
1. A method for processing a panorama image dataset by a computing circuitry, comprising: generating a plurality of data panels from the panorama image dataset; executing a first neural network to process the plurality of data panels to generate a set of embeddings representing geometric features of the plurality of data panels; executing a second neural network to process the plurality of data panels and the set of embeddings to generate a plurality of mapping panels; and fusing the plurality of mapping panels into a mapping dataset of the panorama image dataset.
- Claim:
2. The method of claim 1, wherein the mapping dataset comprises one of a depth map, a layout map, or a semantic map corresponding to the panorama image dataset.
- Claim:
3. The method of claim 1, wherein: the panorama image dataset comprises a data array in two dimensions; and the each of the plurality of data panels comprises a subarray of the data array in an entirety of a first dimension of the two dimensions and a segment of a second dimension of the two dimensions.
- Claim:
4. The method of claim 3, wherein the first dimension represents a gravitational direction of the panorama image dataset and the second dimension represents a horizontal direction of the panorama image dataset.
- Claim:
5. The method of claim 3, wherein the plurality of data panels are generated from the panorama image dataset consecutively using a window having a length of the entirety of the first dimension in the first dimension and a predefined width in the second dimension, the window sliding along the second dimension by a predefined stride.
- Claim:
6. The method of claim 5, wherein the window continuously slides across from one edge of the panorama image dataset in the second dimension into another edge of the panorama image dataset in the second dimension.
- Claim:
7. The method of claim 3, wherein the first neural network is configured to encode local and global geometric features of the plurality of data panels to reduce impact of geometric distortions in the panorama image dataset and to enhance preservation of geometric continuity across the plurality of mapping panels.
- Claim:
8. The method of claim 3, wherein the first neural network comprises a multilayer perceptron (MLP) network LP for processing geometric information extracted from the plurality of data panels to generate the set of embeddings comprising a set of global geometric features and a set of local geometric features of the plurality of the data panels.
- Claim:
9. The method of claim 8, wherein the second neural network is configured to process the plurality of data panels and reduce geometric distortions in the panorama image dataset based on the set of embeddings.
- Claim:
10. The method of claim 8, wherein the second neural network comprises: a down-sampling network; a transformer network; and an up-sampling network.
- Claim:
11. The method of claim 10, wherein: the down-sampling network is configured for processing the plurality of data panels and the set of embeddings to generate a series of down-sampled features with decreasing resolutions; the transformer network is configured for processing lowest resolution down-sampled features to generate transformed low-resolution features; and the up-sampling network is configured for processing the transformed low-resolution features and the series of down-sampled features to generate the plurality of mapping panels.
- Claim:
12. The method of claim 10, wherein the transformer network comprises a feature processor.
- Claim:
13. The method of claim 12, wherein the feature processor is configured to increase continuity of the geometric features.
- Claim:
14. The method of claim 12, wherein the feature processor is configured to aggregate local information within each of the plurality of data panels to capture panel-wise context.
- Claim:
15. An apparatus for processing a panorama image dataset, the apparatus comprising a memory for storing computer instructions and at least one processor for executing the computer instructions to: generate a plurality of data panels from the panorama image dataset; execute a first neural network to process the plurality of data panels to generate a set of embeddings representing geometric features of the plurality of data panels; execute a second neural network to process the plurality of data panels and the set of embeddings to generate a plurality of mapping panels; and fuse the plurality of mapping panels into a mapping dataset of the panorama image dataset.
- Claim:
16. The apparatus of claim 15, wherein: the panorama image dataset comprises a data array in two dimensions; and the each of the plurality of data panels comprises a subarray of the data array in an entirety of a first dimension of the two dimensions and a segment of a second dimension of the two dimensions.
- Claim:
17. The apparatus of claim 16, wherein: the plurality of data panels are generated from the panorama image dataset consecutively using a window having a length of the entirety of the first dimension in the first dimension and a predefined width in the second dimension, the window continuously sliding along the second dimension by a predefined stride and across from one edge of the panorama image dataset in the second dimension into another edge of the panorama image dataset in the second dimension; and the first neural network is configured to encode local and global geometric features of the plurality of data panels to reduce impact of geometric distortions in the panorama image dataset and to enhance preservation of geometric continuity across the plurality of mapping panels.
- Claim:
18. The apparatus of claim 16, wherein: the first neural network comprises a multilayer perceptron (MLP) network LP for processing geometric information extracted from the plurality of data panels to generate the set of embeddings comprising a set of global geometric features and a set of local geometric features of the plurality of the data panels; the second neural network comprises a down-sampling network, a transformer network, and an up-sampling network; the down-sampling network is configured for processing the plurality of data panels and the set of embeddings to generate a series of down-sampled features with decreasing resolutions; the transformer network is configured for processing lowest resolution down-sampled features to generate transformed low-resolution features; and the up-sampling network is configured for processing the transformed low-resolution features and the series of down-sampled features to generate the plurality of mapping panels.
- Claim:
19. The apparatus of claim 18, wherein: the transformer network comprises a feature processor; and the feature processor is configured to increase continuity of the geometric features and to aggregate local information within each of the plurality of data panels to capture panel-wise context.
- Claim:
20. A non-transitory computer readable medium for storing instructions, the instructions, when executed by at least one processor, are configured to cause the processor to process a panorama image dataset by: generating a plurality of data panels from the panorama image dataset; executing a first neural network to process the plurality of data panels to generate a set of embeddings representing geometric features of the plurality of data panels; executing a second neural network to process the plurality of data panels and the set of embeddings to generate a plurality of mapping panels; and fusing the plurality of mapping panels into a mapping dataset of the panorama image dataset.
- Current International Class:
06; 06; 06; 06; 06
- الرقم المعرف:
edspap.20240386528
Processing Request
No Comments.