1. Rmi Cresson Dveloppement des chanes de traitement d'images GEOSUD criture parallle de fichiers GeoTiff 2. Rmi Cresson Pipeline = graphe compos de : Data objects Image (rasters, ...) Mesh (shape, ...) Process Source (readers, ...) Filter Mapper (writers, ... ) Terminologie d'un pipeline 3. Rmi Cresson Composant clef : Writer otb::ImageFileWriter crit un fichier unique Streaming paramtrable (taille/forme des dalles) Supporte des noms de fichiers tendus. Exemple : Path/Image.ext? &key1=&key2= Box Compression JPEG2000 ... 4. Rmi Cresson Paralllisation d'un pipeline streamable Avant : Reader WriterBandMath Threshold Reader MPIWriterBandMath Threshold Aprs : 5. Rmi Cresson Writer squentiel avec passage de message Reader MPIWriterBandMath Threshold Reader MPIWriterBandMath Threshold Reader MPIWriterBandMath Threshold Reader MPIWriterBandMath Threshold Process1 Process2 Process3 Process4 MPI_SEND tif 6. Rmi Cresson tif Process1 Process2 Process3 Process4 t read processing write com MPI_RECV() MPI_SEND() Lgende 7. Rmi Cresson MPIWriter MPIWriter MPIWriter Writer simultan avec passage de message Reader MPIWriterBandMath Threshold Reader BandMath Threshold Reader BandMath Threshold Reader BandMath Threshold Process1 Process2 Process3 Process4 tif1 tif2 tif3 tif4 8. Rmi Cresson tif1 Process1 Process2 Process3 Process4 t read processing write Lgende tif2 tif3 tif4 9. Rmi Cresson MPIWriter MPIWriter MPIWriter Writer simultan avec criture parallle Reader MPIWriterBandMath Threshold Reader BandMath Threshold Reader BandMath Threshold Reader BandMath Threshold Process1 Process2 Process3 Process4 tif MPI_FILE_ WRITE_AT 10. Rmi Cresson tif Process1 Process2 Process3 Process4 t read processing write Lgende Uniquement GeoTiff non compress 11. Rmi Cresson Exemple du pipeline prcdent : dtection de l'eau sur une image SPOT5 //Dclarationdestypesdesdataobjects typedefOTBVectorImageSPOTImageType; typedefOTBImageNDVIImageType; typedefOTBImageMaskImageType; //Dclarationdesprocessobjects OTBImageReaderreader OTBBandMathFilterbandMathFilter ITKBinaryThresholdFilterthresholdFilter OTBImageWriterwriter //Paramtragedesprocessobjects reader>SetFileName('image_spot_ms.tif'); bandMathFilter>SetExpression('(b1b2)/(b1+b2)'); thresholdFilter>SetLowerThreshold(1); thresholdFilter>SetUpperThreshold(0); thresholdFilter>SetInsideValue(true); thresholdFilter>SetOutsideValue(false); writer>SetFileName('detection_eau.tif'); //Chainagedesprocessobjects writer>SetInput(thresholdFilter>GetOutput()); thresholdFilter>SetInput(bandMathFilter>GetOutput()); bandmathFilter>SetInput(reader>GetInput()); //Dclenchementdupipeline writer>Update(); Pour le dveloppeur : avant 12. Rmi Cresson Exemple du pipeline prcdent : dtection de l'eau sur une image SPOT5 //Dclarationdestypesdesdataobjects typedefOTBVectorImageSPOTImageType; typedefOTBImageNDVIImageType; typedefOTBImageMaskImageType; //Dclarationdesprocessobjects OTBImageReaderreader OTBBandMathFilterbandMathFilter ITKBinaryThresholdFilterthresholdFilter OTBWithMPIImageWriterwriter //Paramtragedesprocessobjects reader>SetFileName('image_spot_ms.tif'); bandMathFilter>SetExpression('(b1b2)/(b1+b2)'); thresholdFilter>SetLowerThreshold(1); thresholdFilter>SetUpperThreshold(0); thresholdFilter>SetInsideValue(true); thresholdFilter>SetOutsideValue(false); writer>SetFileName('detection_eau.tif'); //Chainagedesprocessobjects writer>SetInput(thresholdFilter>GetOutput()); thresholdFilter>SetInput(bandMathFilter>GetOutput()); bandmathFilter>SetInput(reader>GetInput()); //Dclenchementdupipeline writer>SetNumberOfMPIProcesses(MPI_NPROC); writer>SetCurrentMPIProcessId(MPI_RANK); writer>Update(); Pour le dveloppeur : aprs* 13. Rmi Cresson Autres composants parallliser Utiliss actuellement : Composants de calcul de statistiques (moy, std, min, max) sur une image: communication des sommes, des sommes des carrs, min, max de chaque instance gtb::StatisticsImageFilter gtb::StatisticalRegressionImageFilter gtb::StreamingStatisticsMosaicFilter - Fait Autres composants utiliss dans le futur ? (e.g. chanes de traitement pas encore dveloppes : segmentation, ...) 14. Rmi Cresson Paralllisation d'un pipeline Avant : Reader WriterBandMath SSMF Reader MPIWriterBandMath SSMFMPI Aprs : 15. Rmi Cresson MPIWriter MPIWriter Reader MPIWriterBandMath SSMFMPI Reader BandMath Reader BandMath Process1 Process2 Process3 gtb::StreamingStatistics MosaicFilterMPI SSMFMPI SSMFMPI MPI_ALLGATHER 16. Rmi Cresson Pour le dveloppeur : avant ... mosaicImageStatisticsFilter>SetInput(bandMathFilter>GetOutput()); writer>SetInput(mosaicImageStatisticsFilter>GetOutput()); writer>Update(); 17. Rmi Cresson ... mosaicImageStatisticsFilter>SetInput(bandMathFilter>GetOutput()); mosaicImageStatisticsFilter>SetNumberOfMPIProcesses(MPI_NPROC); mosaicImageStatisticsFilter>SetCurrentMPIProcessId(MPI_RANK); writer>SetInput(mosaicImageStatisticsFilter>GetOutput()); writer>SetNumberOfMPIProcesses(MPI_NPROC); writer>SetCurrentMPIProcessId(MPI_RANK); writer>Update(); Pour le dveloppeur : aprs 18. Rmi Cresson Performances 19. Rmi Cresson Inter-oprabilit Serveur de calcul, architecture HPC hbergeant les chanes de traitements (excutables) Serveur WPS Client WPS Utilisateur