This tutorial will demonstrate how to setup and run a basic processing scheme in OSARIS. It is assumed that you have a working installation on your machine. Instructions on how to install OSARIS are provided in the in the ReadMe file.
The processing scheme of this tutorial will obtain data for a study area in the vicinity of Bishkek, Kyrgyzstan.
Part I: Acquiring data
Step I.1: First, let’s take a look at what Sentinel-1 data is available for the study area. Start your web browser and go to https://scihub.copernicus.eu/dhus/
- Login to your scihub account or create a new one.
- Move and zoom to the region of interest.
- Click the lower of the two round buttons on the right side to draw a rectangle covering the study area.
-
Move the mouse to the edges of the rectangle and look at the
coordinates displayed in the lower left corner. Write down boundary
box coordinates for the study area, i.e. maximum and minimum values
of longitude and latitude, repsectively (these will be needed in
Part II to configure cutting to area of interest). In our example,
these are approximately:
Upper left corner: 73.85 (minimum longitude) / 43.1 (maximum latitude)
Lower right corner: 74.98 (maximum longitude) / 42.0 (minimum latitude) - Open the ‘Advanced search’ options by clicking on the icon on the left side of the search bar in the top left corner. Check ‘Mission: Sentinel-1’, choose ‘SLC’ as ‘Product type’ and ‘IW’ as ‘Sensor mode’ (Fig. 1). Then click the search button.
- Move your mouse over the items in the result list and see which of the boxes in the main window flashes to identify the scenes that cover the study area best.
- For mid-latitudes, it does make sense to focus on descending orbits (night takes) to minimize ionospheric disturbances in the data. Descending scenes can easily be identified by their rectangular outlines being slightly rotated clockwise.
- Click on the ‘View product details’ button (eye) below scenes that fit nicely. In the ‘Product details’ popup, click on ‘Product’ to see details of the scene. Check that the ‘Pass direction’ is ‘descending’ and write down the numbers given at ‘Relative orbit’ and ‘Slice number’ (these will be needed in Part II to configure automatic scene retrieval). In our example, this is relative orbit 107 and slice numbers 3 and 4.
Step I.2: Digital Elevation Model (DEM) data is required to remove topographic phase during interferometric processing. A DEM of adequate extent and projection can be obtained from https://topex.ucsd.edu/gmtsar/demgen/
- Enter coordinates that cover the study region plus several kilometers in each direction (e.g. +/-0.1°). A too small DEM will cause a failure in interferometric processing, whereas DEMs much larger than needed will substantially increase processing time.
- Start the DEM processing. When finished, copy the link to your clipboard.
Now the DEM needs to be saved to the computer where you will do the processing. So open a terminal (or start a putty session on Windows machines) and login to the computer you have installed OSARIS on.
In order to keep the overview, it does make sense to create a clear directory structure before downloading the the DEM:
$ cd ~
$ mkdir -p topo/tutorial
$ cd topo/tutorial
$ wget --no-check-certificate <copied-link>
Usually, pasting the link to console should work either using Shift+Strg+V or with the right mouse button.
Now let’s take a look at the structure of the <archive.tar> file that was downloaded:
$ tar -tvf <archive.tar>
drwxrwxrwx apache/ssh_topex 0 2019-01-14 19:06 1547489190.141.20.141.30_tmp/
-rw-rw-rw- apache/ssh_topex 672 2019-01-14 19:06 1547489190.141.20.141.30_tmp/dem_grad.cpt
-rw-rw-rw- apache/ssh_topex 9929540 2019-01-14 19:06 1547489190.141.20.141.30_tmp/dem_ortho.grd
-rwxrwxrwx apache/ssh_topex 978 2019-01-14 19:06 1547489190.141.20.141.30_tmp/run_mosaic.com
-rw-rw-rw- apache/ssh_topex 602 2019-01-14 19:06 1547489190.141.20.141.30_tmp/dem_grad.kml
-rw-rw-rw- apache/ssh_topex 1997638 2019-01-14 19:06 1547489190.141.20.141.30_tmp/dem.ps
-rw-rw-rw- apache/ssh_topex 84 2019-01-14 19:06 1547489190.141.20.141.30_tmp/.gmtcommands4
-rw-rw-rw- apache/ssh_topex 98 2019-01-14 19:06 1547489190.141.20.141.30_tmp/filelist
-rw-rw-rw- apache/ssh_topex 9929620 2019-01-14 19:06 1547489190.141.20.141.30_tmp/dem.grd
-rw-rw-rw- apache/ssh_topex 2684 2019-01-14 19:06 1547489190.141.20.141.30_tmp/.gmtdefaults4
-rw-rw-rw- apache/ssh_topex 3790052 2019-01-14 19:06 1547489190.141.20.141.30_tmp/dem_grad.png
The first line provides the name of the <directory> containing all files in <archive.tar>. We will just need the ‘dem.grd’ file from the tar archive:
$ tar -xvf <archive.tar> <directory>/dem.grd --strip=1
The name of <archive.tar> can be inserted automatically by hitting the tab key. The <directory> name will usually be the same as <archive.tar> with just the type identifier ‘.tar’ replaced by ‘_tmp’; So just hit the tab key again, change the ending, and add ‘/dem.grd’. After extraction, check whether the dem.grd is actually there; if so, you may remove the tar archive (optional).
$ ls
<archive.tar> dem.grd
$ rm <archive.tar>
Step I.3: OSARIS’ interferometric processing requires a library of precise Sentinel-1 orbits. Since quite a lot of orbit data has aggregated so far (~11 GB in early 2019), it is recommended to download the bulk of this data in one compressed archive instead of using the internal orbit download function of OSARIS. To obtain a regularly updated archive, proceed as follows:
$ cd ~
$ wget --content-disposition https://hu.berlin/s1-orbits
This should download a file called S1-Orbits.tar.gz which you could then extract:
$ tar -xzvf S1-Orbits.tar.g
Afterwards, you will have a new directory ‘S1-Orbits’. If so, you may safely delete the tar.gz archive:
$ rm S1-Orbits.tar.gz
Step I.4: In case you don’t know the Slurm configuration in detail, this is a good time to obtain some key information that will be needed later on. Let’s use the sinfo command to find out which Slurm partitions exist as well as how many CPUs and how much RAM each node has:
$ sinfo -o "%P %c %m"
PARTITION CPUS MEMORY
compute* 20 64316+
computehm 20 128000
io 6 6415
This shows partition names (* indicates the default partition), CPUs per node, and memory per node in MB. Naturally, the results will look different for every server. In the example given, nodes in the ‘computehm’ partition have approximately twice as much RAM as nodes in the default ‘compute’ partition. Since OSARIS will need at 20 GB RAM for each processing job and RAM is typically assigned per CPU in Slurm, the relevant number is how much RAM is available per CPU. In the above example, this will be ~3200 MB and 6400 MB per CPU in the ‘compute’ and ‘computehm’ partition, respectively. To make sure 20 GB RAM are available for each processing job, we should therefore keep in mind to assign at least 5 CPUs for jobs in the ‘computehm’ partition and 10 CPUs for jobs in the ‘compute’ partition.
Part II: Create login.credentials file
The login.credentials file will contain username and password for ESA ‘s Scihub service. Optionally, you may also include credentials for the Alaska Satellite Facility (ASF) service provided by the University of Fairbanks. Both services are included in OSARIS for obtaining S1 scenes and orbits and switching between both services is easy (cf. Step III.6) with the advantage less dependent in case of downtimes, e.g. through maintenance operations or government shutdowns. To be able to use the ASF service, create an account at https://urs.earthdata.nasa.gov/users/new.
Step II.1: Go to the OSARIS directory and copy the template login.credentials file to the config directory. The following assumes that OSARIS was installed to git/osaris in your home directory:
$ cd ~/git/osaris
$ cp templates/login.credentials config/login.credentials
Edit the login.credentials file with the editor of your choice
$ emacs -nw config/login.credentials
The file should be pretty much self explaining. Just replace the placeholders with the Scihub credentials you have used in the Part I:
ESA_username="your_username"
ESA_password="your_password
Optionally, do the same with your ASF credentials.
Save the file and exit the editor.
Part III: Set up the config file
Step III.1: Move the configuration file and the to ‘tutorial.config’ in the config directory:
$ cp templates/config.template config/tutorial.config
Move the GMTSAR configuration template to ‘GMTSAR.config’ in the config directory:
$ cp templates/GMTSAR-config.template config/GMTSAR.config
Step III.2: Open the config file with the editor of your choice.
$ emacs -nw config/tutorial.config
Step III.3: Let’s start editing the configuration in the GENERAL section. Unless you want you want to increase or decrease the amount of information written to the log files during processing, you may skip editing the ‘debug’ parameter.
prefix=”tutorial”
The prefix acts as identifier for our processing scheme and also the name of the output directory. You may choose the name freely but avoid spaces and special characters.
swaths_to_process=(3)
The area of interest is fully contained in swath 3, so we can save a lot of processing time and disk space by leaving the other two out.
The ‘Area of interest’ determines the spatial extent of our processing. These parameters are relevant for many processing steps, including selection of scenes to download, S1 bursts to process, and region to cut results to. Since the area of interest was already determined in the first step (I.1), we can simply use the coordinates we measured using the scihub DHUS website:
lon_1=73.85
lat_1=43.1
lon_2=74.98
lat_2=42
In order to minimize processing time and have our result data cut into nice rectangles, let’s activate cutting to area of interest:
cut_to_aoi=1
Step III.4: In section BASE DATA PATHES the configuration needs to be modified to fit your file system. Directories that do not yet exist will be created automatically.
base_PATH=”~/osaris-results”
This will put all files in a subdirectory of your home directory called ‘osaris-results’. Change freely according to where you would like to have the data. Remember that there may be several hundreds of GB required if for comprehensive processing jobs. In this tutorial the data will, however, be kept to minimum, so ~20 GB of disk space will be sufficient.
topo_PATH=”~/topo/tutorial”
This must link to the folder where you put the DEM data in Step I.2.
input_files=”download”
Since we have not yet downloaded the Sentinel-1 files, we hereby tell OSARIS to do so. This will create a directory called “Input” below $base_PATH/$prefix and put all files there.
orbits_PATH=”~/S1-Orbits”
Orbit data will be downloaded to this directory. If you followed the steps in I.3, the directory already exists (please check for correct case of letters). Now let’s activate automatic orbit retrieval:
update_orbits=1
Unless you have saved the login.credentials file somewhere else but in the default directory ‘config’ in Step II.1, you may leave the corresponding parameter with its default value:
credentials_file="$OSARIS_PATH/config/login.credentials"
In case you followed the suggestions in Step III.1 you can also leave the GMTSAR config file configuration at its default values:
gmtsar_config_file="config/GMTSAR.config"
Step III.5: In the PROCESSING CONFIGURATION and MODULES sections everything shall be left at the default values for now. Please note that by default the module ‘Summary PDF’ is activated:
post_processing_mods=( summary_pdf )
This will produce a graphic overview of the processing results. However, the module requires its own configuration file, which we will prepare later (see Part IV).
Step III.6: The SLURM CONFIGURATION section requires several parameters to configured according to the setup of your server’s environment. In case you don’t know any of the relevant information here you will probably have to contact the server’s administrator.
First, enter the Slurm account you want to use for processing:
slurm_account="your_account"
In the next lines the Slurm environment parameters will be set up. You will now need the information regarding Slurm partitions, CPUs and RAM obtained in Step I.4.
slurm_partition="your_partition"
This will be the partition that will be used by default, like ‘computehm’ following the example from Step I.4.
slurm_ntasks=4
The parameter slurm_ntasks defines how many CPUs (and thus how much RAM) shall be reserved for processing. Knowing how much RAM is available per CPU from Step I.4, enter a number that ensures ~20 GB RAM are reserved for processing. In the example this was 6400 MB per CPU for ‘computehm’, so reserving 4 CPUs will be a safe option (4 x 6400 = 25600).
Optionally, you may define similar properties for an alternative partition that will be used in case there are not sufficient free CPUs available on $slurm_partition.
Step III.7: Now let’s finalize the configuration by editing the DOWNLOADS AND DATA section. The first two variables in the section determine which data provider shall be used to download S1 scenes and orbits. In case you setup both ESA and ASF in the login.credentials file, you may choose between both options.
scene_provider=ESA
orbit_provider=ASF
Recently, ASF seemed to provide slightly better performance but this may depend on time and location. Next, it is important to set a relative orbit, so that only ascending OR descending scenes will be downloaded. In step I.1 we identified relative orbit 107 to match the test region:
relative_orbit=107
Constraining the time frame for processing is essential for the overall size of OSARIS projects. If you are not totally sure what you are aiming for, it is always a good idea to start over with a relatively limited time interval of several weeks to month, e.g.:
sensing_period_start="2018-07-01T00:00:00.000Z"
sensing_period_end="2018-09-01T00:00:00.000Z
This will limit the time series to July and August 2018. If you want to expand the time series later on, you may just modify these dates – by default OSARIS will keep downloaded scenes in the Input directory and just add new ones.
Usually, the ingestion period configuration may be kept by the default values. This will ensure that only the sensing period is relevant for downloading and exclude premature data takes.
With this, the configuration file should be ready for a first run.
Save the file and exit the editing program.
Step III.8: As a last step, the configuration for the Summary PDF module should be copied to the config directory, so that the module activated in Step III.5 will run properly. The configuration in the template should work for a first test, so all that needs to be done is:
$ cd ~/git/osaris
$ cp templates/module-config/summary_pdf.config.template config/summary_pdf.config
Part IV: Launch and monitor
So, we are ready to go! If you are not already in a tmux session, it is certainly a good idea to start one now:
$ tmux
Working in a tmux environment has great advantages for remote connections in general and OSARIS in particular. First of all, your session won’t be lost if the connection brakes down. Second, you may create multiple windows in one session and multiple panes in one window. This is very handy, e.g. when monitoring log files as described below. Third, you may logout (detach) and back in (attach) to sessions as you like, with your whole working environment still set up. If tmux is not available on the machine you are working on, you may use GNU screen as an alternative with slightly less functionality. Here are good tutorials for tmux and screen.
TLDR; if you are not using tmux or screen, yet, start using them now.
Let’s start the processing:
$ ./osaris.sh ./config/tutorial.config
You should now see a box with the OSARIS version and first lines of output.