Filesystem Utilities

Contains functions for sorting, creating and comparing images as part of the filesystem. Includes any function that works on a generic pyeo timestamp and sentinel 2 functions.

Key functions

init_log() Sets up logging to both console and file

create_file_structure() Creates a recommended file structure for automated work

sort_by_timestamp() Sorts a set of files by timestamp

Function reference

pyeo.filesystem_utilities.check_for_invalid_l1_data(l1_SAFE_file)

Checks the existance of the specified resolution of imagery. Returns True with a warning if passed an invalid SAFE directory; this will prevent disconnected files from being deleted.

Parameters:

l1_SAFE_file (str) – Path to the L1 file to check

Returns:

result – 1 if imagery is valid, 0 if not and 2 if not a safe-file

Return type:

int

pyeo.filesystem_utilities.check_for_invalid_l2_data(l2_SAFE_file, resolution='10m')

Checks the existence of the specified resolution of imagery. Returns a True-value with a warning if passed an invalid SAFE directory; this will prevent disconnected files from being deleted.

Parameters:

• l2_SAFE_file (str) – Path to the L2A file to check

• resolution ({"10m", "20m", "60m"}) – The resolution of imagery to check. Defaults to 10m.

Returns:

result – 1 if imagery is valid, 0 if not and 2 if an invalid .SAFE file

Return type:

int

pyeo.filesystem_utilities.clean_aoi(aoi_dir, images_to_keep=4, warning=True)

Removes all but the last images_to_keep newest images in the L1, L2, merged, stacked and composite directories. Will not affect the output folder. Use with caution.

Parameters:

• aoi_dir (str) – A directory made by create_file_structure()

• images_to_keep (int, optional) – The number of images to keep

pyeo.filesystem_utilities.clean_l2_data(l2_SAFE_file, resolution='10m', warning=True)

Removes a safe file if it doesn’t have bands 2, 3, 4 or 8 in the specified resolution folder. If warning=True, prompts before removal.

Parameters:

l2_SAFE_file (str) – Path to the L2 .SAFE file

pyeo.filesystem_utilities.clean_l2_dir(l2_dir, resolution='10m', warning=True)

Calls clean_l2_data on every SAFE file in l2_dir

Parameters:

• l2_dir (str) – The L2 directory

• resolution ({"10m", "20m", "60m"}, optional) – Resolution to check. Defaults to 10m

• warning (bool, optional) – If True, prompts user before deleting files.

pyeo.filesystem_utilities.conda_check(config_dict: dict, log)

This function takes the path to the config (pyeo.ini) and checks whether the conda environment exists.

Parameters:

config_dict (dict) – config_dict containing conda_directory and conda_env_name, which will be checked to see if they exist.

Return type:

True/False (bool)

pyeo.filesystem_utilities.config_path_to_config_dict(config_path: str)

This function takes the path to the config (pyeo.ini) and simplifies the keys

Parameters:

config_path (str) – path to pyeo.ini

Returns:

config_dict – a config dictionary

Return type:

dict

pyeo.filesystem_utilities.create_file_structure(root: str)

Creates the folder structure used in rolling_s2_composite and some other functions: ::

root -images –L1C –L2A –bandmerged –stacked –stacked_mosaic –stacked_masked -composite –L1C –L2A –bandmerged -output –classified –probabilities

Parameters:

root (str) – The root folder for the file strucutre

pyeo.filesystem_utilities.create_folder_structure_for_tiles(root)

Creates the folder structure used in tile_based_change_detection.py: ::

root -images –L1C –L2A –cloud_masked -composite –L1C –L2A –cloud_masked -output –classified –probabilities

Parameters:

root (str) – The root folder for the file strucutre

pyeo.filesystem_utilities.get_change_detection_date_strings(image_name)

Extracts the before_date and after_date dates from a change detection image name.

Parameters:

image_name (str) – A pyeo-produced image name (example: class_composite_T36MZE_20190509T073621_20190519T073621.tif)

Returns:

before_date, after_date – The dates associated with the image in string format

Return type:

str

pyeo.filesystem_utilities.get_change_detection_dates(image_name)

Extracts the before_date and after_date dates from a change detection image name.

Parameters:

image_name (str) – A pyeo-produced image name (example: class_composite_T36MZE_20190509T073621_20190519T073621.tif)

Returns:

before_date, after_date – The dates associated with the image

Return type:

DateTime

pyeo.filesystem_utilities.get_filenames(path, filepattern, dirpattern)

Finds all file names in a directory for which the file name matches a certain string pattern, and the directory name matches a different string pattern.

Parameters:

• done (path = string indicating the path to a directory in which the search will be) –

• for (dirpattern = string of the directory name pattern to search) –

• for –

Returns:

a list of all found files with the full path directory

pyeo.filesystem_utilities.get_image_acquisition_time(image_name)

Gets the datetime object from a .safe filename of a Sentinel image. No test. Returns None if no timestamp present :param image_name: The .SAFE filename :type image_name: str

Returns:

acquisition_time – A DateTime object providing the acquisition time

Return type:

DateTime

pyeo.filesystem_utilities.get_l1_safe_file(image_name, l1_dir)

Returns the path to the L1 .SAFE file of a L2 image. Gets from granule and timestamp. image_name can be a path or a filename. Returns None if not found.

Parameters:

• image_name (str) – The name of the L2 image.

• l1_dir (str) – The path to the folder containing L1 images.

Returns:

l1_image – The path to the L1 image, or None if not found.

Return type:

str or None

pyeo.filesystem_utilities.get_l2_safe_file(image_name, l2_dir)

Returns the path to the L2 .SAFE file of a L1 image. Gets from granule and timestamp. image_name can be a path or a filename. Returns None if not found.

Parameters:

• image_name (str) – The name of the L1 image.

• l2_dir (str) – The path to the folder containing L2 images.

Returns:

l2_image – The path to the L2 image, or None if not found.

Return type:

str or None

pyeo.filesystem_utilities.get_mask_path(image_path)

Returns the path to the mask file of the image. Does not verify that the mask exists. :param image_path: Path to the image :type image_path: str

Returns:

mask_path – Path to the corresponding mask.

Return type:

str

pyeo.filesystem_utilities.get_preceding_image_path(target_image_name, search_dir)

Finds the image that directly precedes the target image in time.

Parameters:

• target_image_name (str) – A pyeo or Sentinel generated image name

• search_dir (str) – The directory to search for the preceding image.

Returns:

preceding_path – The path to the preceding image

Return type:

str

Raises:

FileNotFoundError – If there is no image older than the target image

pyeo.filesystem_utilities.get_pyeo_timestamp(image_name)

Returns a list of all timestamps in a pyeo image (yyyymmddhhmmss)

Parameters:

image_name (str) – The pyeo image name

Returns:

timestamp_list – A list of the timestamps in the name

Return type:

list of str

pyeo.filesystem_utilities.get_raster_paths(paths, filepatterns, dirpattern)

Iterates over get_filenames for different paths and different file patterns and returns a dataframe of all directory paths that match the conditions together with the root path in which they were found.

Parameters:

• done (paths = list of strings indicating the path to a root directory in which the search will be) –

• for (dirpattern = string of the directory name pattern to search) –

• for –

Returns:

a dataframe of all found file paths, one line per path

pyeo.filesystem_utilities.get_related_images(target_image_name, project_dir)

Finds every image related to the target image (L1, L2, merged, stacked and classified). Based on timestamp, and assumes a project structured by create_file_structure() :param target_image_name: The target image :type target_image_name: str :param project_dir: The root of the project directory :type project_dir: str

Returns:

related_images – A list of all the images related to target_image

Return type:

list of str

pyeo.filesystem_utilities.get_safe_product_type(image_name)

Returns the product string (MSIL1C or MSIL2A) from a .safe file identifier :param image_name: The name of the image :type image_name: str

Returns:

product_string – The product string

Return type:

{“MSIL1C” or “MSIL2A”}

pyeo.filesystem_utilities.get_sen_2_baseline(image_name)

“Returns the baseline orbit identifier of a Sentinel 2 image :param image_name: The Sentinel image name or path :type image_name: str

Returns:

baseline – The baseline orbit identifier (eg: ‘N0214’)

Return type:

str

pyeo.filesystem_utilities.get_sen_2_granule_id(safe_dir)

Returns the unique ID of a Sentinel 2 granule from a SAFE directory path

Parameters:

safe_dir (str) – Path to a .SAFE file

Returns:

id – The full ID of that S2 product (eg: ‘S2B_MSIL2A_20180103T172709_N0206_R012_T13QFB_20180103T192359’)

Return type:

str

pyeo.filesystem_utilities.get_sen_2_image_orbit(image_name)

Returns the relative orbit identifer of a Sentinel 2 image

Parameters:

image_name (str) – The Sentinel image name or path

Returns:

orbit_number – The orbit. (eg: ‘R012’)

Return type:

str

pyeo.filesystem_utilities.get_sen_2_image_tile(image_name)

Returns the tile number of a Sentinel 2 image or path :param image_name: The Sentinel image name or path :type image_name: str

Returns:

tile_id – The tile ID (eg: ‘T13QFB’)

Return type:

str

pyeo.filesystem_utilities.get_sen_2_image_timestamp(image_name)

Returns the timestamps part of a Sentinel image

Parameters:

image_name (str) – The Sentinel image name or path

Returns:

timestamp – The timestamp (yyyymmddThhmmss)

Return type:

str

pyeo.filesystem_utilities.get_sen_2_tiles(image_dir)

Returns a list of Sentinel-2 tile IDs present in a folder

Parameters:

image_dir (str) – Path to the folder containing images

Returns:

tiles – A list of the tile identifiers present in the folder

Return type:

list of str

pyeo.filesystem_utilities.init_log(log_path)

Sets up the log format and log handlers; one for stdout and to write to a file, ‘log_path’. Returns the log for the calling script.

Parameters:

log_path (str) – The path to the file output of this log.

Returns:

log – The logging object.

Return type:

logging.Logger

pyeo.filesystem_utilities.init_log_acd(log_path, logger_name)

This function differs slightly to init_log in that it accomodates a logger_name. This enables

unique logger objects to be created so multiple loggers can be run at a time.

Sets up the log format and log handlers; one for stdout and to write to a file, ‘log_path’. Returns the log for the calling script.

Parameters:

• log_path (str) – The path to the file output of this log.

• logger_name (str) – A unique logger name.

Returns:

log – The logging object.

Return type:

logging.Logger

pyeo.filesystem_utilities.serial_date_to_string(srl_no: int) → str

Converts a serial date (days since X) to a date as a string.

Parameters:

srl_no (int) – Serial number representing days since X.

Returns:

Date in the format “YYYY-MM-DD”.

Return type:

str

Notes

This function assumes the base date as January 1, 2000.

References

• Original implementation by AER: https://stackoverflow.com/a/39988256/6809533

pyeo.filesystem_utilities.sort_by_timestamp(strings, recent_first=True)

Takes a list of strings that contain sen2 timestamps and returns them sorted, from most recent on back by default. Removes any string that does not contain a timestamp

Parameters:

• strings (list of str) – list of timestamped strings to sort

• recent_first (bool, optional) – If True, sorted_strings[0] is the most recent image. If False, sorted_strings[0] is the least recent image.

Returns:

sorted_strings – The strings sorted by timestamp.

Return type:

list of str

Notes

Does not guarantee preservation of ordering of strings with the same timestamp.

pyeo.filesystem_utilities.unzip_contents(zippath: str, ifstartswith=None, ending=None) → None

Unzip the contents of the specified zipped folder.

Parameters:

• zippath (str) – Path to the zipped folder to unzip.

• ifstartswith (str or None, optional) – Prefix to check before extracting. Default is None.

• ending (str or None, optional) – Suffix to add to the extracted folder name. Default is None.

Return type:

None

Notes

• This function assumes the zip file extension to be “.zip”.

• If ifstartswith is provided, extraction occurs only if the folder name starts with the specified prefix.

• If ending is provided, it is added to the extracted folder name.

pyeo.filesystem_utilities.zip_contents(directory: str, notstartswith=None) → None

Zip the contents of the specified directory.

Parameters:

• directory (str) – Path to the directory whose contents to zip.

• notstartswith (list or None, optional) – List of prefixes to exclude from zipping. Default is None.

Return type:

None

Notes

• This function skips files that have the “.zip” extension.

• If notstartswith is provided, files starting with any of the specified prefixes are skipped.