Getting started
Installation
We assume Earth Engine Python API is installed and EE authorised as described here.
To install GEEBAM, run the following command:
pip install geebam
Installation is an optional step; the application can be also run directly by executing geebam.py script as available on GitHub.
We recommend installation within a virtual environment .
Command Line too
Print help with:
geebam -h
usage: geebam.py [-h] [-s SERVICE_ACCOUNT] [-k PRIVATE_KEY]
{delete,upload,cancel,report,copy} ...
Google Earth Engine Batch Asset Manager
positional arguments:
{delete,upload,cancel,report,copy}
delete Deletes collection and all items inside. Supports
Unix-like wildcards.
upload Batch Asset Uploader.
cancel Cancel all running tasks
report Produce summary of all assets.
copy Batch copy of assets. Helps in migrating assets from
Google Maps to GEE
optional arguments:
-h, --help show this help message and exit
-s SERVICE_ACCOUNT, --service-account SERVICE_ACCOUNT
Google Earth Engine service account.
-k PRIVATE_KEY, --private-key PRIVATE_KEY
Google Earth Engine private key file.
To obtain help for a specific functionality, simply call it with help
switch, e.g.: geebam upload -h
. If you didn't install geebam, then you
can run it just by going to geebam directory and running python
geebam.py [arguments go here]
Batch uploader
The script creates an Image Collection from GeoTIFFs in your local directory. By default, the collection name is the same as the local directory name; with optional parameter you can provide a different name. Another optional parameter is a path to a CSV file with metadata for images, which is covered in the next section: Parsing metadata.
geebam upload -h
usage: geebam.py upload [-h] --source SOURCE --dest DEST [-m METADATA]
[--large] [--nodata NODATA] [--bands BANDS] [-u USER]
[-b BUCKET] [-e] [-a] [--headless]
optional arguments:
-h, --help show this help message and exit
Required named arguments.:
--source SOURCE Path to the directory with images for upload.
--dest DEST Destination. Full path for upload to Google Earth
Engine, e.g. users/pinkiepie/myponycollection
-u USER, --user USER Google account name (gmail address).
Optional named arguments:
-m METADATA, --metadata METADATA
Path to CSV with metadata.
--large (Advanced) Use multipart upload. Might help if upload
of large files is failing on some systems. Might cause
other issues.
--nodata NODATA The value to burn into the raster as NoData (missing
data)
--bands BANDS Comma-separated list of names to use for the image
bands. Spacesor other special characters are not
allowed.
-b BUCKET, --bucket BUCKET
Google Cloud Storage bucket name.
-e, --upload-catch-error
Return exit code 1 when upload catches an error
-a, --tolerate-assets-already-exist
Return exit 0 when assets already exist
--headless Run the browser in headless mode (i.e. no user
interface).
Parsing metadata
By metadata we understand here the properties associated with each image. Thanks to these, GEE user can easily filter collection based on specified criteria. The file with metadata should be organised as follows:
filename (without extension) | property1 header | property2 header |
---|---|---|
file1 | value1 | value2 |
file2 | value3 | value4 |
Note that header can contain only letters, digits and underscores.
Example:
id_no | class | category | binomial | system:time_start |
---|---|---|---|---|
my_file_1 | GASTROPODA | EN | Aaadonta constricta | 1478943081000 |
my_file_2 | GASTROPODA | CR | Aaadonta irregularis | 1478943081000 |
The corresponding files are my_file_1.tif and my_file_2.tif. With each of the files five properties are associated: id_no, class, category, binomial and system:time_start. The latter is time in Unix epoch format, in milliseconds, as documented in GEE glosary. The program will match the file names from the upload directory with ones provided in the CSV and pass the metadata in JSON format:
{ id_no: my_file_1, class: GASTROPODA, category: EN, binomial: Aaadonta constricta, system:time_start: 1478943081000}
The program will report any illegal fields, it will also complain if not all of the images passed for upload have metadata associated. User can opt to ignore it, in which case some assets will have no properties.
Having metadata helps in organising your asstets, but is not mandatory - you can skip it.
Usage examples
Delete a collection with content:
The delete is recursive, meaning it will delete also all children assets: images, collections and folders. Use with caution!
geebam delete users/pinkiepie/test
Console output:
2016-07-17 16:14:09,212 :: oauth2client.client :: INFO :: Attempting refresh to obtain initial access_token
2016-07-17 16:14:09,213 :: oauth2client.client :: INFO :: Refreshing access_token
2016-07-17 16:14:10,842 :: root :: INFO :: Attempting to delete collection test
2016-07-17 16:14:16,898 :: root :: INFO :: Collection users/pinkiepie/test removed
Delete all directories / collections based on a Unix-like pattern
geebam delete users/pinkiepie/*weird[0-9]?name*
Upload a directory with images to your myfolder/mycollection and associate properties with each image:
geebam upload -u pinkiepie@gmail.com --source path_to_directory_with_tif -m path_to_metadata.csv --dest users/pinkiepie/myfolder/myponycollection
The script will prompt the user for Google account password. The program will also check that all properties in path_to_metadata.csv do not contain any illegal characters for GEE. Don't need metadata? Simply skip this option.
Upload a directory with images with specific NoData value to a selected destination
geebam upload -u pinkiepie@gmail.com --source path_to_directory_with_tif --dest users/pinkiepie/myfolder/myponycollection --nodata 222
In this case we need to supply full path to the destination, which is helpful when we upload to a shared folder. In the provided example we also burn value 222 into all rasters for missing data (NoData).