This is the multi-page printable view of this section. Click here to print.
Welcome to the official sett documentation page starting from
version 5.0.0, released on 22nd of April 2024.
If you are looking for the documentation of the previous versions, please refer
to the old documentation.
Please use the table of contents menu to the left or below to navigate the topics.
sett stands for "Secure Encryption and Transfer Tool" and is a Rust-based program that automates the packaging, encryption, and transfer of data.
sett is available both as a desktop app, and as a command line tool. Most functionalities of sett are available in both the desktop app and in the command line tool.
sett is developed as part of the BioMedIT project. It is licensed under the LGPLv3 (GNU Lesser General Public License) and the source code is available from its public GitLab repo
If you do not already have a private/public PGP key pair, go to the Keys tab and create one clicking on Add > Generate new key pair. See also the instructions given in the Generate a new public/private PGP key pair section.
You should then see your new key listed in the Keys tab, along with “Private” label that indicates that the private material for this key is present in the local keystore.
If not already done, download the public PGP key of the recipient(s) to whom you intend to send data (or from whom you will receive data). In sett-gui, go to the Keys tab and click on Add > Import from keyserver. See also the instructions given in the download public PGP keys from the keyserver section.
Just after downloading the recipient’s PGP key, verify it to make sure that it is genuine. This can be done by either:
Go to the Encrypt tab of the sett-gui interface.
Add one or more files and directories to encrypt by clicking the Add files or Add directories buttons.
Select sender: select your own PGP key. This is the key that will be used to sign the encrypted data.
Select recipients: add one or more recipients by selecting them in the drop-down. These are the keys that will be used to encrypt the data, i.e. only these recipients will be able to decrypt the data.
Transfer ID: specifying a valid Data Transfer Request ID is mandatory when a data package is transferred into the BioMedIT network. For other destinations, the Transfer ID field can be left empty (or set to any arbitrary value), and the Verify package checkbox must be disabled (in the Settings tab).
Select destination: select local to encrypt to your local file system, and choose a destination directory.
Click Encrypt package - or Send package, if you chose s3 or sftp as your destination - to run the encryption workflow on your data.
Go to the Transfer tab of the sett-gui interface.
Select a file to transfer using the Select file button.
Select the Destination to be used (sftp, s3).
Enter the required destination parameters.
For transfers into the BioMedIT network, the destination parameters are provided by:
Click Send package to start transferring your files.
The main commands to manage keys, encrypt, transfer and decrypt data with sett command line interface are given here.
# Generate a new key pair:
sett keys generate
# Import sender/recipient(s) public keys:
sett keys import from-keyserver alice@example.com
# Data encryption:
sett encrypt local --signer alice@email.com --recipient bob@example.com FILES_OR_DIRECTORIES_TO_ENCRYPT
# Data transfer
# to SFTP server:
sett transfer sftp --host HOST --username USERNAME --base-path DESTINATION_DIRECTORY --key-path SSH_KEY_LOCATION --key-pwd SSH_KEY_PASSWORD FILES_TO_TRANSFER
# to S3 object store:
sett transfer s3 --endpoint ENDPOINT --bucket BUCKET --access-key ACCESS_KEY --secret-key SECRET_KEY FILES_TO_TRANSFER
# Data decryption:
sett decrypt ENCRYPTED_FILES.zip
You can download the latest version of sett for your operating system by clicking on the corresponding link above.
For a complete list of available versions, please visit the releases page.
As only the latest version of sett is officially supported and guaranteed to work, it is strongly recommended to always keep your local installation of sett up-to-date.
If you installed sett using the installer available for your operating system, sett will automatically check for updates and prompt you to install the newest version as soon as it's available.
If you are running a portable executable, make sure you regularly check this page and download the latest version of the executable.
To encrypt and decrypt data, sett uses public key cryptography. If you are not familiar with public key cryptography concepts such as public and private keys, revocation signatures, or keyservers you are advised to read this introductory section.
A prerequisite to encrypt, decrypt and transfer files with sett is to have a public/private PGP key pair.
To generate a new public/private key pair using sett-gui:
Go to the Keys tab and click on + Add. From the dropdown menu, choose Generate new key pair. A dialog box will appear.
Fill-in the required fields:
Click Generate key and wait for a few seconds, a new key pair will be created.
A new pop-up window will appear to confirm that the key was generated successfully and to display the revocation signature associated with the new key, as shown in the figure below.
Copy the revocation signature to a safe location, ideally a password manager, and click Next.
The next screen of the pop-up window allows to upload the public part of the key to the keyserver. Make sure to check the checkbox to also verify the key with its associated email address. In this context “to verify a key” means that, after your key is uploaded, the keyserver will send an email to verify that you are the legitimate owner of the email associated with the key.
The new key should now be listed in sett.
Now that your new PGP key is created, make sure to register it in the BioMedIT portal
To generate a new key pair using sett command line interface:
sett keys generate "Alice Smith <alice.smith@example.com>"
Additional options can be listed with sett keys generate --help.
During the key generation process, sett will ask for a pass phrase (password) that will protect the private key. Please use a reasonably strong password and make sure to save it in a password manager.
If the password is forgotten, there is no way to retrieve or re-create it. Your PGP key pair will thus become unusable and will need to be revoked.
After creating the new key, sett will display a revocation signature for it.
Make sure you keep it in a safe place, such as a password manager. Revocation
signatures can optionally be automatically exported to a file by using the -r, --rev-sig option:
sett keys generate --rev-sig "Alice Smith <alice.smith@example.com"
Private and public keys available in the keystore can be listed with the following command:
sett keys list
In some situations (new computer setup, remote sett environment, ...) you might need to copy or move your private key to a different machine. This can be done by exporting the private key to a file, transferring the file to the new machine, and importing it.
Here is an example of how to export a private key to a file with sett:
# The key identifier can be its fingerprint, key ID or the user's email address.
sett keys export -o private_key.pgp -p alice.smith@example.com
Note that keys can also be exported in ASCII format (instead of binary) by
adding the -a, --armor option.
After the key has been transferred to the new machine (in either binary or ASCII format), importing it is as easy as:
sett keys import from-file -p private_key.pgp
Note the usage of the -p option to import both the private and public parts of
the key. If -p is omitted, only the public part of the key is imported.
Verify that the key has been correctly imported with sett keys list.
Ensure that you store any backed-up secret keys in a secure location and in an encrypted form (typically in a password manager).
sett allows users to upload their public PGP key to the keyserver specified in its settings.
By default, a new sett installation is configured to upload keys to the OpenPGP keyserver.
keys.openpgp.org is a public keyserver for the distribution and discovery of public PGP keys. This service is GDPR-compliant, based on the open source software Hagrid, and already used by over 300'000 users worldwide. For more information about the service, please see this link.
Uploading your public PGP key to the keyserver specified in sett is straightforward:
To upload your public PGP key to the keyserver, run:
sett keys upload alice.smith@exaple.com
In order to also trigger email verification for your key, add the -v, --verify
option:
sett keys upload --verify alice.smith@exaple.com
If you are not a BioMedIT user, this section is not relevant for you and can be skipped.
PGP keys used to encrypt, sign and decrypt data transiting via the BioMedIT network require the approval of the BioMedIT key validation authority. The information of whether a key is trusted or not is stored as key status in the BioMedIT portal. This is the reason why all PGP keys used within BioMedIT must be registered with the BioMedIT portal.
When a PGP key is first registered in the BioMedIT portal, its status is
initially set to PENDING (i.e. it is awaiting approval). A keys must be in
APPROVED status before it can be used to encrypt or sign data packages
transiting via the BioMedIT network.
The list of key statuses is as follows:
PENDING: a key approval request was submitted, but the key has not been
approved yet. This is a manual process and can take from a few hours or up
couple of days.APPROVED: key is approved for usage within the BioMedIT network. Only
approved keys can be used to encrypt, sign and decrypt data packages that are
transiting via the BioMedIT network.APPROVAL-REVOKED: approval of the key has been revoked by the BioMedIT key
validation authority.KEY-REVOKED: key has been revoked by its owner.REJECTED: key is not trusted by the BioMedIT key validation authority.DELETED: key has been removed from the keyserver by its owner.UNKNOWN KEY: key has not been registered on the BioMedIT portal. If it is
your own key, please register it. If it is the key of someone else, please ask
them to register their key.To verify that a key is trusted, sett connects to the BioMedIT portal and retrieves the status of the key. For this reason, it is important that BioMedIT users register their PGP key with the BioMedIT portal.
In cases where sett is used outside of the BioMedIT project, or the portal is
not reachable, sett can still be used to encrypt, decrypt, and transfer data.
In this case, you need to uncheck [ ] Verify DTR and [ ] Verify key approval
in the Settings tab.
The status of a public key in the BioMedIT portal can be easily checked in sett-gui by going to the Keys tab, and expanding the key in the List of available keys.
Each BioMedIT user can only have 1 active PGP key registered in the BioMedIT portal at any time.
If you wish to replace your currently active key with another one, please connect to the BioMedIT portal, go to the "My Keys" tab and click on the "Retire Key" icon (Actions column) associated with your key.
The email associated with your PGP key must be one of the emails associated to the Switch edu-ID account that you are using with the BioMedIT portal.
If the email associated to your PGP key is different (e.g. a shared service email is used for your PGP key), please contact the BioMedIT support.
To register a new PGP key with the BioMedIT portal, proceed as follows:
Make sure you have successfully uploaded your key to the keyserver and that you have completed the email verification procedure with the keyserver (i.e. your key must be verified with the keyserver).
To make sure that your key is present on the keyserver and is verified, go to the keyserver home page in your browser and search for the email associated with your key. You should get a message saying that a key was found for your email address, as shown in the example below.
Your key fingerprint will also be shown - see the right-most part of the link in this screenshot:
Copy the fingerprint (40-character string) of your key.
Log in to the BioMedIT portal.
Go to the My Keys tab.
Click on the "+ KEY" button, a dialog box will open.
Note: if the button is missing, it is probably because you already have an active key in the portal. Each user can only have 1 active key at a time - see the information box above.
Enter your full key fingerprint (must be exactly 40 characters long) in the dialog box, then press the green search icon to the right.
This will retrieve the user ID and email address associated with the fingerprint from the keyserver and display them in the dialog box.
Verify the user ID and mail address. If they are correct for your PGP key, then click on Confirm.
A request to approve your key has now been sent to the BioMedIT key validation authority. Generally requests are processed quickly (in a matter of hours), but occasionally it might take slightly longer as this is a manual process.
Please contact the BioMedIT support if your key has not been approved after a couple of days.
In order to encrypt data for a specific recipient (who will be able to decrypt it), you will need to have the public PGP key of that recipient(s) available in your local keyring. sett allows to easily search and retrieve public PGP keys from the keyserver specified in the settings.
To download a public PGP key from the keyserver:
In the Keys tab, click on + Add. From the dropdown menu, choose Import from keyserver. A dialog box will open, allowing to search for public keys stored on the keyserver.
In the search field, enter either the full email address or fingerprint of the public key you are looking for and hit Search: the matching key found on the keyserver will be displayed.
For instance, if searching for the key “Bob Test Key bob@example.com” with fingerprint “AEED7A96F339881F6FE8291464A1E0150613807D”, one can search for either “bob@example.com” or “AEED7A96F339881F6FE8291464A1E0150613807D”.
If the key you are looking for was found, click on Import. You should then see your recipient’s key listed in the List of available keys.
Expand the details of the newly downloaded key (by clicking on the key in the list or on the small ^ button to the right), and verify that its Approval status is set to “Approved” - also indicated by a green check mark.
The following command shows how to download a public PGP key from the keyserver. The search term can be an email or a fingerprint.
sett keys import from-keyserver alice.smith@example.com
While it is not possible to remove an actual key from the keyserver, it is possible to remove all personal identification from it (user ID and email). Such keys are called unverified.
To remove personal information associated with a key, go to the keyserver's manage key page, enter the email associated to the key and click on Send link.
You will receive an email with further instructions on how to proceed to remove your key's user ID and email from the keyserver.
Removing a key's user ID and email from the keyserver makes it no longer searchable by email. It remains searchable by its full fingerprint.
Important: keys without user ID and email cannot be used within the BioMedIT network. Only remove your personal identifying information from a key if you are no longer using it.
Deleting public and/or secret PGP keys form your local keystore is not possible with sett. This is intentional, and users are encouraged to revoke their key instead.
Should you nevertheless wish to delete a key from your local keystore (e.g. because you created a test key that is no longer needed), it is possible to do so by manually deleting the files containing the keys using your file explorer or a shell command.
In the local sett keystore, keys are stored as files named after the
fingerprint of the key’s primary key. For instance, a key with fingerprint
3b17f529665fe012ef54f4a1714fdf98b6e828df would be stored under:
<keystore-path>/pgp.cert.d/3b/17f529665fe012ef54f4a1714fdf98b6e828df<keystore-path>/pgp.cert.d.sec/3b/17f529665fe012ef54f4a1714fdf98b6e828dfThe location of the keystore (<keystore-path> above) is operating-system
dependent:
~/.local/share/pgp.cert.d for public keys, and
~/.local/share/pgp.cert.d.sec for secret keys.%UserProfile%\AppData\Roaming\pgp.cert.d for public keys, and
%UserProfile%\AppData\Roaming\pgp.cert.d.sec for secret keys.~/Library/Application Support/pgp.cert.d for public keys, and
~/Library/Application Support/pgp.cert.d.sec for secret keys.To get the exact location of the key by using the sett-gui app:
In the Keys tab, look for your PGP key from the List of available keys and click on the ellipsis button to its right.
From the dropdown menu, select Delete.
A dialog box with important information about key deletion, as well as the exact location of the key will be displayed.
A prerequisite for revoking a PGP key is to have generated a revocation signature for it. If the PGP key to revoke was generated with sett, you should in principle already have a revocation signature ready to use. If you do not have a revocation signature yet, you can generate one with sett-gui:
In the Keys tab, look for your PGP key from the List of available keys and click on the ellipsis button to its right.
From the dropdown menu, select Create revocation signature. A dialog box will appear.
Fill-in the required fields:
Click Generate.
A new pop-up window will appear to confirm that the revocation signature was generated successfully. Copy the revocation signature to a safe location, ideally a password manager (anyone with access to a revocation signature can revoke the key for which it was generated).
If a private PGP key has been compromised, is no longer usable (e.g. password is lost), or should no longer be used for any other reason, it must be revoked.
A prerequisite for revoking a PGP key is to have generated a revocation signature for it. If the PGP key to revoke was generated with sett, you should in principle already have a revocation signature ready to use. If you do not have a revocation signature yet, please generate one by referring to the Generate a revocation signature section.
To revoke a PGP key in sett-gui:
In the Keys tab, look for your PGP key from the List of available keys and click on the ellipsis button to its right.
From the dropdown menu, select Revoke. A dialog box will appear.
Either paste you revocation signature from the clipboard or load it from a file.
Warning: proceed with caution, a revoked key cannot be “un-revoked".
Click on Revoke.
When expanding the revoked key from the List of available keys, you should now see the Revocation status as “Revoked”. From this point on, the key can no longer be used with sett.
If you have previously shared your key via a keyserver (e.g. keys.openpgp.org), you must also re-upload your revoked key to that keyserver.
This will allow other users to update their local copy of your public key, informing them that it is no longer valid. To upload your revoked public key, please refer to the Upload your public PGP key to the keyserver .
If your key was never present on any keyserver, this step should be skipped.
To revoke a key, run:
sett keys revoke alice.smith@example.com compromised "My dog ate it"
After a key has been revoked, it must be uploaded again to any keyserver(s) where it is present, so that the revocation can be shared with others. This can be done with sett as illustrated in the Upload your public PGP key to the keyserver.
PGP keys located in your GnuPG keyring are not automatically detected by sett, and they must be migrated to sett’s keystore.
Note that to migrate public keys (i.e. keys from other people), you can simply re-download them from the keyserver as shown in the Download public PGP keys from the keyserver section.
Using sett-gui, this procedure is straightforward:
To migrate a key from your GnuPG keyring, run:
# The search term can be an email or fingerprint.
sett keys import from-gpg alice.smith@example.com
Public-key cryptography is a method for secure communication between two or more users. In this system, each user has a pair of unique keys consisting of a private key and a public key. Public and private keys are linked in the sense that, data encrypted with a given public key can only be decrypted with the matching private key, and data signed with a given private key will only be recognized by the matching public key.
Because these keys are based on the OpenPGP protocol, they will here be referred to as PGP keys.
Public and private PGP keys:
Public keys are used to encrypt data, as well as for verifying signatures made on files or emails. By design, public keys are intended to be shared with other people and therefore no particular effort is required to keep them secret. In fact, public keys are often uploaded to public servers, known as keyservers, where they are accessible to anyone. No password is required to use a public key.
Typically, public keys are used by data senders to encrypt data for one or more recipient(s), and by data recipients to verify signatures of files or emails (to ensure the sender is genuine).
Private keys, sometimes also referred to as a secret keys, are used to decrypt data, sign files and sign other people's public keys. To increase security, private keys should always be password protected.
Private keys and their password must be kept secret at all times. Never share your private key or password with anyone.
Private keys should be stored in a directory only accessible by the key owner, and their password should be stored in a password manager.
Typically, private keys are used by data recipients to decrypt data, and by data senders to sign the files they encrypt.
sett uses the open source implementation of public-key cryptography provided by Sequoia-PGP: a modular OpenPGP implementation in Rust.
It is also possible - and often desirable - to both encrypt and sign a file. This ensures that the data can only be read by the intended recipient, and that the recipient can be confident the sender is legitimate. This is precisely what sett does:
Each pair of public/private PGP keys is identified by a unique fingerprint. Fingerprints are 40 characters long hexadecimal strings (digits and upper case A-F letters) that look like this:
238565936FCFF3F200219990941A3EC20555F781
Since nothing is preventing two PGP keys to have the same user name and email address, it is critical that users always verify the genuineness of new keys before (or just after) importing them into their local keyring (i.e. their local PGP key database).
Ensuring a key is genuine can be done in two different ways:
In public key cryptography, the sender encrypts a file using one or more recipient(s) public key(s). Once a file is encrypted, no one can read the file without having access to a private key that matches the public key(s) used for encryption. This ensures that only the intended recipient(s) can decrypt the file, because they are the only one to have access to the matching private key.
The objective of file signing is to guarantee to the recipient of a file (or email) that the sender is genuine, and not someone else trying to impersonate the sender.
To achieve this, the sender signs the file with their private key (password required), and shares their public key with the recipient (typically via a keyserver). The recipient can then validate the authenticity of the signature using the public key of the sender. Since public keys are non-sensitive, they can be distributed publicly. In fact they are intended for this purpose, hence their name.
In the unfortunate event that a user either i) forgets their private key’s password or ii) have their private key and password stolen/compromised, they will need a way to let other people know that their public key should no longer be trusted and used.
This is because:
This situation is what revocation signatures are for: by applying a revocation signature to a public key, and then sharing the revoked key with others (e.g. via a keyserver), the key owner signals that their key is now "revoked" and should no longer be trusted nor used. After a key has been revoked, it can no longer be used to encrypt/decrypt data with sett.
Revocation signatures can be generated at anytime from the private key, but the best practice is to generate them directly after a new key pair is created. This ensures that the revocation signature will be available even if the private key or its password is lost.
Since anyone with access to a revocation signature will be able to revoke the associated key, revocation signatures must be stored securely - e.g. in a password manager - and should never be shared with anyone.
Encrypting files for a specific recipient requires to have the recipient’s public key in one's local keyring (a keyring is a local database containing PGP keys). Similarly, verifying a signature on a file or a public key requires to have the signee's public key available in one’s local keyring.
Public keys are not sensitive data, and therefore can in principle be sent unencrypted via email. However, when having frequent key exchanges between multiple actors, sending public PGP keys around by email quickly becomes cumbersome. A solution to this problem is using a so called keyserver to share public keys. Keyservers are public or private servers whose sole purpose is to store public PGP keys and allow users to search for them. In addition, public keyservers often form a network and are regularly synchronized among themselves to ensure redundancy. Private keyservers have the same role, but do not share/synchronize the stored public keys with any other server, and possibly have a restricted access policy so that only authorized people can search for keys.
SSH (Secure SHell) keys are pairs of small text files that are used to securely identify users and give them access to remote servers, e.g. when transferring data via SFTP.
SSH keys use public-key encryption and always come in pairs: a public and a private (or secret) key.
Generating a new pair of SSH keys must be done only once, and, in the context of sett, is only needed if you intend to transfer data. If you are a user who only decrypts data, you do not need an SSH key.
Also, do not confuse SSH keys - used to identify yourself on a remote server - with PGP keys - used to encrypt and sign data.
To generate a new SSH key pair, type the command below in your terminal (Linux
and Mac) or PowerShell (Windows users - to start it, search for "powershell"
in the Start menu). Note that you must replace "alice@example.org" with your
own email. This will generate an SSH key pair using the "ed25519" algorithm,
currently the most secure public-key algorithm:
ssh-keygen -a 100 -t ed25519 -C "alice@example.org"
Windows users who do not have the ssh-keygen command installed, please see section-install-ssh-keygen-windows below.
When executing the ssh-keygen command above, you will be prompted for the following information:
~/.ssh/id_ed25519 on
Linux and MaxOS, and C:\Users\%username%\.ssh\id_ed25519 on Windows.When the command completes, two new files are produced: id_ed25519.pub (the
public key) and id_ed25519 (the private key).
id_ed25519) must be kept
secret. Never share it with anyone, not even your system administrator.
On Linux and MacOS systems, after the public key is generated, its permissions must be changed with the following command (this step is not needed for Windows users):
chmod 600 ~/.ssh/id_ed25519.pub
Not all versions of windows come with the ssh-keygen command pre-installed. If this command is unavailable on your machine, please install it as follows:
Even though it is possible to create an SSH key pair using a password containing non-ASCII characters, it seems like those characters are encoded differently between different operating systems.
As an SSH key might be moved to a machine with another operating system, or encoding might change with a new version, it is impossible to guess the correct encoding in any case. For this reason, we recommend not to use non-ASCII characters to protect SSH private keys.
The sett application allows the encryption of any combination of individual files and directories.
The files are first compressed into a single data.tar.gz archive, which is
then encrypted with the public key of one or more recipient(s), and signed with
the sender’s key. The encrypted data (data.tar.gz.gpg) is then bundled with a
metadata file - a plain text file that contains information about who is
sending the file and to whom it should be delivered - into a single .zip file.
The specifications of the output .zip files produced by sett are described
in the sett packaging specifications section.
sett supports multi-recipient data encryption. This allows the encrypted file to be decrypted by multiple recipients.
sett also ensures the integrity of the transferred files by computing checksums on each file that is packaged, and adding this information to the encrypted data. The integrity of each file is verified automatically upon decryption of the file by sett, providing the guarantee that all files were transferred flawlessly.
Data Transfer Requests: each data transfer into the BioMedIT network must have an authorized Data Transfer Request ID (DTR ID). This ID must be specified at the time the data is encrypted (see below). The ID is added to the encrypted file’s metadata information by sett. A valid and authorized DTR ID value is mandatory for any data transfer into the BioMedIT network. Non-compliant packages will be rejected.
Recipients: each data transfer into the BioMedIT network must be to a recipient assigned to the role of Data Manager for the given project. The recipient’s PGP key must also be approved by the BioMedIT key validation authority. If these conditions are not met, sett will not encrypt the data.
By default, encrypted output files produced by sett are named after the pattern:
<project code>_<YYYYMMDD>T<HHMMSS>.zip
where:
<project code> is the abbreviation/code associated with the project. If no
DTR ID value was provided or if Verify DTR is disabled, no project
code is added as a prefix to the output file name.<YYYYMMDD> is the current date (Year, Month, Day).<HHMMSS> is the current time (Hours, Minutes, Seconds).Example: demo_20220211T143311.zip, here demo is the project code.
Using the sett command line when encrypting to the local file system, it is
possible to completely override the above output file naming scheme by passing
the -o, --output option. Overriding the naming scheme is not possible when
using sett-gui or when encrypting to a remote S3 or SFTP destination.
To encrypt data:
Go to the Encrypt tab of the sett application.
Select files and/or directories to encrypt: using the Add file and Add directory buttons, select at least one file or directory to encrypt.
After adding files/directories, they will be listed in the top box of the tab (see figure above).
Select data sender: in the drop-down list found under Select sender, select your own PGP key (you are the data sender). For most users, there should in principle be only one key in the Sender drop-down menu: their own key.
Select data recipients: add one or more recipients by selecting them from the drop-down list found under Select recipients. Recipients are the people for whom data should be encrypted: their public PGP key will be used to encrypt the data, and only they will be able to decrypt it.
Transfer ID (DTR): Data Transfer Request ID associated with the data package that is being encrypted. Specifying a valid DTR ID is mandatory to transfer data into the BioMedIT network.
For data not intended to be transferred into the BioMedIT network, the DTR ID field can be left empty (or set to any arbitrary value). In this case, Verify package must be disabled (in the Settings tab).
Select destination: destination where to encrypt the data. Possible destinations are:
Local: The local file system. When using this destination, it’s possible to specify:
S3: A remote S3 compatible object store.
SFTP: A remote SFTP server.
When encrypting to a remote destination, a number of options must be specified. Please refer to the transferring files for details about these options.
You are now ready to compress and encrypt the data: click Encrypt package or Send package if you are encrypting to a remote S3 or SFTP destination. A pop-up will appear, asking for the password associated with the sender’s key. After the password is entered, data compression and encryption will start. Progress and error messages are displayed in the Tasks tab.
When the encryption completed successfully, a notification will pop-up with a message that reads: “Encryption job finished”.
At this point, all input files are compressed, encrypted and bundled into a
single .zip file. If the destination was SFTP or S3, data has also been
transferred to the remote destination.
The sett command to encrypt data is the following. Note that the SENDER and
RECIPIENT values can be specified either as a PGP key fingerprint, or as
an email address.
# General syntax:
sett encrypt local --signer SENDER --recipient RECIPIENT --dtr DATA_TRANSFER_ID --output OUTPUT_FILENAME_OR_DIRECTORY FILES_OR_DIRECTORIES_TO_ENCRYPT
# Example:
# long command line options:
sett encrypt local --signer alice@example.com --recipient bob@example.com --dtr 42 --output test_output ./test_file.txt ./test_directory
# short command line options:
sett encrypt -s alice@example.com -r bob@example.com -dtr 42 -o test_output ./test_file.txt ./test_directory
Data can be encrypted for more than one recipient by repeating the flag
--recipient, e.g. --recipient RECIPIENT1 --recipient RECIPIENT2 option:
# In this example, Alice encrypts a set of files for both Bob and Chuck.
sett encrypt local --signer alice@example.com --recipient bob@example.com chuck@example.com FILES_OR_DIRECTORIES_TO_ENCRYPT
local can be replaced with s3 or sftp to encrypt data to a remote
destination.
Adding the --check option will run the encrypt command in test mode, i.e.
checks are made but no data is encrypted.
The data compression level used by sett can be manually adjusted using the
--compression-level option. Compression levels value must be integers between
0 (no compression) and 9 (highest compression). Higher compression levels
produce smaller output files but require more computing time, so you may choose
a lower level to speed-up compression (e.g. --compression-level=1), or a
higher level (e.g. --compression-level=9) to produce smaller output files. The
default level is 6.
Before encrypting data, sett verifies that there is enough free disk space
available on the local machine to save the encrypted output file (relevant is
the current working directory or target folder pointed by --output). If this
is not the case an error message is displayed and the operation is aborted.
Since the compression ratio of the input data cannot be known in advance, sett
uses the conservative estimate that the minimum disk space required is equal to
the total size of all input files to be encrypted.
To automate the encryption process, the --password option can be used to
specify the password of the singer PGP key.
sett performs DTR verification if the --verify option is passed. For
non-BioMedIT-related transfers, the --verify option should not be passed.
--dtr option.
To override the sett output file naming scheme, the
--output option can be used to specify the path and name that the output file
should have.
Data packages can be transferred to remote servers that support one of the following protocols:
To transfer encrypted files:
Go to the Transfer tab of the sett application.
Select encrypted files to transfer: click Select file and select a
.zip file that was generated using the sett application.
Select destination: destination where to transfer the data. Possible destinations are:
S3: A remote S3 compatible object store. When using this destination a number of options must be specified:
SFTP: A remote SFTP server. When using this destination, a number of options must be specified:
Host: URL address of the server where the files should be sent.
User name: the user name with which to connect to the SFTP server.
Destination directory: absolute path of directory where files should be saved on the server.
SSH key location: name and full path of the private SSH key used for authentication to the SFTP server. This is only required if the SSH key is in a non-standard location. Only RSA keys are accepted.
Do not confuse SSH keys - which are used to authenticate yourself when connecting to an SFTP server during file transfer - with PGP keys - which are used to encrypt and sign data.
SSH key password: password associated with the private SSH key given under SSH key location. If your SSH key password contains characters that are not ASCII characters, and that this results in an error, please see the SSH private key with non-ASCII characters section of this guide.
You are now ready to transfer the data. Click Send package and follow the progress of the transfer using the Tasks tab. By default, sett verifies data packages before initializing file transfers. These checks are required within the BioMedIT network, but can be skipped in other contexts by disabling the Verify package checkbox in the Settings tab.
sett command to transfer data:
# General syntax:
sett transfer sftp --host HOST --username USERNAME --base-path DESTINATION_DIRECTORY --key-path SSH_KEY_LOCATION --key-pwd SSH_KEY_PASSWORD FILES_TO_TRANSFER
sett transfer s3 --endpoint ENDPOINT --bucket BUCKET --access-key ACCESS_KEY --secret-key SECRET_KEY FILES_TO_TRANSFER
# Example:
sett transfer s3 --endpoint https://my-node.ch --bucket my-project --access-key pub --secret-key sec encrypted_data.zip
Adding the --check option will run the transfer command in test mode, i.e.
checks are made but no data is transferred.
For SFTP transfers, an SSH key is required for authentication on the host server. The private SSH key can be provided via 2 mechanisms:
--key-path option.--key-path argument should be skipped.To display help for the transfer command: sett transfer --help. To display
help for a specific transfer protocol - e.g. s3: sett transfer s3 --help.
The sett application allows the decryption and decompression of files in a single step. However, only files encrypted with the sett application, or files that follow the sett packaging specifications can be decrypted with sett.
To decrypt and decompress a file:
Go to the Decrypt tab of the sett application (see figure below).
Select a file to decrypt with Select file.
Select destination directory: select a location where to decrypt/decompress the file.
By default, output files are saved to the user’s home directory. This default behavior can be changed by changing Default output directory in the Settings tab.
Click Decrypt package to start the decryption/decompression process. A pop-up dialog box will appear to ask for the password associated with the PGP key used to encrypt the files.
sett command to decrypt data:
# General syntax:
sett decrypt --output OUTPUT_DIRECTORY ENCRYPTED_FILES.zip
# Example:
sett decrypt --output /home/alice/data/unpack_dir /home/alice/data/test_data.zip
To display help for the decrypt command: sett decrypt --help.
To decrypt data without decompressing it, add the -d, --decrypt-only option.
If the -o, --output option is omitted, the data is decrypted in the current
working directory.
If you want to automate the decryption process, you can use the -p, --password
option to provide the password of the recipient PGP key.
The sett desktop app allows a number of options to be customized via its Settings page. For instance, you may change the default output directory, or enable/disable package verification before a transfer.
Each setting has a predefined default value, which is used when first running the tool or if loading the current settings fails for any reasons.
To change the settings in the desktop app, navigate to the Settings tab:
Changes made to Settings become effective immediately. Changes can be reset back to their factory default by clicking on the Reset settings button.
For detailed settings explanations, please refer to the settings section below.
The following options can be set in the config file:
When enabled (the default value), the following verifications are made before encrypting or transferring data:
<project_code>_<date_format>.zip. This ensures no sensitive information
is mistakenly included in the file name.Note that that some of the above verifications require communication with the BioMedIT portal. When using sett outside of a BioMedIT project, this setting should therefore be disabled.
URL of a BioMedIT portal instance. The portal is used for key approval, DTR (Data Transfer Request) validation, and retrieval of data associated with a given DTR (when sett is being used in authenticated mode).
The default value of this setting is: https://portal.dcc.sib.swiss.
Default destination directory for operations such as encryption to the local filesystem or decryption. Defaults are:
$XDG_DATA_HOME or $HOME/.local/share;$HOME/Library/Application Support;{FOLDERID_LocalAppData}.This section displays values of setting that cannot be modified by the user. These settings are displayed here for convenience. They can be copied to the clipboard via a dedicated “copy to clipboard” button.
The sett-cli is stateless by design, meaning that there is no persistent
configuration file where settings can be modified and stored.
Instead, settings can be set via the following shell
environment variables. All settings are optional and have a default value.
SETT_OPENPGP_KEY_PWDPassword to unlock the secret OpenPGP key used to decrypt or sign data. When this environmental variable is set, sett uses its content instead of interactively asking the user to enter a password.
SETT_OPENPGP_KEY_PWD_FILEFull path and name of a file containing the password to unlock the secret OpenPGP key used to decrypt or sign data. When this environmental variable is set, sett uses its content instead of interactively asking the user to enter a password. The file containing the password should not be encrypted.
SETT_PORTAL_URLURL of the BioMedIT portal instance to be used. For details see the description of the GUI Portal URL setting. This setting defaults to https://portal.dcc.sib.swiss.
SETT_OIDC_CLIENT_IDClient ID with which sett should identify with the OpenID Connect issuer
(see SETT_OIDC_ISSUER_URL). Only relevant when using sett in authenticated
mode. This setting defaults to sett.
SETT_OIDC_ISSUER_URLURL of the OpenID Connect issuer used when authenticating with the BioMedIT Portal. Only relevant when using sett in authenticated mode. This setting defaults to https://login.biomedit.ch/realms/biomedit.
sett compresses, encrypts and packages files in a single .zip file whose
specifications are described below. Only files adhering to these specifications
can be transferred or decrypted by sett, and failure to comply with the
specification will generate an error.
sett [.zip]{.title-ref} files have the following structure:
YYYYMMDDThhmmss.zip
├── metadata.json
├── metadata.json.sig
└── data.tar.gz.gpg
└── data.tar.gz
├── content/
| ├── [file1]
| ├── [file2]
| └── ...
└── checksum.sha256
metadata.jsonMetadata file containing the following information:
YYYY-MM-DDThh:mm:ss±ZZZZ (where ZZZZ is the
UTC offset).data.tar.gz.gpg
file. This allows verifying the integrity of the transferred
file without decrypting it, and is used to make sure nothing was
corrupted during the transfer process.sha256.gzip is used by default. An empty string indicates no
compression.PRODUCTION or TEST.metadata.json.sigDetached PGP signature for the metadata.json file.
data.tar.gz.gpgA tarball encrypted using the receiver's public PGP key and signed with the
sender's private key. It is compressed with compression_algorithm.
[file1], [file2]One or several data files can be transferred. Data to be sent can be in any
format, e.g. .txt, .csv, .dat.
checksum.sha256sha256 checksum file of all files present in data.tar.gz. This is used to
make sure nothing was corrupted during the encryption/transfer/decryption
process.
Examples of the content and structure of the metadata and checksum files.
metadata.json
{
"transfer_id": 42,
"sender": "AAABFBC698539AB6CE60BDBE8220117C2F906548",
"recipients": ["D99AD936FC83C9BABDE7C33E1CF8C1A2076818C3"],
"timestamp": "2020-01-29T15:31:42+0100",
"checksum": "a8eb0ee5a6a53326b1c6f9bf94136da5d98a1dc6dceee21d62f694d71c4cf184",
"checksum_algorithm": "SHA256",
"compression_algorithm": "gzip",
"purpose": "PRODUCTION",
"version": "0.7"
}
checksum.sha256
41421f0c4a5353a5a0cdd37de3fd80a840a190ca997ad8044a67c4c1683f7b63
file1.csv
35ba157ed1c3269d731a438c466790a4f481bb49805e2d1f380df0c636792ff6
folder1/file.txt
fd9ebdbcc1a5fc35ded6e78a6b16ef658502c9d0b05dd4a2185d0f94ccf165cf
folder1/folder2/file.txt
On Windows, you can check if you are using a proxy server to access the Internet by going to Start > Settings > Network & Internet > Proxy (Windows 10). If the slider under Use a proxy server is "off", no proxy is being used.
If you are told that you need to set a proxy, input the Address and Port details and click Save. When in doubt, please consult your IT department.
On Mac OS the proxy information is located under the System Preferences > Network > Advanced > Proxies tab of the network interface, usually Ethernet or Wi-Fi.
In order to run sett behind a proxy, the shell environment variable
ALL_PROXY or HTTPS_PROXY must be set. This is the recommended and global way
to specify a proxy. Note that, while certain programs support a proxy option
(e.g. pip with --proxy), there is currently no such option in sett.
Example:
ALL_PROXY=https://host.domain:port sett-gui
To report a problem with sett or if you have a question not covered in the present documentation please open an issue on our public gitlab repo https://gitlab.com/biomedit/sett-rs/-/issues
sett is licensed under the LGPLv3 (GNU Lesser General Public License) and the source code is available at https://gitlab.com/biomedit/sett-rs
sett is developed as part of the BioMedIT project.
The benchmarks were run on an Apple Silicon M3 machine with 16-Core CPUs, 128GB memory, and 2TB disk. Version 5.0.0 of sett CLI was used.
One single binary file was used, with default compression (Zstandard, level 3). Decreasing the compression level and/or using text data will influence the results.
Transfer speed is highly dependent on the infrastructure connecting the data sender and recipients. In these benchmarks, data was transferred to the same machine sending the data, therefore representing a best-case scenario in terms of transfer speed.
All values shown un the results below are averages of 5 runs performed with hyperfine.
The table and figure below present throughput speeds for different sett workflows:
| Workflow | Throughput (MB/second) | Throughput (GB/minute) |
|---|---|---|
| Encrypt local | 127.17 | 7.63 |
| Encrypt S3 | 79.36 | 4.76 |
| Encrypt SFTP | 58.84 | 3.53 |
| Transfer S3 | 159.95 | 9.60 |
| Transfer SFTP | 126.95 | 7.62 |
| Decrypt | 155.94 | 9.36 |
Throughput as function of file size: as can be seen, the curves in the figure below are linear, indicating that the throughput is constant regardless of the file size.
(click on the figure to enlarge).