To navigate, click on the topics on the left-hand side of the navigation bar or use the search functionality.
This is the multi-page printable view of this section. Click here to print.
User Guides
- 1: sett - Secure Encryption and Transfer Tool
- 1.1: What is sett?
- 1.2: Quick start guide
- 1.3: Download sett
- 1.4: OpenPGP key management
- 1.5: Encrypting, transferring, and decrypting data
- 1.6: Generating SSH keys
- 1.7: Settings
- 1.8: Data package specification
- 1.9: Docker image
- 1.10: Frequently asked questions and bug reports
- 1.11: Source code
- 1.12: Benchmarks
- 2: BioMedIT Portal User Guide
- 2.1: Login
- 2.2: Home
- 2.3: Profile
- 2.4: Groups
- 2.5: Projects
- 2.6: Data Transfers
- 2.7: Contact Us
- 2.8: Central Services
- 3: Quick links for SWITCH edu-ID
- 4: SPHN-DCC Confluence
1 - sett - Secure Encryption and Transfer Tool
Welcome to the official sett documentation page.
Please use the table of contents menu to the left or below to navigate the topics.
Table of Contents
1.1 - What is sett?
sett stands for “Secure Encryption and Transfer Tool” and is an application that facilitates and automates data packaging, encryption, and transfer. It is written in Rust, a fast and memory-safe programming language. sett is available both as a desktop and a command line application.
sett is developed as part of the BioMedIT project. It is licensed under the GPLv3 (GNU General Public License) and the source code is available from its public GitLab repository.
BioMedIT
When using sett to transfer data into the SPHN BioMedIT network, a number of additional constraints apply. These specific instructions are highlighted throughout this documentation in BioMedIT labeled boxes, just like this one.1.2 - Quick start guide
For the complete guide on how to use sett, please refer to Encrypting, transferring, and decrypting data and OpenPGP key management.
GUI (Graphical User Interface)
Initial setup
- Download sett-gui from the download page. If you downloaded an installer, install sett-gui by double-clicking on the installer file.
- Run sett-gui by double-clicking on the executable file or by launching the installed app.
Key management
-
If you do not already have a private/public OpenPGP key pair, go to the Keys tab and create one clicking on Add > Generate new key pair. See also the instructions given in the Generate a new public/private OpenPGP key pair section.
You should then see your new key listed in the Keys tab, along with “Private” label that indicates that the private material for this key is present in the local keystore.
BioMedIT
Your OpenPGP key must be registered with the BioMedIT portal before it can be used to sign or decrypt data packages. Please see the Register your public OpenPGP key in the BioMedIT portal section for details. -
If not already done, download the public OpenPGP key of the recipient(s) to whom you intend to send data (or from whom you will receive data). Go to the Keys tab and click on Add > Import from keyserver. See also the instructions given in the download public OpenPGP keys from the keyserver section.
-
Just after downloading the recipient’s OpenPGP key, verify it to make sure that it is genuine. This can be done by either:
- If you are a BioMedIT user: verify that the recipient’s key is labelled with a green Approved label. You can also expand the details of the key by clicking on the key in the list or on the small down arrow button to the right and verify that the Approval status is set to “Key is approved on Portal”, and the Revocation status is set to “Valid”.
- Alternatively, contact the key owner and verify the key fingerprint with them.
Encrypting and sending data
Authenticated mode (BioMedIT)
sett provides “authenticated mode” that simplifies the encryption and transfer process by providing a list of available Data Transfer Requests (DTRs) and automatically fetching the required destination parameters and credentials (only available for the S3 destination).
To access this mode, go to the Profile tab and click “Sign in”. You will be redirected to the BioMedIT authentication system. After successful authentication, proceed to the Encrypt and Transfer Data tab.
-
Go to the Encrypt and Transfer Data tab.
-
Add one or more files and directories to encrypt by clicking the Add files or Add directories buttons.
-
Select sender: select your own OpenPGP key. This is the key that will be used to sign the encrypted data.
-
Select recipients: add one or more recipients by selecting them in the drop-down. These are the keys that will be used to encrypt the data, i.e. only these recipients will be able to decrypt the data.
BioMedIT
The selected recipients must be approved Data Managers of the project for which data is being encrypted. -
Data Transfer ID: specifying a valid Data Transfer Request ID is mandatory when a data package is transferred into the BioMedIT network. For other destinations, the Data Transfer ID field can be left empty (or set to any arbitrary value), and the Verify package checkbox must be disabled (in the Settings tab).
BioMedIT
The Verify package checkbox should always be enabled, since a valid Data Transfer ID is required by the BioMedIT network. -
Select destination: select local and choose a destination directory to encrypt to your local file system. Select s3 or sftp to encrypt and transfer directly to an S3 object store or an SFTP server, respectively.
-
Click Encrypt data (local) or Encrypt and transfer data (s3 or sftp) to run the encryption workflow on your data.
Sending existing data packages
-
Go to the Encrypt and Transfer Data tab.
-
Select a file to transfer using the add sett Package button.
-
Select the Destination to be used (sftp, s3).
-
Enter the required destination parameters.
BioMedIT
For transfers into the BioMedIT network, the destination parameters are provided by:
- Your local BioMedIT node, for transfers over sftp;
- By BioMedIT Portal, for transfers over s3. Please see the dedicated section of the Portal user guide for more information.
-
Click Transfer data to start transferring your data package.
Decrypting data
- Go to the Decrypt tab.
- Select a data package to decrypt using the Select Package button.
- Specify your desired destination directory.
- Click on Decrypt package.
CLI (Command Line Interface)
The main commands to manage keys, encrypt, transfer and decrypt data with sett command line interface are given here.
sett help
In the CLI each command and subcommand provides a help message that can be
used to get more information about the available options
(-h
and --help
for short and long help message respectively).
For example:
sett --help
sett encrypt local -h
sett encrypt local --help
# Generate a new key pair:
sett keys generate
# Import sender/recipient(s) public keys:
sett keys import from-keyserver alice@example.com
# Data encryption:
sett encrypt local --signer alice@email.com --recipient bob@example.com --output . FILES_OR_DIRECTORIES_TO_ENCRYPT
# Data transfer
# to S3 object store:
sett transfer s3 --endpoint ENDPOINT --bucket BUCKET --access-key ACCESS_KEY --secret-key SECRET_KEY FILES_TO_TRANSFER
# to SFTP server:
sett transfer sftp --host HOST --username USERNAME --base-path DESTINATION_DIRECTORY --key-path SSH_KEY_LOCATION --key-pwd SSH_KEY_PASSWORD FILES_TO_TRANSFER
# Data decryption:
sett decrypt local ENCRYPTED_FILES.zip
1.3 - Download sett
Show all downloadsYou can download the latest version of sett for your operating system by clicking on the corresponding link above.
Downloading a specific version of sett
For a complete list of available versions, please visit the releases page.
Updating sett
As only the latest version of sett is officially supported and guaranteed to work, it is strongly recommended to always keep your local installation of sett up-to-date.
If you installed sett using the installer available for your operating system, sett will automatically check for updates and prompt you to install the newest version as soon as it's available.
If you are running a portable executable, make sure you regularly check this page and download the latest version of the executable.
1.4 - OpenPGP key management
To encrypt and decrypt data, sett uses public key cryptography. If you are not familiar with public key cryptography concepts such as public and private keys, revocation signatures, or keyservers you are advised to read this introductory section.
All key management operations can be performed using both the graphical user interface (GUI) and the command line interface (CLI) of sett.
sett help
In the CLI each command and subcommand provides a help message that can be
used to get more information about the available options
(-h
and --help
for short and long help message respectively).
For example:
sett --help
sett encrypt local -h
sett encrypt local --help
Generate a new public/private key pair
A prerequisite to encrypt, decrypt, and transfer files with sett is to have a public/private PGP key pair.
Important
Always keep your private key, its password, and its revocation signature in a secure location. Never share any of this information with anyone.Generate a new key pair (GUI)
To generate a new public/private key pair:
-
Go to the Keys tab and click on + Add. From the drop-down menu, choose Generate new key pair. A dialog box will appear.
-
Fill-in the required fields:
- Name: key owner first and last name.
- Comment: an optional comment to help identify the owner of the key (e.g., owner’s institution).
- Email: enter the email to be associated with the key pair.
- Password: the password must be at least 10 characters long, and it is highly recommended that it contain a mix of letters, numbers and special characters.
- Expiration date: a date in the future to ensure that the key is not used indefinitely (default: 3 years). It is possible to extend the expiration date later.
Important
It is not possible to retrieve or reset a password from a generated key. Make sure to remember it and store it in a safe place (e.g. a password manager). -
Click Next to see the summary.
-
Click “Generate key” to create the new key pair (this can take a few seconds).
-
A new pop-up window will appear to confirm that the key was generated successfully and to display the revocation signature associated with the new key.
Copy the revocation signature to a safe location, ideally a password manager, and click Next.
-
The next screen of the pop-up window allows to upload the public part of the key to the keyserver. Make sure to check the checkbox to associate the key with your email address. To establish this association, the keyserver will send an email to verify that you are the legitimate owner of the email associated with the key. It also means that other people will now be able to search for your key on the keyserver using your email address, otherwise your key will only be searchable using its full fingerprint.
-
The new key should now be listed in the keys tab.
-
Now that your new OpenPGP key is created, make sure to register it in the BioMedIT portal.
Generate a new key pair (CLI)
To generate a new key pair using sett command line interface:
sett keys generate --name "Alice Smith" --email alice.smith@example.com
Important
During the key generation process, sett will ask for a pass phrase (password) that will protect the private key. Please use a reasonably strong password and make sure to save it in a password manager.
If the password is forgotten, there is no way to retrieve or re-create it. Your OpenPGP key pair will thus become unusable and will need to be revoked.
After creating the new key, sett will display a revocation signature for it.
Make sure you keep it in a safe place, such as a password manager. Revocation
signatures can optionally be automatically exported to a file by using the
--rev-sig
option:
sett keys generate --name "Alice Smith" --email alice.smith@example.com --rev-sig alice.rev
List keys
List keys (GUI)
Go to the Keys tab.
List keys (CLI)
Use the keys list
subcommand.
sett keys list
Export/import private keys
In some situations (e.g., new computer setup, remote sett environment) you might need to copy or move your private key to a different machine. This can be done by exporting the private key to a file, transferring the file to the new machine, and importing it.
Export key (GUI)
Click on the ...
icon next to the key you want to export and select
“Export public and private key”.
Export key (Cli)
Use the keys export
subcommand:
# The key identifier can be its fingerprint, key ID or the user's email address.
sett keys export -o private_key.pgp -p alice.smith@example.com
Note that keys can also be exported in ASCII format (instead of binary) by
adding the -a, --armor
option.
Import key (GUI)
Click on the + Add
button and select “Import from file or clipboard”.
Import key (CLI)
Use the keys import
subcommand.
sett keys import from-file -p private_key.pgp
Note the usage of the -p
option to import both the private and public parts of
the key. If -p
is omitted, only the public part of the key is imported.
Verify that the key has been correctly imported with sett keys list
.
Ensure that you store any backed-up secret keys in a secure location and in an encrypted form (typically in a password manager).
Upload public key to the keyserver
sett allows users to upload their public OpenPGP key to a keyserver.
BioMedIT
sett uploads keys to the OpenPGP keyserver.
keys.openpgp.org is a public keyserver for the distribution and discovery of public PGP keys. This service is GDPR-compliant, based on the open source software Hagrid, and already used by over 500'000 users worldwide. For more information about the service, please see this link.
Upload public key to the keyserver (GUI)
- Click on the
...
icon next to the key you want to upload and select “Upload public key to keyserver”. - A dialog box will appear to ask for confirmation.
- Make sure to check the checkbox to associate the key with your email address. To establish this association, the keyserver will send an email to verify that you are the legitimate owner of the email associated with the key. It also means that other people will now be able to search for your key on the keyserver using your email address, otherwise your key will only be searchable using its full fingerprint.
- Click Upload to keyserver.
- If you have selected the “associate key with email” checkbox, you will shortly receive an email from keyserver@keys.openpgp.org that contains a confirmation link to prove that you have access to your email.
- Open the email and click on the link to associate your key with your email address. Other users now can find your public key with your email address.
Upload public key to the keyserver (CLI)
Use the keys upload
subcommand.
sett keys upload alice.smith@exaple.com
In order to also trigger email verification for your key, add the -v, --verify
option:
sett keys upload --verify alice.smith@exaple.com
Register public OpenPGP key in the BioMedIT portal
BioMedIT
Registering your public OpenPGP key in the BioMedIT portal is mandatory to be able to use it within the BioMedIT project (e.g. create data transfer requests, encrypt data, etc.).If you are not a BioMedIT user, this section is not relevant for you and can be skipped.
BioMedIT OpenPGP key status
OpenPGP keys used to encrypt, sign, and decrypt data within the BioMedIT network require the approval of the BioMedIT key validation authority. The information of whether a key is trusted or not is stored as key status in the BioMedIT portal. This is the reason why all OpenPGP keys used within BioMedIT must be registered with the BioMedIT portal.
When an OpenPGP key is first registered in the BioMedIT portal, its status is
initially set to PENDING
(i.e. it is awaiting approval). A key must have the
APPROVED
status before it can be used to encrypt or sign data packages
within the BioMedIT network.
The list of key statuses is as follows:
PENDING
: a key approval request was submitted, but the key has not been approved yet. This is a manual process and can take from a few hours or up couple of days.APPROVED
: key is approved for usage within the BioMedIT network. Only approved keys can be used to encrypt, sign, and decrypt data packages within the BioMedIT network.APPROVAL-REVOKED
: approval of the key has been revoked by the BioMedIT key validation authority.KEY-REVOKED
: key has been revoked by its owner.REJECTED
: key is not trusted by the BioMedIT key validation authority.DELETED
: key has been removed from the keyserver by its owner.UNKNOWN KEY
: key has not been registered on the BioMedIT portal. If it is your own key, please register it. If it is the key of someone else, please ask them to register their key.
To verify that a key is trusted, sett connects to the BioMedIT portal and retrieves the status of the key. For this reason, it is important that BioMedIT users register their OpenPGP key with the BioMedIT portal.
In cases where sett is used outside of the BioMedIT project, or the portal is
not reachable, sett can still be used to encrypt, decrypt, and transfer data.
In this case, you need to uncheck Verify package
in the Settings tab.
The status of a public key in the BioMedIT portal can be easily checked in GUI by going to the Keys tab and clicking on one of the keys. Approved keys will also have a green “Approved” badge.
How to register your public OpenPGP key in the BioMedIT portal?
Only one active key
Each BioMedIT user can only have 1 active OpenPGP key registered in the BioMedIT portal at any time.
If you wish to replace your currently active key with another one, please connect to the BioMedIT portal, go to the Profile / OpenPGP Keys tab and click on the “Retire Key” icon (Actions column) associated with your key.
Key email must match user email
The email associated with your OpenPGP key must be one of the emails associated to the SWITCH edu-ID account that you are using with the BioMedIT portal.
If the email associated to your OpenPGP key is different (e.g. a shared service email is used for your OpenPGP key), please contact the BioMedIT support.
To register a new OpenPGP key with the BioMedIT portal, proceed as follows:
-
Make sure you have successfully uploaded your key to the keyserver and that you have completed the email verification procedure with the keyserver (i.e. your key must be verified with the keyserver).
-
To make sure that your key is present on the keyserver and is verified, go to the keyserver home page in your browser and search for the email associated with your key. You should get a message saying that a key was found for your email address and the corresponding key fingerprint should be displayed.
-
Copy the fingerprint (40-character string) of your key.
-
Log in to the BioMedIT portal.
-
Go to the Profile / OpenPGP Keys.
-
Click on the "+ OPENPGP KEY" button, a dialog box will open.
Note: if the button is missing, it is probably because you already have an active key in the portal. Each user can only have 1 active key at a time - see the information box above.
-
Enter your full key fingerprint (must be exactly 40 characters long) in the dialog box, then press the green search icon to the right.
This will retrieve the User ID and Email address associated with the fingerprint from the keyserver and display them in the dialog box.
-
Verify the user ID and mail address. If they are correct for your OpenPGP key, then click on Confirm.
-
A request to approve your key has now been sent to the BioMedIT key validation authority. Generally requests are processed quickly (in a matter of hours), but occasionally it might take slightly longer as this is a manual process.
Please contact the BioMedIT support if your key has not been approved after a couple of days.
Download public keys from the keyserver
In order to encrypt data for a specific recipient (who will be able to decrypt it), you will need to have the public OpenPGP key of that recipient(s) available in your local keyring.
Download public keys (GUI)
To download a public PGP key from the keyserver:
-
In the Keys tab, click on + Add. From the drop-down menu, choose Import from keyserver. A dialog box will open, allowing to search for public keys stored on the keyserver.
-
In the search field, enter either the full email address or fingerprint of the public key you are looking for and click Import key.
For instance, if searching for the key “Bob Test Key bob@example.com” with fingerprint “AEED7A96F339881F6FE8291464A1E0150613807D”, one can search for either “bob@example.com” or “AEED7A96F339881F6FE8291464A1E0150613807D”.
-
If the key you are looking for was found, you will see a success message with key details.
Download public keys (CLI)
The keys import from-keyserver
subcommand can be used to download public OpenPGP
keys from the keyserver.
sett keys import from-keyserver alice.smith@example.com
Remove your public keys from the keyserver
While it is not possible to remove an actual key from the keyserver, it is possible to remove all personally identifiable information from it (user ID and email). Such keys are called unverified.
To remove personal information associated with a key, go to the keyserver’s manage key page, enter the email associated to the key and click on Send link.
You will receive an email with further instructions on how to proceed to remove your key’s user ID and email from the keyserver.
Warning
Removing a key’s user ID and email from the keyserver makes it no longer searchable by email. It remains searchable by its full fingerprint.
Important: keys without user ID and email cannot be used within the BioMedIT network. Only remove your personal identifying information from a key if you are no longer using it.
Delete keys from your local machine
Users are strongly discouraged from deleting keys from their local keystore. Instead of deleting a key, users should revoke it. Revoking a key informs others that the key should not be used anymore, and is a safer way to handle unused or compromised keys.
Deleting private keys is possible from sett.
-
GUI: Click on the
...
icon next to the key you want to delete and select “Delete”. -
CLI: Use the
keys delete-private-key
subcommand.sett keys delete-private-key alice.smith@exaple.com
Deleting public keys is not possible with sett. However, it is possible to delete a public key by manually deleting the file containing the keys using your file explorer or a shell command.
In the local sett public keystore, keys are stored as files.
They are named after the fingerprint of the key’s primary key.
For instance, a key with fingerprint
3b17f529665fe012ef54f4a1714fdf98b6e828df
would be stored under:
- Public key:
<keystore-path>/3b/17f529665fe012ef54f4a1714fdf98b6e828df
The location of the keystore (<keystore-path>
above) is operating-system
dependent:
- Linux:
~/.local/share/pgp.cert.d
- Windows:
%UserProfile%\AppData\Roaming\pgp.cert.d
- MacOS:
~/Library/Application Support/pgp.cert.d
You can also get the exact location of the key in sett GUI:
- Click on the
...
icon next to the key you want to delete and select “Delete”. - A dialog box with important information about key deletion, as well as the exact location of the key will be displayed.
Generate a revocation signature
A prerequisite for revoking an OpenPGP key is to have generated a revocation signature for it. If the OpenPGP key to revoke was generated with sett, you should already have a revocation signature ready to use. If you do not have a revocation signature yet, you can generate one with GUI:
-
Click on the
...
icon next to the key you want to generate revocation signature and select “Create revocation signature”. -
Fill-in the required fields:
- Reason: the reason for revoking the key.
- Message: a short explanation to the reason for revoking the key.
- Password: the password for your private key.
-
Click Generate signature.
-
A new pop-up window will appear to confirm that the revocation signature was generated successfully. Copy the revocation signature to a safe location, ideally a password manager (anyone with access to a revocation signature can revoke the key for which it was generated).
Revoke your key
If a private OpenPGP key has been compromised, is no longer usable (e.g. password is lost), or should no longer be used for any other reason, it must be revoked.
A prerequisite for revoking a PGP key is to have generated a revocation signature for it. If the OpenPGP key to revoke was generated with sett, you should already have a revocation signature ready to use. If you do not have a revocation signature yet, please generate one by referring to the Generate a revocation signature section.
Warning
A revoked key cannot be “un-revoked”. Only revoke a key if you are certain you want to do so. Revoked keys can no longer be used to encrypt/decrypt data with sett.Revoke your key (GUI)
-
Click on the
...
icon next to the key you want to revoke and select “Revoke”. -
Either paste you revocation signature from the clipboard or load it from a file.
Warning: proceed with caution, a revoked key cannot be “un-revoked”.
-
Click on Revoke key.
-
When expanding the revoked key from the List of available keys, you should now see the Revocation status as “Revoked”. From this point on, the key can no longer be used with sett.
-
If you have previously shared your key via a keyserver (e.g. keys.openpgp.org), you must also re-upload your revoked key to that keyserver.
This will allow other users to update their local copy of your public key, informing them that it is no longer valid. To upload your revoked public key, please refer to the Upload your public PGP key to the keyserver .
If your key was never present on any keyserver, this step should be skipped.
Revoke your key (CLI)
Use the keys revoke
subcommand.
sett keys revoke alice.smith@example.com compromised "My dog ate it"
After a key has been revoked, it must be uploaded again to any keyserver(s) where it is present, so that the revocation can be shared with others. This can be done with sett as illustrated in the Upload your public PGP key to the keyserver.
Update expiration date for your key
It is possible to set an expiration date for an OpenPGP key. This is useful to ensure that the key is not used indefinitely. It is possible to extend the expiration date at any time.
Update expiration date (GUI)
sett GUI warns you when your key is about to expire. On the Keys tab, you will notice an Expiring badge starting from three months before the expiration date. On top of that, a warning will be displayed every time you use that key for signing or decryption.
To set a new expiration date for your key from the GUI:
-
Click on the
...
icon next to the key you want to update the expiration date for. -
Click on Update expiration date.
-
Choose a new expiry date.
-
Click on Set new expiry date.
-
Unlock your key by entering the password, if needed.
Update expiration date (CLI)
sett CLI warns you every time you use a key which is about to expire for signing or decryption. This happens starting from three months before the expiration date.
To set a new expiration date for your key from the CLI:
sett keys expire alice.smith@example.com --expiration 3y
For details about the supported time specifications for the --expiration
option, run:
sett keys expire --help
Migrating keys from the GnuPG keyring to the sett keystore
Public OpenPGP keys located in your GnuPG keyring are not automatically detected by sett, and they must be migrated to sett’s public keystore. In contrast, private keys present in GnuPG can be used in sett without migration (however, it is still possible to migrate them).
Note that to migrate public keys (e.g., keys from other people), you can simply re-download them from the keyserver as shown in the Download public PGP keys from the keyserver section.
-
GUI:
- In the Keys tab, click on + Add. From the drop-down menu, choose Import from GnuPG. A dialog box will open, containing a list of keys in your GnuPG keyring.
- Look for the correct key and click on Import. You should now see the key listed in the List of available keys. Note that if you are importing a private key, you will be asked to enter its password.
-
CLI: use the
keys import from-gpg
subcommand.# The search term can be an email or fingerprint. sett keys import from-gpg alice.smith@example.com
Introduction to public-key cryptography and OpenPGP
Public-key cryptography is a method for secure communication between two or more users. In this system, each user has a pair of unique keys consisting of a private key and a public key. Public and private keys are linked in the sense that, data encrypted with a given public key can only be decrypted with the matching private key, and data signed with a given private key will only be recognized by the matching public key.
Because these keys are based on the OpenPGP protocol, they will here be referred to as OpenPGP keys.
Public and private OpenPGP keys:
-
Public keys are used to encrypt data, as well as for verifying signatures made on files or emails. By design, public keys are intended to be shared with other people and therefore no particular effort is required to keep them secret. In fact, public keys are often uploaded to public servers, known as keyservers, where they are accessible to anyone. No password is required to use a public key.
Typically, public keys are used by data senders to encrypt data for one or more recipient(s), and by data recipients to verify signatures of files or emails (to ensure the sender is genuine).
-
Private keys, sometimes also referred to as a secret keys, are used to decrypt data, sign files and sign other people’s public keys. To increase security, private keys should always be password protected.
Important
Private keys and their password must be kept secret at all times. Never share your private key or password with anyone.
Private keys should be stored in a directory only accessible by the key owner, and their password should be stored in a password manager.
Typically, private keys are used by data recipients to decrypt data, and by data senders to sign the files they encrypt.
Important
If anyone else than the legitimate owner has access to a private key and/or its password, the key pair is considered compromised. It must be revoked, and never used again.sett uses the open source implementation of public-key cryptography provided by Sequoia-PGP: a modular OpenPGP implementation in Rust.
It is possible - and often desirable - to both encrypt and sign a file. This ensures that the data can only be read by the intended recipient, and that the recipient can be confident the sender is legitimate. This is precisely what sett does:
- Encrypting files, so that only the intended recipient(s) can read them.
- Signing files, so that the recipient(s) can trust the sender is genuine.
Key fingerprints
Each pair of public/private OpenPGP keys is identified by a unique fingerprint. Fingerprints are 40 characters long hexadecimal strings (digits and upper case A-F letters) that look like this:
238565936FCFF3F200219990941A3EC20555F781
Since nothing is preventing two OpenPGP keys to have the same user name and email address, it is critical that users always verify the genuineness of new keys before (or just after) importing them into their local keyring (i.e. their local OpenPGP key database).
Ensuring a key is genuine can be done in two different ways:
- Ask the key owner to provide their key’s fingerprint via a trusted communication channel (e.g. over the phone), and then verify that the fingerprint of the newly imported key does indeed match the fingerprint communicated to you by its owner.
- Using sett-gui, verify that the key status of the key is APPROVED (can only be checked after you imported the key).
File encryption
In public key cryptography, the sender encrypts a file using one or more recipient(s) public key(s). Once a file is encrypted, no one can read the file without having access to a private key that matches the public key(s) used for encryption. This ensures that only the intended recipient(s) can decrypt the file, because they are the only one to have access to the matching private key.
File signing
The objective of file signing is to guarantee to the recipient of a file (or email) that the sender is genuine, and not someone else trying to impersonate the sender.
To achieve this, the sender signs the file with their private key (password required), and shares their public key with the recipient (typically via a keyserver). The recipient can then validate the authenticity of the signature using the public key of the sender. Since public keys are non-sensitive, they can be distributed publicly. In fact they are intended for this purpose, hence their name.
Revocation signatures
In the unfortunate event that a user either i) forgets their private key’s password or ii) have their private key and password stolen/compromised, they will need a way to let other people know that their public key should no longer be trusted and used.
This is because:
- If the password was forgotten: the key owner won’t be able to decrypt data anymore.
- If the private key was compromised: someone else might be able to decrypt data encrypted with the public key, and to illegitimately sign files!
This situation is what revocation signatures are for: by applying a revocation signature to a public key, and then sharing the revoked key with others (e.g. via a keyserver), the key owner signals that their key is now “revoked” and should no longer be trusted nor used. After a key has been revoked, it can no longer be used to encrypt/decrypt data with sett.
Revocation signatures can be generated at anytime from the private key, but the best practice is to generate them directly after a new key pair is created. This ensures that the revocation signature will be available even if the private key or its password is lost.
Note
sett will automatically generate a revocation signature each time a new key is created.Since anyone with access to a revocation signature will be able to revoke the associated key, revocation signatures must be stored securely - e.g. in a password manager - and should never be shared with anyone.
Important
- Revocation signatures must be stored in a safe location (e.g. a password manager) and never shared with anyone.
- It is best to generate a revocation signature immediately after a new public/private key pair is created.
- The key revocation process is described in Revoke your OpenPGP key section.
Exchanging public keys via a keyserver
Encrypting files for a specific recipient requires to have the recipient’s public key in one’s local keyring (a keyring is a local database containing OpenPGP keys). Similarly, verifying a signature on a file or a public key requires to have the signee’s public key available in one’s local keyring.
Public keys are not sensitive data, and therefore can be sent unencrypted via email. However, when having frequent key exchanges between multiple actors, sending public OpenPGP keys around by email quickly becomes cumbersome. A solution to this problem is using a so called keyserver to share public keys. Keyservers are public or private servers whose sole purpose is to store public OpenPGP keys and allow users to search for them.
BioMedIT
Within the BioMedIT project, all exchanges of public keys are done via the public keyserver at https://keys.openpgp.org. This keyserver can be accessed directly from sett or via a web browser.1.5 - Encrypting, transferring, and decrypting data
The instructions below describe the process of encrypting, transferring, and decrypting data. sett provides both a graphical user interface (GUI) and a command line interface (CLI) for these operations.
sett help
In the CLI each command and subcommand provides a help message that can be
used to get more information about the available options
(-h
and --help
for short and long help message respectively).
For example:
sett --help
sett encrypt local -h
sett encrypt local --help
Encrypting and sending data
sett allows the encryption of any combination of individual files and directories.
The files are first compressed into a single data.tar.gz
archive, which is
then encrypted with the public key of one or more recipient(s), and signed with
the sender’s key. The encrypted data (data.tar.gz.gpg
) is then bundled with a
metadata file - a plain text file that contains information about who is
sending the file and to whom it should be delivered - into a single .zip
file.
The specifications of the output .zip
files produced by sett are described
in data package specification.
sett supports multi-recipient data encryption. This allows the encrypted file to be decrypted by multiple recipients.
sett also ensures the integrity of the transferred files by computing checksums on each file that is packaged, and adding this information to the encrypted data. The integrity of each file is verified automatically upon decryption of the file by sett, providing the guarantee that all files were transferred flawlessly.
BioMedIT
Data Transfer Requests: each data transfer into the BioMedIT network must have an authorized Data Transfer Request ID (DTR ID). This ID must be specified at the time the data is encrypted (see below). The ID is added to the encrypted file’s metadata information by sett. A valid and authorized DTR ID value is mandatory for any data transfer into the BioMedIT network. Non-compliant packages will be rejected.
Recipients: each data transfer into the BioMedIT network must be to a recipient assigned to the role of Data Manager for the given project. The recipient’s PGP key must also be approved by the BioMedIT key validation authority. If these conditions are not met, sett will not encrypt the data.
Output file naming scheme
By default, encrypted output files produced by sett are named after the pattern:
<project code>_<YYYYMMDD>T<HHMMSS>.zip
where:
<project code>
is the abbreviation/code associated with the project. If Verify package is disabled, no project code is added as a prefix to the output file name.<YYYYMMDD>
is the current date (Year, Month, Day).<HHMMSS>
is the current time (Hours, Minutes, Seconds).
Example: demo_20220211T143311.zip
, here demo
is the project code.
Using the sett command line interface when encrypting to the local file
system, it is possible to completely override the above output file naming
scheme by passing the -o, --output
option. Overriding the naming scheme
is not possible when using sett-gui or when encrypting to a remote S3 or
SFTP destination.
Encrypting and sending data (GUI)
Authenticated mode (BioMedIT)
sett provides “authenticated mode” that simplifies the encryption and transfer process by providing a list of available Data Transfer Requests (DTRs) and automatically fetching the required destination parameters and credentials (only available for the S3 destination).
To access this mode, go to the Profile tab and click “Sign in”. You will be redirected to the BioMedIT authentication system. After successful authentication, proceed to the Encrypt and Transfer Data tab.
Data can be encrypted and either saved to the local file system, or sent directly to a remote server that supports one of the following protocols:
- S3 object storage
- SFTP
-
Go to the Encrypt and Transfer Data tab.
-
Select files and/or directories to encrypt: using the Files and Folders buttons, select at least one file or directory to encrypt. Files and directories can also be dragged and dropped into sett.
Note
If you already have an encrypted data package in your local file system, you can select it with sett Package (or drag and drop it into sett), then choose a remote destination (continue reading at Select destination). -
Select data sender: in the drop-down list found under Select sender, select your own OpenPGP key (you are the data sender). For most users, there should be only one key in the Sender drop-down menu: their own key.
Note
The Sender key is used to sign the encrypted data, so that the recipient(s) can be confident that the data have been sent by a trusted source. -
Select data recipients: add one or more recipients by selecting them from the drop-down list found under Select recipients. Recipients are the people for whom data should be encrypted: their public OpenPGP key will be used to encrypt the data, and only they will be able to decrypt it.
BioMedIT
Only recipients assigned to the role of Data Manager of the project for which data is being encrypted are permitted as data recipients. -
Data Transfer ID: Data Transfer Request ID associated with the data package that is being encrypted. Specifying a valid DTR ID is mandatory to transfer data into the BioMedIT network.
For data not intended to be transferred into the BioMedIT network, the DTR ID field can be left empty (or set to any arbitrary value). In this case, Verify package must be disabled (in the Settings tab).
BioMedIT
DTR ID field is mandatory. Only files encrypted with a valid and authorized DTR ID value can be transferred into the secure BioMedIT network. For this reason, BioMedIT users should always leave the Verify package checkbox enabled. -
This section is only visible if “Enable extra metadata” is selected in the Settings tab.
Here you can add extra metadata to the data package in the form of key-value. This metadata will be stored in the metadata file of the encrypted data package. After inserting the key and value, click the + button (or hit “Enter”) to add the new entry. You can add multiple key-value pairs.
-
Select destination: destination where to encrypt (and send) the data.
-
Local: The local file system. When using this destination, it’s possible to specify:
- Output location: directory where the encrypted file should be saved. By default, output files are saved to the user’s home directory. This default behavior can be changed by changing Default output directory in the Settings tab.
-
S3: A remote S3 compatible object store.
-
SFTP: A remote SFTP server.
When encrypting to a remote destination, a number of options must be specified. Please refer to remote destination options for details.
-
-
You are now ready to create an encrypted data package: click Encrypt data or Encrypt and transfer data if you are encrypting to a remote S3 or SFTP destination. A pop-up will appear, asking for the password associated with the sender’s key. After the password is entered, data compression and encryption will start. Progress and error messages are displayed in the Tasks tab.
When the encryption completed successfully, a notification will pop-up with a message that reads: “Encryption job finished”.
At this point, all input files are compressed, encrypted, and bundled into a
single .zip
file. If the destination was S3 or SFTP, data has also been
transferred to the remote destination.
Encrypting and sending data (CLI)
To create an encrypted data package and save it to the local file system, use
the encrypt
subcommand. If you already have an encrypted data package in your
local file system and want to transfer it to a remote destination, use the
transfer
subcommand. Both subcommands share the same options for specifying
the remote destination.
Note
SENDER
and RECIPIENT
values can be specified either as an OpenPGP key
fingerprint, or as an email address.
# General syntax:
sett encrypt local --signer SENDER --recipient RECIPIENT --dtr DATA_TRANSFER_ID --output OUTPUT_FILENAME_OR_DIRECTORY FILES_OR_DIRECTORIES_TO_ENCRYPT
# Example (long command line options):
sett encrypt local --signer alice@example.com --recipient bob@example.com --dtr 42 --output . ./file_to_encrypt.txt ./directory_to_encrypt
# Example (short command line options):
sett encrypt local -s alice@example.com -r bob@example.com -dtr 42 -o . ./file_to_encrypt.txt ./directory_to_encrypt
Data can be encrypted for more than one recipient by repeating the flag
-r
/--recipient
, e.g. -r RECIPIENT1 -r RECIPIENT2
option:
# In this example, Alice encrypts data for both Bob and Chuck.
sett encrypt local -s alice@example.com -r bob@example.com -r chuck@example.com -o . FILES_OR_DIRECTORIES_TO_ENCRYPT
--output
is an optional argument for specifying the location and/or name
for the encrypted output file. The --output
argument can be one of the following:
- Not provided: the encrypted data is written to
stdout
. This can e.g. be useful to pipe the data into another application. - A directory: the encrypted data package file is written to the specified
directory, and is given a name that follows the default naming convention in
sett
. - A file name: the encrypted data is written to a new file with the specified name, and in the specified directory, if the file name contains one. This overrides the default output file naming schema.
local
subcommand can be replaced with s3
or sftp
to encrypt and transfer data
directly to a remote destination in a single command. When using s3
or sftp
,
the data is encrypted and streamed directly to the destination, without
creating any files in the local file system. The streaming approach is
faster than creating a data package locally and transferring it separately.
It also saves space on the local machine when transferring large datasets.
For more information about remote destination options, please refer to the remote destination options section.
sett encrypt s3 -s alice@example.com -r bob@example.com --endpoint https://minio.my-node.ch --bucket my-project \
--access-key 23VO8RB2SIB2SF8EUL9V --secret-key wvrt7YoTTERGftf0zWnppWYSdcGplNtxuLHMn7op --session-token eyJhbGciOiJ\
IUzUxMiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NLZXkiOiI5Vk84UkIyvimUMlKIFUVVTDc3WSIsImF0X7hhc2giOiIyRnVlZ3JmSjhTUWFXYkw2V0puek\
F3IiwiYXVkLjpbIm1pbmlvIl0sImF1dGhfdGltTRI6MTcyMTezODIxMywiZXhwIjoxNzIx0DMxODEzLCJpYXQiOjE3MjE4MzfiLKLsImlzwqI6Imh\
0dHBzOi8vcD3ydGFsLXN0YWasfmcuZGNjLnNpYi5zd2lzcy9hdXRoL38hdXRoIiwibmFtZSI6ImJpd2ciLCJqt5xpY5kiOiJjb25zb2xlQWRfgW4iLC\
JzdWIiOiIxOSJ9.PcvXcAli5Bz8ete1T265TPB1cbfgX7k8NDXU5gXy1nflxq203cG5qwAF9Oxyn1mKmwa87jsHj8HU2VUY9p5S1Q \
FILES_OR_DIRECTORIES_TO_ENCRYPT
Adding the --check
option will run the encrypt
command in the test mode, i.e.,
checks are made but no data is encrypted or sent.
Data compression algorithm can be changed using the --compression-algorithm
flag. The available options are:
zstandard
(default), optimal compression and speed.gzip
, available for compatibility with older versions ofsett
.stored
, no compression.
The data compression level used by sett can be manually adjusted using the
--compression-level
option.
Possible values depend on the selected compression algorithm:
zstandard
(default: 3)- 1 (lowest compression, fastest)
- 21 (highest compression, slowest).
gzip
(default: 6)- 1 (lowest compression, fastest)
- 9 (highest compression, slowest)
Before encrypting data using the local
subcommand and --output
flag,
sett verifies that there is enough free disk space
available on the local machine to save the encrypted output file. If this
is not the case an error message is displayed and the operation is aborted.
Since the compression ratio of the input data cannot be known in advance, sett
uses the conservative estimate that the minimum disk space required is equal to
the total size of all input files to be encrypted.
To automate the encryption process, you can use environment variables to store the OpenPGP password.
sett performs DTR verification if the --verify
option is passed. For
non-BioMedIT-related transfers, the --verify
option should not be passed.
BioMedIT
A valid DTR ID must be specified via the--dtr
option.
Decrypting files
Decrypt and decompress encrypted data packages.
Please note that the decryption process includes the verification of the sender’s signature, which ensures the authenticity of the data. Verification of the sender’s signature is only possible if the sender’s public key is available in your local keyring. For security reasons, the sender’s key will not be automatically downloaded from a key server; it must be downloaded/imported manually (for more information see download public keys).
To decrypt data, you must therefore have in your local keyring:
- The private key for which the data was encrypted. In principle, this is your own private key. This key will be used to decrypt the data.
- The data sender’s public key. This key is used for signature verification purposes.
Only files that follow the sett packaging specification can be decrypted with sett.
Decrypting data (GUI)
To decrypt and decompress a file:
-
Go to the Decrypt tab.
-
Select the source from which you wish to decrypt data - either local or s3.
-
When decrypting from local, select a data package to decrypt with Select Package, or drag and drop the file into sett. When decrypting from s3, a number of options must be specified - please refer to remote destination options for details. After doing so, click on Load package.
-
Select destination directory: select a location where to decrypt/decompress the file.
By default, output files are saved to the user’s home directory. This default behavior can be changed by changing Default output directory in the Settings tab.
-
Click Decrypt package to start the decryption and decompression process. A pop-up dialog box will appear to ask for the password associated with the OpenPGP key used to encrypt the files.
Decrypting data (CLI)
Use the decrypt
subcommand to decrypt and decompress data:
# General syntax:
sett decrypt local --output OUTPUT_DIRECTORY ENCRYPTED_FILES.zip
# Example:
sett decrypt local --output /home/alice/data/unpack_dir /home/alice/data/test_data.zip
local
subcommand can be replaced with s3
to decrypt data package from an
S3 object store. When using s3
, the data is streamed and decrypted directly
from the S3 object store, without creating any temporary files in the local
file system.
To decrypt data without decompressing it, add the -d, --decrypt-only
option.
If the -o, --output
option is omitted, the data is decrypted in the current
working directory.
To automate the decryption process, you can use environment variables to store the OpenPGP password.
Remote destination options
Both GUI and CLI require specific parameters when transferring data to a remote destination. The parameters are different depending on the destination type.
-
S3: A remote S3 compatible object store. When using this destination a number of options must be specified:
- URL: The URL of the S3 object store.
- Bucket: The name of the bucket where the encrypted data should be stored.
- Access key: The access key to use to authenticate with the S3 object store. It is also possible to use a username instead.
- Secret key: The secret key to use to authenticate with the S3 object store. It is also possible to use a password instead.
- Session token: The session token to use to authenticate with the S3 object store. It is optional and only required when authenticating using temporary credentials (STS), which is the case of most users interacting with the BioMedIT infrastructure.
BioMedIT
This information must be retrieved from BioMedIT Portal by a data provider data engineer. Please refer to the dedicated section of the Portal user guide for instructions on how to do so. Portal allows copying the credentials to the clipboard, so that they can be pasted into sett by using the Paste from clipboard button. -
SFTP: A remote SFTP server. When using this destination, a number of options must be specified:
-
Host: URL address of the server where the files should be sent.
-
User name: the user name with which to connect to the SFTP server.
-
Destination directory: absolute path of directory where files should be saved on the server.
-
SSH key location: path of the private SSH key used for authentication to the SFTP server. This is only required if the SSH key is in a non-standard location. If missing, sett will use the SSH agent to provide the key.
Do not confuse SSH keys - which are used to authenticate yourself when connecting to an SFTP server during file transfer - with OpenPGP keys - which are used to encrypt and sign data.
-
SSH key password: password associated with the private SSH key given under SSH key location. If your SSH key password contains characters that are not ASCII characters, and that this results in an error, please see the SSH private key with non-ASCII characters section of this guide.
BioMedIT
For BioMedIT users, the SFTP connection parameters User name, Host URL, and Destination directory will be provided by your local BioMedIT node. -
1.6 - Generating SSH keys
SSH (Secure SHell) keys are pairs of small text files that are used to securely identify users and give them access to remote servers, e.g. when transferring data via SFTP.
SSH keys use public-key encryption and always come in pairs: a public and a private (or secret) key.
- Public key: the public key of an SSH key pair is meant to be placed on the remote machine to which a user wants to connect. Public keys are non-sensitive, and are typically shared by users with system administrators, who will place them on the machine to which the user is granted access.
- Private key: the private key of an SSH key pair is what uniquely identifies a user as the legitimate owner of a public key. In other words, having the private key that matches a given public key will give access to any machine on which a copy of public key is stored. Private SSH keys are sensitive information: they must be kept private at all times and should never be shared with anyone - not even your system administrator. Private keys can (and should) be protected by a password, so that even if someone else has access to them, they remain unusable.
Generating a new pair of SSH keys must be done only once, and, in the context of sett, is only needed if you intend to transfer data. If you are a user who only decrypts data, you do not need an SSH key.
Also, do not confuse SSH keys - used to identify yourself on a remote server - with OpenPGP keys - used to encrypt and sign data.
To generate a new SSH key pair, type the command below in your terminal (Linux
and Mac) or PowerShell (Windows users - to start it, search for “powershell”
in the Start menu). Note that you must replace "alice@example.org"
with your
own email. This will generate an SSH key pair using the
ed25519 algorithm, currently the most
secure public-key algorithm:
ssh-keygen -a 100 -t ed25519 -C "alice@example.org"
Windows users who do not have the ssh-keygen command installed, please see section below.
When executing the ssh-keygen command above, you will be prompted for the following information:
- The name and location where to save the newly created keys. Here you should
simply accept the default values by not entering anything and pressing
“Enter” on your keyboard. Default locations are
~/.ssh/id_ed25519
on Linux and MacOS, andC:\Users\%username%\.ssh\id_ed25519
on Windows. - A password to protect your private SSH key against illegitimate use. Please use a password that is long enough (>=12 characters) and composed only of ASCII characters.
When the command completes, two new files are produced: id_ed25519.pub
(the
public key) and id_ed25519
(the private key).
Keep the SSH private key secret
Remember that the file containing the private key (id_ed25519
) must be kept
secret. Never share it with anyone, not even your system administrator.
On Linux and MacOS systems, after the public key is generated, its permissions must be changed with the following command (this step is not needed for Windows users):
chmod 600 ~/.ssh/id_ed25519.pub
Windows users: enabling the ssh-keygen command
Not all versions of windows come with the ssh-keygen command pre-installed. If this command is unavailable on your machine, please install it as follows:
- Open the windows settings (shortcut: windows key + i).
- Search for, and select, “Add an optional feature”.
- Click on “Add a feature”.
- Search for, and select, “Open SSH Client”.
- Click “Install”.
- Restart your computer.
SSH private key with non-ASCII characters password
Even though it is possible to create an SSH key pair using a password containing non-ASCII characters, it seems like those characters are encoded differently between different operating systems.
As an SSH key might be moved to a machine with another operating system, or encoding might change with a new version, it is impossible to guess the correct encoding in any case. For this reason, we recommend not to use non-ASCII characters to protect SSH private keys.
1.7 - Settings
GUI settings
The sett desktop app allows a number of options to be customized via its Settings ⚙️ page. For instance, you may change the default output directory, or enable/disable package verification before a transfer.
Each setting has a predefined default value, which is used when first running the tool or if loading the current settings fails for any reasons.
Changes made to Settings become effective immediately. Changes can be reset back to their factory default by clicking on the Reset settings button.
Settings are divided into three sections: “basic”, “advanced”, and “non-editable”.
Basic
- Verify package
-
When enabled (the default value), the following checks are made before encrypting or transferring data:
- DTR ID is valid and the transfer is authorized.
- Sender and Recipients public OpenPGP keys are approved by the BioMedIT key validation authority.
- Recipients are approved Data Managers of the BioMedIT project for which data is being encrypted.
- The name of the data package matches the pattern
<project_code>_<date_format>.zip
. This ensures no sensitive information is mistakenly included in the file name.
Note that that some of the above checks require communication with the BioMedIT portal. When using sett outside of a BioMedIT project, this setting should therefore be disabled.
- Default output directory
-
Default destination directory for operations such as encryption to the local filesystem or decryption. User’s home directory is used by default.
Advanced
- Enable extra metadata
-
When enabled, additional metadata can be added to the package in the data encryption form.
- OIDC issuer URL
-
URL of the OpenID Connect issuer used for authentication (BioMedIT specific).
- Portal URL
-
URL of a BioMedIT portal instance. Portal is used for key approval verification, DTR (Data Transfer Request) validation, and retrieval of data associated with a given DTR (when sett is being used in authenticated mode). The default value of this setting is: https://portal.dcc.sib.swiss.
- Public key store
-
Directory where public OpenPGP keys are stored.
- Private key store
-
Directory where private OpenPGP keys are stored.
Non-editable
This section displays values of setting that cannot be modified by the user. These settings are displayed here for convenience. They can be copied to the clipboard via a dedicated “copy to clipboard” button.
- Log directory
-
Directory where log files are stored. Location of the log directory depends on the operating system:
- Linux:
${XDG_DATA_HOME}/ch.biomedit.sett/log
or$HOME/.local/share/ch.biomedit.sett/log
- macOS:
$HOME/Library/Application Support/ch.biomedit.sett/log
- Windows:
{FOLDERID_RoamingAppData}\ch.biomedit.sett\log
- Keyserver URL
-
URL of the OpenPGP key server used to retrieve and publish public keys. The default value is https://keys.openpgp.org.
CLI settings
The sett-cli is stateless by design, meaning that there is no persistent
configuration file where settings can be modified and stored.
Instead, settings can be set via the following shell
environment variables. All settings are optional and have a default value.
SETT_OPENPGP_KEY_PWD
-
Password to unlock the secret OpenPGP key used to decrypt or sign data. When this environmental variable is set, sett uses its content instead of interactively asking the user to enter a password.
SETT_OPENPGP_KEY_PWD_FILE
-
Full path and name of a file containing the password to unlock the secret OpenPGP key used to decrypt or sign data. When this environmental variable is set, sett uses its content instead of interactively asking the user to enter a password. The file containing the password should not be encrypted.
SETT_PORTAL_URL
-
URL of the BioMedIT Portal instance to be used. For details see the description of the GUI Portal URL setting. This setting defaults to https://portal.dcc.sib.swiss.
SETT_OIDC_CLIENT_ID
-
Client ID with which sett should identify with the OpenID Connect issuer (see
SETT_OIDC_ISSUER_URL
). Only relevant when using sett in authenticated mode. This setting defaults tosett
. SETT_OIDC_ISSUER_URL
-
URL of the OpenID Connect issuer used when authenticating with the BioMedIT Portal. Only relevant when using sett in authenticated mode. This setting defaults to https://login.biomedit.ch/realms/biomedit.
SETT_KEYSTORE
-
Directory where private OpenPGP keys are stored. Location is platform dependent:
- Linux:
${XDG_DATA_HOME}/sequoia/keystore
or$HOME/.local/share/sequoia/keystore
- MacOS:
$HOME/Library/Application Support/org.Sequoia-PGP.sequoia/keystore
- Windows:
{FOLDERID_RoamingAppData}\org.Sequoia-PGP.sequoia\keystore
PGP_CERT_D
-
Directory where public OpenPGP keys are stored. Location is platform dependent:
- Linux:
${XDG_DATA_HOME}/pgp.cert.d
or$HOME/.local/share/pgp.cert.d
- MacOS:
$HOME/Library/Application Support/pgp.cert.d
- Windows:
{FOLDERID_RoamingAppData}\pgp.cert.d
SETT_METADATA_EXTRA
-
Additional metadata to be included in the data package. Extra fields must be provided in the form of key=value pairs separated by a comma and without spaces. For example:
SETT_METADATA_EXTRA="foo='value 1',bar=value_2"
1.8 - Data package specification
sett compresses, encrypts, and packages files in a single .zip
file whose
specification is described below. Only files adhering to these specifications
can be transferred or decrypted by sett, and failure to comply with the
specification will generate an error.
File structure
sett .zip
files have the following structure:
YYYYMMDDThhmmss.zip
├── metadata.json
├── metadata.json.sig
└── data.tar.gz.gpg
└── data.tar.gz
├── content/
| ├── [file1]
| ├── [file2]
| └── ...
└── checksum.sha256
metadata.json
-
Metadata file containing the following information:
- transfer_id: transfer ID.
- sender: fingerprint of the sender's public OpenPGP key.
- recipients: list of fingerprints of the recipients' public OpenPGP keys.
- timestamp: datetime of metadata generation. Uses the RFC 3339 format.
- checksum:
sha256
checksum of the encrypteddata.tar.gz.gpg
file. This allows verifying the integrity of the transferred file without decrypting it, and is used to make sure nothing was corrupted during the transfer process. - checksum_algorithm: algorithm used to compute the checksum.
Currently
sha256
. - compression_algorithm: algorithm used to compress the inner
tarball.
zstandard
is used by default. Other options aregzip
andstored
(no compression). - purpose:
PRODUCTION
,TEST
, ornull
(optional). - version: the version of the metadata specifications.
- extra: optional field containing additional information in the key-value format. Both keys and values are strings.
metadata.json.sig
-
Detached PGP signature for the
metadata.json
file. data.tar.gz.gpg
-
A tarball encrypted using the receiver's public PGP key and signed with the sender's private key. It is compressed with
compression_algorithm
. [file1]
,[file2]
-
One or several data files can be transferred. Data to be sent can be in any format, e.g.
.txt
,.csv
,.dat
. checksum.sha256
-
sha256 checksum file of all files present in
data.tar.gz
. This is used to make sure nothing was corrupted during the encryption/transfer/decryption process.
File example
Examples of the content and structure of the metadata and checksum files.
metadata.json
{
"transfer_id": 42,
"sender": "AAABFBC698539AB6CE60BDBE8220117C2F906548",
"recipients": ["D99AD936FC83C9BABDE7C33E1CF8C1A2076818C3"],
"timestamp": "2020-01-29T15:31:42+0100",
"checksum": "a8eb0ee5a6a53326b1c6f9bf94136da5d98a1dc6dceee21d62f694d71c4cf184",
"checksum_algorithm": "SHA256",
"compression_algorithm": "gzip",
"purpose": "PRODUCTION",
"version": "0.7",
"extra": {
"key1": "value1",
"key2": "value2"
}
}
checksum.sha256
41421f0c4a5353a5a0cdd37de3fd80a840a190ca997ad8044a67c4c1683f7b63 file1.csv
35ba157ed1c3269d731a438c466790a4f481bb49805e2d1f380df0c636792ff6 folder1/file.txt
fd9ebdbcc1a5fc35ded6e78a6b16ef658502c9d0b05dd4a2185d0f94ccf165cf folder1/folder2/file.txt
1.9 - Docker image
In addition to the desktop app and a standalone CLI tool, sett is also distributed as a Docker image, which can be used to run the tool in a containerized environment. This page provides instructions on how to use the sett Docker image.
All images are available in the GitLab container registry.
Pull an image with the following command:
docker pull registry.gitlab.com/biomedit/sett-rs/sett:5.2.0
# To simplify the subsequent commands, tag the image as `sett`
docker tag registry.gitlab.com/biomedit/sett-rs/sett:5.2.0 sett:5.2.0
Most sett commands require access to the public and private OpenPGP keys. Keys can be provided in two ways:
- As individual key files (see the
--signer-path
and--recipient-path
flags in theencrypt
anddecrypt
subcommands). - As key stores.
When using individual key files, mount the directory containing these files as a volume in the container (or mount individual files):
docker run -it --rm -v ./openpgp:/openpgp:ro -v ./data:/data \
sett:5.2.0 encrypt local -S /openpgp/alice.pgp -R /openpgp/bob.pgp -o /data /data/README.md
Key stores can be mounted in a similar way. However, they must be mounted at the specific location within the container.
docker run --rm \
-v ./pub_store:/root/.local/share/pgp.cert.d \
-v ./priv_store:/root/.local/share/sequoia/keystore \
sett:5.2.0 keys list
If you want to use custom locations for the key stores, you can specify them using environment variables. For more information on key store location see CLI settings.
docker run --rm \
-e PGP_CERT_D=/certificate_store \
-e SETT_KEYSTORE=/keystore \
-v ./pub_store:/pub_store \
-v ./priv_store:/priv_store \
sett:5.2.0 keys list
Encryption and decryption subcommand might need a password for unlocking the secret OpenPGP key. This can be provided as an environment variable (see CLI settings):
# Password as an environment variable
docker run -it --rm -v ./openpgp:/openpgp:ro -v .:/data \
-e SETT_OPENPGP_KEY_PWD="secret" \
sett:5.2.0 encrypt local -S /openpgp/alice.pgp -R /openpgp/bob.pgp -o /data /data/README.md
# Password from a file
docker run -it --rm -v ./openpgp:/openpgp:ro -v .:/data \
-e SETT_OPENPGP_KEY_PWD_FILE=/pgp_key_pwd \
-v ./password_file:/pgp_key_pwd:ro \
sett:5.2.0 encrypt local -S /openpgp/alice.pgp -R /openpgp/bob.pgp -o /data /data/README.md
1.10 - Frequently asked questions and bug reports
1. Proxy
1.1 How do I know I am using a proxy?
On Windows, you can check if you are using a proxy server to access the Internet by going to Start > Settings > Network & Internet > Proxy (Windows 10). If the slider under Use a proxy server is "off", no proxy is being used.
If you are told that you need to set a proxy, input the Address and Port details and click Save. When in doubt, please consult your IT department.
On Mac OS the proxy information is located under the System Preferences > Network > Advanced > Proxies tab of the network interface, usually Ethernet or Wi-Fi.
1.2 How do I run sett behind a proxy?
In order to run sett behind a proxy, the shell environment variable
ALL_PROXY
or HTTPS_PROXY
must be set. This is the recommended and global way
to specify a proxy. Note that, while certain programs support a proxy option
(e.g. pip
with --proxy
), there is currently no such option in sett.
Example:
ALL_PROXY=https://host.domain:port sett-gui
2. Logging
2.1 Does sett generate logs?
Yes, it does. sett logs are stored in the following locations:
- Linux:
${XDG_DATA_HOME}/ch.biomedit.sett/log
or$HOME/.local/share/ch.biomedit.sett/log
- macOS:
$HOME/Library/Application Support/ch.biomedit.sett/log
- Windows:
{FOLDERID_RoamingAppData}\ch.biomedit.sett\log
Log files are rotated daily and named according to the following format:
<date>.<app>.jsonl
. For example, 2024-08-29.sett-gui.jsonl
and
2024-08-29.sett.jsonl
for GUI and CLI applications respectively.
Each line within the log file is a JSON
object corresponding to a single
logging event.
Bug reports
To report a problem with sett or if you have a question not covered in the present documentation please open an issue on our public gitlab repo https://gitlab.com/biomedit/sett-rs/-/issues
1.11 - Source code
sett is licensed under the GPLv3 (GNU General Public License) and the source code is available at https://gitlab.com/biomedit/sett-rs
sett is developed as part of the BioMedIT project.
1.12 - Benchmarks
sett performance benchmarks
Test setup
The benchmarks were run on an Apple Silicon M3 machine with 16-Core CPUs, 128GB memory, and 2TB disk. Version 5.2.0 of sett CLI was used.
One single binary file was used, with default compression (Zstandard, level 3). Decreasing the compression level and/or using text data will influence the results.
Transfer speed is highly dependent on the infrastructure connecting the data sender and recipients. In these benchmarks, data was transferred to the same machine sending the data, therefore representing a best-case scenario in terms of transfer speed.
Results
The table and figure below present throughput speeds for different sett workflows, all values are averages of 5 runs performed with hyperfine:
- Encrypt: compress, encrypt package and transfer (S3, SFTP) data.
- Transfer: transfer already packaged data. There is no compression and encryption involved in this workflow.
- Decrypt: decrypt and decompress a data package, either locally or streamed from S3.
Workflow | Throughput (MB/second) | Throughput (GB/minute) |
---|---|---|
Encrypt local | 230.65 | 13.84 |
Encrypt S3 | 179.98 | 10.8 |
Encrypt SFTP | 119.79 | 7.19 |
Transfer S3 | 478.68 | 28.72 |
Transfer SFTP | 123.49 | 7.41 |
Decrypt local | 244.74 | 14.68 |
Decrypt s3 | 205.22 | 12.31 |
Throughput as function of file size: as can be seen, the curves in the figure below are linear, indicating that the throughput is constant regardless of the file size.
2 - BioMedIT Portal User Guide
In the following sections we introduce the BioMedIT portal, how it works and how
to get started using it.
Please use the table of contents menu to the left or below to navigate the topics.
What is the BioMedIT Portal?
The BioMedIT Portal is the one-stop entry to the BioMedIT Network. It provides users with a single point of access to the tools and services of the BioMedIT Network in a secure web-based environment.
With the Portal:
- Project Leads have a consolidated view of their projects and resources, enabling them to manage the research team’s members and access the project space.
- Data Providers and Data Managers oversee data transfers and monitor them.
- Researchers gain access to an expanding collection of collaborative and research-oriented tools, which encompass in-house, commercial, and open-source resources.
Access is secured with a login with your SWITCH edu-ID account and two-factor authentication.
Table of Contents
2.1 - Login
To either register for the first time or login to the BioMedIT portal:
- Go to: https://portal.dcc.sib.swiss/. The SWITCH edu-ID authentication button will appear.
- Click on > SWITCH edu-ID.
- Enter your email address and click on Continue. You will be asked to enter your password.
- Your second authentication method prompt will appear (SMS, OTP). Enter your Code and click on Login.
- You will be asked to:
- Enter a username of your choice.
- Choose which of the emails associated to your SWITCH edu-ID account to use for the BioMedIT Portal.
- Agree with the Portal being notified when your affiliation changes.
- Agree with the BioMedIT Portal Private Policy.
- The BioMedIT Portal Home will be displayed.
2.2 - Home
As soon as you log in with your SWITCH edu-ID, the home page displays the menu bar of the left side, the Quick Access Panel, with short-cuts to central services in the middle, and the BioMedIT Feed on the right, where you may find news and announcements from the BioMedIT Team.
The Portal classifies user accounts by role, and the menu options visible for each user will depend on the role assigned. A Data Provider, a Project Leader, or a visitor without a specific role will all see different default menu options
Quick Access links
2.3 - Profile
From the profile menu option, users can check and update their account details, and manage their OpenPGP Keys and SSH Keys.
My Profile tab
The My Profile tab shows the user’s account details.
Account details:
-
Name: User’s name
-
Username: User SWITCH edu-id identifier
-
Local Username: The username the user chose during their first login to Portal.
-
Email: Contact email currently selected from the list retrieved from the user’s SWITCH edu-ID
-
Affiliations: List of organizations (e.g., university, research institute) linked to users SWITCH edu-ID account
-
IP Address: The IP address from where the user is connecting
-
Flags: List of flags enabled for the user
-
Groups: The list of Groups the user is a member of.
Note that personal details are retrieved from the user’s SWITCH edu-ID. For any changes to their personal data or affiliated organizations, user may update their SWITCH edu-ID via this link.
For more resources about SWITCH edu-ID, refer to SWITCH edu-ID / Quick Links.
OpenPGP Keys
From the OpenPGP Keys tab, users can see the PGP Keys they have registered in the Portal, along with their status.
Each key has the following information:
-
Fingerprint: This is a unique identified for each PGP key. You should make sure sure that this value is matching with the fingerprint of your key.
-
User ID: User ID associated with the PGP key. People usually have their full name (and sometimes affiliation) in this field.
-
Email: Email address associated with the PGP key
-
Status: The approval status of your key. The list of key statuses is as follows:
PENDING
: a key approval request was submitted, but the key has not been approved yet. This is a manual process and can take from a few hours or up couple of days.APPROVED
: key is approved for usage within the BioMedIT network. Only approved keys can be used to encrypt, sign and decrypt data packages that are transiting via the BioMedIT network.APPROVAL-REVOKED
: approval of the key has been revoked by the BioMedIT key validation authority.KEY-REVOKED
: key has been revoked by its owner.REJECTED
: key is not trusted by the BioMedIT key validation authority.DELETED
: key has been removed from the keyserver by its owner.UNKNOWN KEY
: key has not been registered on the BioMedIT portal. If it is your own key, please register it. If it is the key of someone else, please ask them to register their key.
Registering a new PGP key
Important
- The key has been created, uploaded to the Open PGP keyserver, and verified with the keyserver (i.e. the keyserver will send you an automated email to verify that you have control over the email associated with the PGP key). The standard way to generate a key pair in BioMedIT is via the Secure Encryption and Transfer Tool - sett GUI application, but PGP keys generated using other methods are also accepted (e.g. using the GnuPG command line tools).
- The PGP key is using one of the email addresses associated to your SWITCH edu-ID.
- You do not already have an APPROVED key registered on the Portal. Only one key per user can be used for data encryption/decryption at any one time. If you wish to register a new key and already have an approved key, please contact the DCC to request the revocation of your current key (if you still intend to use that key outside of BioMedIT) or revoke it yourself on the keyserver (if you want to permanently revoke your key for all applications).
Once the above prerequisites are met, perform the following steps to register your key and request its approval:
-
Connect to the BioMedIT portal and go to Profile and OpenPGP Keys tab.
-
Click on +OPENPGP KEY.
-
Add Key dialogue box will open. Enter your key’s fingerprint: please copy-paste it from your computer to avoid typing errors. Then click on the green search icon.
-
If your key is present on the Open PGP keyserver, the user ID and email associated with your key will be retrieved and displayed in the dialog box. Please double check that the value is correct.
-
If the displayed user ID and email values are correct, click on CONFIRM.
Your key is now registered and an approval request as been automatically sent to the DCC. Please note that the verification and approval of your key by the DCC is a manual process and may take a few days.
Note that your key’s approval status will be PENDING
until approved by the
DCC. Once approved, the status will change to APPROVED
, at which point the key
can be used to encrypt and decrypt data to be transferred into the BioMedIT
network.
SSH Keys
From the SSH Keys tab, users can add their public SSH key for a specific project.
Registering a new SSH key
- Click on +SSH KEY.
- Select the project from the drop-down list
- Copy the content of your public ssh key
- click on SAVE
2.4 - Groups
From the Groups menu option, Data Providers can manage users and assign them roles.
Data Provider User roles and responsibilities
The Data Provider roles are intended to provide Data Providing institutions with the ability to manage the users involved in data transfers to BioMedIT and assign permission levels according to their responsibilities.
DP Manager
The DP Manager represents the Data Provider institution within the scope of BioMedIT. A DP Manager is responsible for the institution’s internal processes related to BioMedIT and delegates responsibilities to other users in their organization by assigning/removing DP roles. In addition, a DP Manager is responsible for maintaining and keeping the roles assigned up to date, i.e., if an employee left or changed responsibilities, and informing the BioMedIT Node and the DCC accordingly. Finally, the DP Manager delegates the internal operation coordination to the DP Coordinator.
DP Managers are:
- Responsible for the ensuring data provisioning at the Data Provider institution.
- Responsible for escalations and security incidents within the Data Provider.
- Responsible to assign/revoke the DP roles to other DP users.
DP Coordinator
The DP Coordinator is responsible for internally coordinating the DP’s readiness to send data, ensuring the legal compliance, i.e., DTUA, DTPA, etc. and all technical measures required from the DP side are in place for projects in which Data Provider institution participates. The DP Coordinator must send confirmation of readiness for data transfers to the DCC when required, and act as the single point of contact for follow-up questions and/or issues in the process.
DP Coordinators are:
- Responsible for coordinating internal data transfer processes of the Data Provider institution
- Responsible for confirmation of legal readiness for data transfers
- Responsible for confirmation of technical readiness for data transfers
DP Technical Admin
The DP Technical Admin is responsible for all technical tasks required for the setup and maintenance of a secure connection from the DP organization to the BioMedIT Node e.g., network components configuration. He is also responsible to support DP Data Engineer in the onboarding process. There could be more than one DP Technical Admin.
DP Technical Admins are:
- Responsible for the configuration of the network components from the DP side to permit data transfers.
- Responsible for supporting the onboarding of the DP Data Engineer(s).
DP Data Engineer
The DP Data Engineer is responsible to prepare, encrypt and sign the data packages, and transfer them from the DP organization to the assigned BioMedIT Node. There could be more than one DP Data Engineer.
DP Data Engineers are:
- Responsible for preparing and executing the data package transfers between the Data Provider and the assigned BioMedIT Node, following the BioMedIT process for secure data transfer.
DP Security Officer
The DP Security Officer is the designated point of contact from the DP organization to receive incident and security notifications from BioMedIT such as (but not limited to) scheduled and non-scheduled downtimes in response to security incidents, emergency changes and systems upgrades.
DP Security Officers are:
- Responsible for serving as a point of contact for incident, maintenance and security notifications from BioMedIT, and distributing them internally.
DP Viewer
A DP Viewer can monitor the status of data transfers from their DP institution.
How to add a user to a Data Provider group
Note
- Only users with the role of DP Manager can add or remove users from their organization.
- The DP Manager role can only be assigned by the DCC (send an email to biomedit@sib.swiss for updates).
- Users must have have a BioMedIT portal account.
To add a user to a group:
- Click on the edit button of the group.
- In the Users field, type the user’s email address and click on +USER.
- When no more users need to be added to the group, click on SAVE.
2.5 - Projects
In the Projects menu option:
- Project users find a consolidated view of their projects, the data transfers, and the tools and resources enabled for the project.
- Project Leaders and Permission Managers can manage project users.
- Data Managers of a project can submit a data transfer request.
Projects view
From the project’s menu option, users can see the list of their projects.
The projects displayed can be filtered by using the Search function.
By clicking over a project, users can display the project’s details:
- Project Details tab
- Data Transfers tab
- Resources tab
- Services tab
- Users tab
- Additional information tab
Project Details tab
The details tab shows the project’s general settings:
-
Code: A unique identifier for the project
-
Hosting Node: Name and code of the node where the project is hosted
-
Archived: Yes / No
-
Expiration Date: Date when the project expires.
-
Legal Approval Group: Group responsible to confirm compliance with legal basis and approve production DTRs.
-
Data Specification Approval Group: Group responsible to review data transfer’s Data Specifications and approve production DTRs.
-
Legal Support Contact: Contact email address to engage support about ethical and legal questions.
-
Enable SSH Access: Indicates if SSH access has been enabled for the project.
-
Enable Resources: Indicates if resources have been enabled for the project. The resources tab will be shown/hidden accordingly.
-
Enable Services: Indicates if services have been enabled for the project. The services tab will be shown/hidden accordingly.
-
Identity Provider: Source user identity provider specified for the project. By default is SWITCH edu-iD, but a node-specific pre-defined identity providers may be selected.
-
IP Ranges: IP range(s) from where client connections to access B-spaces is permitted.
If the user is connecting from an IP outside of the IP range, the message “Your IP address is not within the valid IP address ranges of the project” is displayed.
Data Transfers tab
The data transfers tab displays the list of data transfer requests submitted by the project.
-
From the Search box, users can search for specific data transfer requests by any content as search criteria.
-
+DATA TRANSFER: Function to submit a new Data Transfer Request. Note that this option is only visible to the project’s Data Manager(s). The data transfer request list is also accessible from the Data Transfers menu option.
Users tab
Displays the list of users and their role in the project.
Authorized flag:
This badge is displayed next to a project user when they have completed the mandatory BioMedIT training modules: Information Security Awareness Training
Additional information tab
Notes, comments and relevant information details. Note that this field is only editable by admins (Node Managers and BioMedIT Support).
Resources tab
Displays the access links to the node-specific tools and applications configured for the project. It offers these fields:
- Name
- Location - a valid URL that points to the resource
- Contact - a valid email address whom to contact
- Description - a simple textfield
These resources are defined per project only. Because of this limitation, we added the services tab (see below) which offer more flexibility.
Services tab
Similar to the resources tab but more flexible. Displays the node-specific tools
assigned to a given project as well as the tools assigned to individual members
of that project. The aim is that Portal serves as a central point for node-specific
information. The data
field (see example below) allows the use of markdown
language which will be automatically nicely rendered as rich text in Portal.
To create a new entry, node admins simply need to issue a POST request:
POST /backend/projects/<project_pk>/services/
{
"name": "Special Tools",
"description": "A collection of special tools for a project",
"data": "# Title\n* feat 1 explained\n* feat 2 explained", # <- Markdown language
"state": "CREATED" # valid states: INITIAL, PROGRESS, CREATED, LOCKED, ARCHIVED
}
Where the project_pk
is the primary key of a given project (not the project code).
Node admins and -viewers will then see the service(s) nicely rendered in the
services tab.
To view, update or delete a project service entry, use the
/backend/projects/<project_pk>/services/<service_pk>/
endpoint and the GET
, PUT
or DELETE
http verbs. service_pk
is the automatically
assigned primary key of a service.
In addition, services can also be assigned to project members, by sending a POST request to this endpoint:
POST /backend/projects/<project_pk>/members/<user_pk>/services
{
"name": "Special Tools",
"description": "A collection of special tools for a project",
"data": "# Title\n* feat 1 explained\n* feat 2 explained",
"state": "CREATED" # valid states: INITIAL, PROGRESS, CREATED, LOCKED, ARCHIVED
}
Where the project_pk
is the primary key of a given project (not the project code)
and user_pk
the primary key of the user. If the user is not member of the specified
project, you’ll receive a 404 error.
Once a project service is assigned to a project member, users then are able to see the current state of their assigned project services in the services tab.
To view, update or delete a project service entry, use the
/backend/projects/<project_pk>/members/<user_pk>/services/<service_pk>/
endpoint and the GET
, PUT
or DELETE
http verbs. service_pk
is the automatically
assigned primary key of a service.
Note 1: these services serve information purposes only, have no implicit functionality. In particular, they do not trigger any mails being sent to the user.
Note 2: the service name must be unique and a service can only be assigned once to a project or project member. The service name can be freely chosen.
Note 3: currently the action
field has no meaning or implicit functionality.
Note 4: there is no GUI planned to edit service information data.
User roles
User roles are set for each project. Each authorized BioMedIT user can have several user roles and can be assigned access to several projects.
Roles and responsibilities
Project Leaders (PL):
- responsible and accountable for the project
- responsible for the data lifecycle management of the project
- acts as contact point for escalations and security incidents within the project
- responsible to discuss the setup and configuration of the project space for the project with representatives of the BioMedIT Nodes
- responsible for case-by-case authorization of Data Transfers out of the B-space
- assigned to the project in the User Role of “Permissions Manager” by default and with that is responsible to assign authorized BioMedIT users
- responsible to assign the User Role of “Permissions Manager” to authorized BioMedIT users
- responsible to ensure that at least one authorized BioMedIT user was assigned to the project space in the User Role of “Data Manager”
- can act as “Permissions Manager” and/or “Data Manager” and/or “(default) User”
- responsible to revoke access rights to the project space
- informed about every user change (add/remove) via email notification
- there can be multiple PLs per project
Permissions Manager (PM):
- accountable to the PL
- each project must have at least one project member with the User Role of “Permissions Manager”
- responsible for assigning the User Roles of “(default) User” and “Data Manager” to authorized BioMedIT users and with that granting them access to the project space
- responsible to revoke access rights to the project space
- acts as contact for the new project members to support them with registration and authorization as users of BioMedIT
Data Manager (DM):
- accountable to the PL
- each project must have at least one project member with the User Role “Data Manager”
- responsible for unpacking and decryption of data packages within the B-space
- responsible to extract data from the B-space for transfer out of the project space only following explicit authorization of the PL
- responsible to place the request(s) for Data Transfer(s) from a Data Provider to the project
User:
- accountable to the PL
- has access to one or several project spaces following authorization
- responsible for data processing within the B-space
No Role:
- accountable to the PL
- can access the project’s details in the Portal, but has no access to the B-space.
- mutually exclusive with other roles
Managing project users
Note
This option is only visible to users with the Project Leader and/or Permissions Managers role.When clicking on the Edit project button the top-right corner, you will be taken to edit page.
Adding users to a project
- In the Users field, enter the user’s email address and click on +USER. Note that the user(s) must have an account in the portal.
- You will be asked to confirm the user’s details before adding it to the project.
- Review the details of the user, and click on ADD to confirm it.
- You will return to the list of project users. Assign a role to the user by clicking on the corresponding checkbox(es).
- Click on SAVE.
Removing users from a project
To remove users from the project, the Project Leader or the Permissions Manager should:
- Click on the 'x' next to the user’s name.
- After any change, click on SAVE
Submitting a Data Transfer Request
Important
- This option is only available for users with the Data Manager role
- Prior to submitting a DTR, SPHN projects are required to prepare the project’s RDF Schema, external terminologies, and de-identification rules to ensure compliance with the SPHN Semantic Interoperability Framework. For more information, please consult the process documentation.
To submit a Data Transfer Request (DTR), click on +Data transfer. The data transfer creation form will be displayed.
Complete the required fields:
-
Unlimited / One Package: Select if the DTR will cover one or multiple data transfers
-
Data Provider: Select the Data Provider from the drop-down list
-
TEST / PRODUCTION: Select TEST if mock data will be sent, or PRODUCTION if the data will contain patient real data
-
Additional Data Recipients: Enter the email(s) of the data recipient(s) and click on the +USER button.
Note that recipients can only be users with the Data Manager role in the project. -
Legal Basis: If transferring real data (purpose PRODUCTION), enter the legal agreement (i.e. DTUA, DTPA) document name
-
Data Specification: If transferring real data (purpose PRODUCTION), include the link to the Data Specification
Click on SAVE.
A new DTR will be created in INITIAL
status. Confirmation emails with the
DTR’s details are sent to the DTR requester, recipient(s), and approvers.
Confirmation email example: Subject: BioMed-IT Portal: [For information] DTR-999 - Data Transfer Request from ‘Universität ABC’ to ‘SIB Project’ project Dear Data Manager (Patricia Fernandez), Your Data Transfer Request (DTR-999) has been successfully submitted. DTR details:
You will be informed about the status of the approval process. If you have any questions or need support, please contact biomedit@sib.swiss. Kind Regards, BioMedIT Team |
DTR Approvals
When the DTR is created, the following approval requests are submitted in parallel:
1. Node(s) Approval: As data processors, the involved node(s) must confirm the presence of a pre-existing legal basis for the data transfer and ensure that the necessary technical infrastructure is in place to support it.
2. Legal Compliance Approval: For SPHN/NDS projects, facilitated by the DCC ELSI Help Desk group who verifies the existence of a legal basis (e.g., DTUA, Consortium agreement) for the data transfer. For non-SPHN projects, a dedicated local Legal Compliance group may be optionally configured.
3. Data Specification Approval: For SPHN projects, the DCC will review and approve the data specification documents referenced in the ‘Data Specification’ link. This process ensures the compliance with the SPHN Semantic Interoperability Framework.
Note
The Legal Compliance Approval (2) and the Data Specification Approval (3) are exclusively required in transfers of real data. In any other case (test data), they are excluded from the approval workflow.
Email example: Subject: BioMed-IT Portal: [Action needed] DTR-999 - Data Transfer Request from ‘Universitätsspital ABC’ to ‘SIB’ project Dear <BioMedIT Node> , <Legal Compliance Group> , <Data Specification Group> , You are kindly requested to review and approve or reject the Data Transfer Request associated with the SIB project by using the following Data Transfer Request Approval form: https://portal.dcc.sib.swiss/data-transfers/999 Below are the details of the Data Transfer Request (DTR-ID):
If you have any questions or need support, please contact biomedit@sib.swiss Kind Regards, BioMedIT Team |
4. Data Provider Coordinator Approval: This approval request is triggered only when the previous approvers, (1),(2) and (3) have submitted theirs.
The Data Provider Coordinator is then requested to confirm if the data delivery has been approved through their internal governance processes.
When all approvers approve the request, the DTR status changes to AUTHORIZED
,
and data can then be transferred. Notification emails are sent to all parties
involved in the data transfer:
Subject: BioMed-IT Portal: [For information] DTR-999 - Data Transfer Request from ‘SIB Data Provider’ to ‘SIB Project’ project is approved Dear All, Clearance has been granted to transfer data in accordance with DTR 999 for project SIB Project (sib_project). The DP Data Engineer from SIB Data Provider is now authorized to encrypt, sign, and transfer the data package(s) using the sett tool according with the agreed procedures to the SIS node. Detailed instructions on how to use sett can be found at https://biomedit.gitlab.io/docs/sett/. The transferred data will be in accordance with the specification outlined here: https://git.dcc.sib.swiss/admin/projects/project-space/sib-project/data-transfer/-/tree/main/data-transfer-1?ref_type=heads. Upon completion of the data package transfer, please inform the Project’s Data Manager(s), Patricia Fernandez (patricia.fernandezpinilla@sib.swiss), so that they can confirm the reception, integrity, and successful decryption of the data package in the B-space. For any further questions, please don’t hesitate to contact biomedit@sib.swiss. Kind Regards BioMedIT Team Approval log:
|
If any of them rejects the DTR, its status is set to UNAUTHORIZED
.
Monitoring the approval status of a Data Transfer Request
The status of a DTR is displayed in the last column of the DTR list.
Approval Status
INITIAL
: The DTR was submitted, but it has not yet been approved.AUTHORIZED
: All approvers have authorized the DTR. The Data Provider can now send the data.EXPIRED
: The Data Provider sent the maximum allowed number of data packages, and no additional data can be sent under this DTR IDUNAUTHORIZED
: The DTR was previously authorized but is currently unauthorized for some reason (i.e., The BioMedIT Node is offline, problems with user permissions, issues on the Data Provider's end, etc.)|
Data Transfer details
When clicking on a particular data transfer, a pop-up window shows additional information
Details tab
Displays the attribute values of the DTR as submitted by the project’s data manager.
Logs tab
Displays the individual data packages transfer logs - routing information and whether the data has arrived to the project workspace.
Approvals tab
Displays the current approval status of a DTR by all approval groups.
2.6 - Data Transfers
In the Data Transfers menu option:
- Data Managers and Data Providers can search and view the list of submitted Data Transfers Requests (DTRs), monitor their authorization status and transfer logs
- Data Providers (DP Data Engineers) can fetch their credentials to transfer data to a research project.
- Data Provider Coordinators can approve or reject a DTR
DTR Approval status
The approval status of a DTR is displayed in the last column of the table:
INITIAL
: The DTR was submitted, but it has not yet been approved by the Data Provider and/or the BioMedIT node(s).AUTHORIZED
: All approval groups have authorized the DTR. The Data Provider can now send the data.EXPIRED
: The Data Provider sent the maximum allowed number of data packages, and no additional data can be sent under this DTR IDUNAUTHORIZED
: The DTR was previously authorized but is currently unauthorized for some reason (i.e., The BioMedIT Node is offline, problems with user permissions, issues on the Data Provider's end, etc.)
Additional details about which group approved when are shown when clicking on a specific DTR. See approval details
Data Transfer details
When clicking on a particular DTR, a pop-up window shows additional information
DTR’s details are accessible using the following link structure: https://portal.dcc.sib.swiss/data-transfers/dtr_id
Details tab
Displays the attribute values of the DTR, as submitted by the project’s data manager:
- Project: Name of the project
- Transfer ID: Unique DTR identifier
- Data Provider: the Data Provider code
- Status: Approval status of the DTR
- Max Number of packages: 1 (single transfer) or UNLIMITED (multiple data transfers)
- Transferred Packages: Count of packages sent so far under this DTR ID
- Requestor: Name of the Data Manager that submitted the DTR
- Data Specification: link to the Data Specification
- Purpose: TEST (for mock data) / PRODUCTION (for patient real data)
- Legal Basis: Legal agreement (i.e. DTUA, DTPA) document name
Approvals tab
Displays the current approval status of a DTR.
Credentials tab
From this option, DP Data Engineers can fetch their credentials, required for authentication and authorization to transfer data when the transfer method is S3/HTTPs.
Note
Notice that this tab is only visible by users with the DP Data Engineer roleTo fetch and copy their credentials, as DP Data Engineer:
- click on FETCH CREDENTIALS
- click on COPY ALL
Logs tab
Displays the individual data packages transfer logs - routing information and whether the data has arrived to the project workspace.
Approving DTRs
When the DTR is submitted by a project’s Data Manager, the groups involved in the data transfer will receive a notification by email with all the details, for their information or for their action (request for approval), depending on the case, as explained below.
Example: Dear Universitätsspital ABC, We have received the following Data Transfer Request (DTR-999) from the sib_project Project:
Please note that Data Provider approval will only be available upon completion of approvals from the Nodes, DCC (Data Specification), ELSI Help Desk. If you have any questions or need support, please contact biomedit@sib.swiss Kind Regards, BioMedIT Team |
Approvals
When a DTR is submitted by a Data Manager to a research project, the following approvals must be collected:
1. Node(s) Approval: As data processors, the involved node(s) must confirm the presence of a pre-existing legal basis for the data transfer and ensure that the necessary technical infrastructure is in place to support it.
2. Legal Compliance Approval: For SPHN/NDS projects, facilitated by the DCC ELSI Help Desk group who verifies the existence of a legal basis (e.g., DTUA, Consortium agreement) for the data transfer. For non-SPHN projects, a dedicated local Legal Compliance group may be optionally configured.
3. DCC Data Specification Approval: For SPHN projects, the DCC will review and approve the data specification documents referenced in the ‘Data Specification’ link. This process ensures the compliance with the SPHN Semantic Interoperability Framework.
Note
- The approval requests above, (1), (2) and (3), are submitted in parallel
- The Legal Compliance Approval (2) and the Data Specification Approval (3) are exclusively required in transfers of real data. In any other case (test data), they are excluded from the approval workflow
Only when the above groups, (1), (2) and (3) confirm their approval, the DP
Coordinator is requested to submit theirs:
Example: To: DP Coordinator Dear Universitätsspital ABC , You are kindly requested to review and approve or reject the Data Transfer Request associated with the SIB project by using the following Data Transfer Request Approval form: https://portal.dcc.sib.swiss/data-transfers/999 Below are the details of the Data Transfer Request (DTR-ID):
If you have any questions or need support, please contact biomedit@sib.swiss Kind Regards, BioMedIT Team |
By clicking on the link in the email above (https://portal.dcc.sib.swiss/data-transfers/<DTR_ID>), DP Coordinators will be redirected to the BioMedIT Portal to approve or rejects the request.
To approve the DTR, click on APPROVE and declare, by clicking on the checkboxes, if:
- All technical measures are in place to send the data
- There is an existing legal basis for the Data Transfer
- This data delivery has been approved through internal governance processes
Then, click on APPROVE.
To reject the DTR, click on REJECT and provide a reason for the rejecting. Then, click on REJECT.
When all approvers approve the request, the DTR status changes to AUTHORIZED
,
and data can then be transferred. Notification emails are sent to all parties
involved in the data transfer:
Subject: BioMed-IT Portal: [For information] DTR-999 - Data Transfer Request from ‘SIB Data Provider’ to ‘SIB Project’ project is approved Dear All, Clearance has been granted to transfer data in accordance with DTR 999 for project SIB Project (sib_project). The DP Data Engineer from SIB Data Provider is now authorized to encrypt, sign, and transfer the data package(s) using the sett tool according with the agreed procedures to the SIS node. Detailed instructions on how to use sett can be found at https://biomedit.gitlab.io/docs/sett/. The transferred data will be in accordance with the specification outlined here: https://git.dcc.sib.swiss/admin/projects/project-space/sib-project/data-transfer/-/tree/main/data-transfer-1?ref_type=heads. Upon completion of the data package transfer, please inform the Project’s Data Manager(s), Patricia Fernandez (patricia.fernandezpinilla@sib.swiss), so that they can confirm the reception, integrity, and successful decryption of the data package in the B-space. For any further questions, please don’t hesitate to contact biomedit@sib.swiss. Kind Regards BioMedIT Team Approval log:
|
If any of them rejects the DTR, its status is set to UNAUTHORIZED
.
Download data transfer list
By clicking on the download icon at the top-right corner, users can download the list of DTRs in csv format.
2.7 - Contact Us
It is the contact form to submit feature requests or support to the BioMedIT Help Desk (biomedit@sib.swiss).
2.8 - Central Services
2.8.1 - DCC GIT
The BioMedIT Code Repository Service (based on Git), is accessible to registered BioMedIT users to create, collaborate and share their application codes with other BioMedIT users.
1. Login to GitLab
1.1 GitLab is available via federated login from BioMedIT portal’s Quick Access link:
1.2 The following page will be displayed.
Click on Federated Login
2. Resources
2.8.2 - Harbor
Harbor is an open source registry that secures artifacts with policies and role-based access control, ensures images are scanned and free from vulnerabilities, and signs images as trusted. Harbor, a CNCF Graduated project, delivers compliance, performance, and interoperability to help you consistently and securely manage artifacts across cloud native compute platforms like Kubernetes and Docker.
1. How to request access
1.1 Registry is available via federated login from the BioMedIT portal’s Quick Access links:
1.2 Click on the button: Login via OIDC provider
2. Resources
3 - Quick links for SWITCH edu-ID
What is a SWITCH edu-ID?
SWITCH edu-ID is your persistent identity and gives you access to all federated services.
The account is easy to use, controlled by the user and provides secure access to academic services.
The service is provided by SWITCH for Swiss universities and their partners.
- As a student, you can use your account during and even after your studies.
- As an employee, you can use your account during and even after your employment at a university.
- As a private individual, you can use your account for your entire lifetime (e.g. to access library services).
Important
Do not create duplicate accounts!
If you create duplicates, you are in violation of the Terms of Use and risk your accounts being deleted or merged. You also risk losing account data. Account merging may have consequences:
-
You will have to verify the contact information again.
-
Account merging data is sent to services and IdM administrators to solve potential problems.
-
You will no longer be able to access some services. If you accidentally created a duplicate, follow the instructions here:
Quick Links
4 - SPHN-DCC Confluence
SPHN-DCC Confluence is a collaboration space and repository hosted in Confluence Cloud, for the working and implementation groups, projects and iniatives.
How to request access
To access SPHN-DCC Confluence, users must create an Atlassian account. Here are steps:
Access the link https://sphn-dcc.atlassian.net. The following page will display
Select the option Create an account
The following page will display.
Enter your email address and click on Sign up
Confluence will send you an email with a verification link:
Check your mailbox and open the email sent by Confluence. Open the message and click on Verify your email
After verifying your email, Confluence will redirect you to the following screen to finish setting up your account:
Enter your full name
Create a password
Click on Continue
The SPHN-DCC welcome space should display.
Note that to gain access to specific spaces, you must contact the space owner to assign you permissions.