User Guides

• 1: sett - Secure Encryption and Transfer Tool
• 1.1: What is sett?
• 1.2: Quick start guide
• 1.3: Download sett
• 1.4: PGP key management with sett
• 1.5: Generating SSH keys
• 1.6: Encrypting, transferring and decrypting data with sett
• 1.7: Settings for GUI and CLI
• 1.8: sett packaging file format specifications
• 1.9: Frequently asked questions and bug reports
• 1.10: Source code
• 1.11: Benchmarks
• 2: BioMedIT Portal User Guide
• 2.1: Login
• 2.2: Home
• 2.3: Profile
• 2.4: Groups
• 2.5: Projects
• 2.6: Data Transfers
• 2.7: Contact Us
• 2.8: Central Services
• 2.8.1: DCC GIT
• 2.8.2: Harbor
• 2.8.3: Federated Query System
• 3: BioMedIT Training
• 4: FAQs
• 5: Quick links for SWITCH edu-ID
• 6: SPHN-DCC Confluence

1 - sett - Secure Encryption and Transfer Tool

Welcome to the official sett documentation page.

Please use the table of contents menu to the left or below to navigate the topics.

Table of Contents

1.1 - What is sett?

sett stands for "Secure Encryption and Transfer Tool" and is an application that facilitates and automates data packaging, encryption, and transfer. It is written in Rust, a fast and memory-safe programming language. sett is available both as a desktop and a command line application.

sett is developed as part of the BioMedIT project. It is licensed under the GPLv3 (GNU General Public License) and the source code is available from its public GitLab repository.

1.2 - Quick start guide

sett-gui quick start

Initial setup

1. Download sett-gui from the download page. If you downloaded an installer, install sett-gui by double-clicking on the installer file.
2. Run sett-gui by double-clicking on the executable file or by launching the installed app.

Key management

1. If you do not already have a private/public PGP key pair, go to the Keys tab and create one clicking on Add > Generate new key pair. See also the instructions given in the Generate a new public/private PGP key pair section.

   You should then see your new key listed in the Keys tab, along with “Private” label that indicates that the private material for this key is present in the local keystore.

   BioMedIT

   Your PGP key must be registered with the BioMedIT portal before it can be used to encrypt, decrypt, or sign data packages. Please see the Register your public PGP key in the BioMedIT portal section for details.

2. If not already done, download the public PGP key of the recipient(s) to whom you intend to send data (or from whom you will receive data). In sett-gui, go to the Keys tab and click on Add > Import from keyserver. See also the instructions given in the download public PGP keys from the keyserver section.

3. Just after downloading the recipient’s PGP key, verify it to make sure that it is genuine. This can be done by either:

   • If you are a BioMedIT user: in sett-gui, verify that the recipient’s key is labelled with a green Approved label. You can also expand the details of the key by clicking on the key in the list or on the small ^ button to the right and verify that the Approval status is set to “Approved”, and the Revocation status is set to “Valid”.
   • Alternatively, contact the key owner and verify the key fingerprint with them.

Encrypting data

1. Go to the Encrypt tab of the sett-gui interface.

2. Add one or more files and directories to encrypt by clicking the Add files or Add directories buttons.

3. Select sender: select your own PGP key. This is the key that will be used to sign the encrypted data.

4. Select recipients: add one or more recipients by selecting them in the drop-down. These are the keys that will be used to encrypt the data, i.e. only these recipients will be able to decrypt the data.

   BioMedIT

   The selected recipients must be approved Data Managers of the project for which data is being encrypted.

5. Transfer ID: specifying a valid Data Transfer Request ID is mandatory when a data package is transferred into the BioMedIT network. For other destinations, the Transfer ID field can be left empty (or set to any arbitrary value), and the Verify package checkbox must be disabled (in the Settings tab).

   BioMedIT

   The Verify package checkbox should always be enabled, since a valid Transfer ID is required by the BioMedIT network.

6. Select destination: select local to encrypt to your local file system, and choose a destination directory.

7. Click Encrypt package - or Send package, if you chose s3 or sftp as your destination - to run the encryption workflow on your data.

Transferring data

1. Go to the Transfer tab of the sett-gui interface.

2. Select a file to transfer using the Select file button.

3. Select the Destination to be used (sftp, s3).

4. Enter the required destination parameters.

   BioMedIT

   For transfers into the BioMedIT network, the destination parameters are provided by:

   • Your local BioMedIT node, for transfers over sftp;
   • By BioMedIT Portal, for transfers over s3. Please see the dedicated section of the Portal user guide for more information.

5. Click Send package to start transferring your files.

Decrypting data

1. Go to the Decrypt tab of the sett-gui interface.
2. Select a file to decrypt using the Select file button.
3. Specify your desired destination directory.
4. Click on Decrypt package.

sett command line quick start

The main commands to manage keys, encrypt, transfer and decrypt data with sett command line interface are given here.

# Generate a new key pair:
sett keys generate
# Import sender/recipient(s) public keys:
sett keys import from-keyserver alice@example.com

# Data encryption:
sett encrypt local --signer alice@email.com --recipient bob@example.com FILES_OR_DIRECTORIES_TO_ENCRYPT

# Data transfer
# to SFTP server:
sett transfer sftp --host HOST --username USERNAME --base-path DESTINATION_DIRECTORY --key-path SSH_KEY_LOCATION --key-pwd SSH_KEY_PASSWORD FILES_TO_TRANSFER
# to S3 object store:
sett transfer s3 --endpoint ENDPOINT --bucket BUCKET --access-key ACCESS_KEY --secret-key SECRET_KEY FILES_TO_TRANSFER

# Data decryption:
sett decrypt ENCRYPTED_FILES.zip

1.3 - Download sett

CLI

x86_64

Desktop

Standalone Installer

CLI

aarch64 x86_64

Desktop

.AppImage x86_64 .deb

CLI

aarch64 x86_64

Desktop

.dmg

You can download the latest version of sett for your operating system by clicking on the corresponding link above.

Downloading a specific version of sett

For a complete list of available versions, please visit the releases page.

Updating sett

As only the latest version of sett is officially supported and guaranteed to work, it is strongly recommended to always keep your local installation of sett up-to-date.

If you installed sett using the installer available for your operating system, sett will automatically check for updates and prompt you to install the newest version as soon as it's available.

If you are running a portable executable, make sure you regularly check this page and download the latest version of the executable.

1.4 - PGP key management with sett

To encrypt and decrypt data, sett uses public key cryptography. If you are not familiar with public key cryptography concepts such as public and private keys, revocation signatures, or keyservers you are advised to read this introductory section.

Generate a new public/private PGP key pair

A prerequisite to encrypt, decrypt and transfer files with sett is to have a public/private PGP key pair.

Generate a new key pair with sett-gui

To generate a new public/private key pair using sett-gui:

1. Go to the Keys tab and click on + Add. From the dropdown menu, choose Generate new key pair. A dialog box will appear.

   

2. Fill-in the required fields:

   • Full name: key owner first and last name.
   • Email: enter the email to be associated with the key pair.
   • Password: the password must be at least 10 characters long, and it is highly recommended that it contain a mix of letters, numbers and special characters.

   Important

   It is not possible to retrieve a password from a generated key, so make sure to remember it and store it in a safe place (e.g. a password manager).

3. Click Generate key and wait for a few seconds, a new key pair will be created.

4. A new pop-up window will appear to confirm that the key was generated successfully and to display the revocation signature associated with the new key, as shown in the figure below.

   Copy the revocation signature to a safe location, ideally a password manager, and click Next.

   

5. The next screen of the pop-up window allows to upload the public part of the key to the keyserver. Make sure to check the checkbox to also verify the key with its associated email address. In this context “to verify a key” means that, after your key is uploaded, the keyserver will send an email to verify that you are the legitimate owner of the email associated with the key.

   

6. The new key should now be listed in sett.

   

7. Now that your new PGP key is created, make sure to register it in the BioMedIT portal

Generate a new key pair in command line

To generate a new key pair using sett command line interface:

sett keys generate "Alice Smith <alice.smith@example.com>"

Additional options can be listed with sett keys generate --help.

After creating the new key, sett will display a revocation signature for it. Make sure you keep it in a safe place, such as a password manager. Revocation signatures can optionally be automatically exported to a file by using the -r, --rev-sig option:

sett keys generate --rev-sig "Alice Smith <alice.smith@example.com"

List your local PGP keys in command line

Private and public keys available in the keystore can be listed with the following command:

sett keys list

Export/import private keys in command line

In some situations (new computer setup, remote sett environment, ...) you might need to copy or move your private key to a different machine. This can be done by exporting the private key to a file, transferring the file to the new machine, and importing it.

Here is an example of how to export a private key to a file with sett:

# The key identifier can be its fingerprint, key ID or the user's email address.
sett keys export -o private_key.pgp -p alice.smith@example.com

Note that keys can also be exported in ASCII format (instead of binary) by adding the -a, --armor option.

After the key has been transferred to the new machine (in either binary or ASCII format), importing it is as easy as:

sett keys import from-file -p private_key.pgp

Note the usage of the -p option to import both the private and public parts of the key. If -p is omitted, only the public part of the key is imported.

Verify that the key has been correctly imported with sett keys list.

Ensure that you store any backed-up secret keys in a secure location and in an encrypted form (typically in a password manager).

Upload your public PGP key to the keyserver

sett allows users to upload their public PGP key to the keyserver specified in its settings.

Upload your public PGP key with sett-gui

Uploading your public PGP key to the keyserver specified in sett is straightforward:

1. In the Keys tab, look for your PGP key from the List of available keys and click on the ellipsis button to the right.
2. From the dropdown menu, click on Upload to keyserver.
3. A dialog box will appear to ask for confirmation.
4. To verify your key with the keyserver (i.e. associate your email address with your key on the keyserver), make sure that [x] Associate the key with your email address is checked. In practical terms, this means that people will now be able to search for your key on the keyserver using your email address, otherwise your key will only be searchable using its full fingerprint.
5. Click Upload to keyserver.
6. If you have selected the “associate key with email” checkbox, you will shortly receive an email from keyserver@keys.openpgp.org that contains a confirmation link to prove that you have access to your email.
7. Open the email and click on the link to associate your key with your email address. Other users now can find your public key with your email address.

Upload your public PGP key in command line

To upload your public PGP key to the keyserver, run:

sett keys upload alice.smith@exaple.com

In order to also trigger email verification for your key, add the -v, --verify option:

sett keys upload --verify alice.smith@exaple.com

Register your public PGP key in the BioMedIT portal

If you are not a BioMedIT user, this section is not relevant for you and can be skipped.

PGP key status

PGP keys used to encrypt, sign and decrypt data transiting via the BioMedIT network require the approval of the BioMedIT key validation authority. The information of whether a key is trusted or not is stored as key status in the BioMedIT portal. This is the reason why all PGP keys used within BioMedIT must be registered with the BioMedIT portal.

When a PGP key is first registered in the BioMedIT portal, its status is initially set to PENDING (i.e. it is awaiting approval). A keys must be in APPROVED status before it can be used to encrypt or sign data packages transiting via the BioMedIT network.

The list of key statuses is as follows:

• PENDING: a key approval request was submitted, but the key has not been approved yet. This is a manual process and can take from a few hours or up couple of days.
• APPROVED: key is approved for usage within the BioMedIT network. Only approved keys can be used to encrypt, sign and decrypt data packages that are transiting via the BioMedIT network.
• APPROVAL-REVOKED: approval of the key has been revoked by the BioMedIT key validation authority.
• KEY-REVOKED: key has been revoked by its owner.
• REJECTED: key is not trusted by the BioMedIT key validation authority.
• DELETED: key has been removed from the keyserver by its owner.
• UNKNOWN KEY: key has not been registered on the BioMedIT portal. If it is your own key, please register it. If it is the key of someone else, please ask them to register their key.

To verify that a key is trusted, sett connects to the BioMedIT portal and retrieves the status of the key. For this reason, it is important that BioMedIT users register their PGP key with the BioMedIT portal.

In cases where sett is used outside of the BioMedIT project, or the portal is not reachable, sett can still be used to encrypt, decrypt, and transfer data. In this case, you need to uncheck [ ] Verify DTR and [ ] Verify key approval in the Settings tab.

The status of a public key in the BioMedIT portal can be easily checked in sett-gui by going to the Keys tab, and expanding the key in the List of available keys.

Register your public PGP key with the BioMedIT Portal

To register a new PGP key with the BioMedIT portal, proceed as follows:

1. Make sure you have successfully uploaded your key to the keyserver and that you have completed the email verification procedure with the keyserver (i.e. your key must be verified with the keyserver).

2. To make sure that your key is present on the keyserver and is verified, go to the keyserver home page in your browser and search for the email associated with your key. You should get a message saying that a key was found for your email address, as shown in the example below.

   Your key fingerprint will also be shown - see the right-most part of the link in this screenshot:

   

3. Copy the fingerprint (40-character string) of your key.

4. Log in to the BioMedIT portal.

5. Go to the My Keys tab.

6. Click on the "+ KEY" button, a dialog box will open.

   Note: if the button is missing, it is probably because you already have an active key in the portal. Each user can only have 1 active key at a time - see the information box above.

7. Enter your full key fingerprint (must be exactly 40 characters long) in the dialog box, then press the green search icon to the right.

   This will retrieve the user ID and email address associated with the fingerprint from the keyserver and display them in the dialog box.

8. Verify the user ID and mail address. If they are correct for your PGP key, then click on Confirm.

9. A request to approve your key has now been sent to the BioMedIT key validation authority. Generally requests are processed quickly (in a matter of hours), but occasionally it might take slightly longer as this is a manual process.

   Please contact the BioMedIT support if your key has not been approved after a couple of days.

Download public PGP keys from the keyserver

In order to encrypt data for a specific recipient (who will be able to decrypt it), you will need to have the public PGP key of that recipient(s) available in your local keyring. sett allows to easily search and retrieve public PGP keys from the keyserver specified in the settings.

Download PGP keys with sett-gui

To download a public PGP key from the keyserver:

1. In the Keys tab, click on + Add. From the dropdown menu, choose Import from keyserver. A dialog box will open, allowing to search for public keys stored on the keyserver.

2. In the search field, enter either the full email address or fingerprint of the public key you are looking for and hit Search: the matching key found on the keyserver will be displayed.

   

   For instance, if searching for the key “Bob Test Key bob@example.com” with fingerprint “AEED7A96F339881F6FE8291464A1E0150613807D”, one can search for either “bob@example.com” or “AEED7A96F339881F6FE8291464A1E0150613807D”.

3. If the key you are looking for was found, click on Import. You should then see your recipient’s key listed in the List of available keys.

4. Expand the details of the newly downloaded key (by clicking on the key in the list or on the small ^ button to the right), and verify that its Approval status is set to “Approved” - also indicated by a green check mark.

   Important

   If the check mark is red, the Approval status and the Revocation status (visible by expanding the key) will give more details. Please see the PGP key status section for details. Hovering your mouser over the check mark icon will also display additional details.

Download PGP keys in command line

The following command shows how to download a public PGP key from the keyserver. The search term can be an email or a fingerprint.

sett keys import from-keyserver alice.smith@example.com

Remove your public PGP keys from the keys.openpgp.org keyserver

While it is not possible to remove an actual key from the keyserver, it is possible to remove all personal identification from it (user ID and email). Such keys are called unverified.

To remove personal information associated with a key, go to the keyserver's manage key page, enter the email associated to the key and click on Send link.

You will receive an email with further instructions on how to proceed to remove your key's user ID and email from the keyserver.

Delete PGP keys from your local machine

Deleting public and/or secret PGP keys form your local keystore is not possible with sett. This is intentional, and users are encouraged to revoke their key instead.

Should you nevertheless wish to delete a key from your local keystore (e.g. because you created a test key that is no longer needed), it is possible to do so by manually deleting the files containing the keys using your file explorer or a shell command.

In the local sett keystore, keys are stored as files named after the fingerprint of the key’s primary key. For instance, a key with fingerprint 3b17f529665fe012ef54f4a1714fdf98b6e828df would be stored under:

• Public key: <keystore-path>/pgp.cert.d/3b/17f529665fe012ef54f4a1714fdf98b6e828df
• Secret key: <keystore-path>/pgp.cert.d.sec/3b/17f529665fe012ef54f4a1714fdf98b6e828df

The location of the keystore (<keystore-path> above) is operating-system dependent:

• Linux: ~/.local/share/pgp.cert.d for public keys, and ~/.local/share/pgp.cert.d.sec for secret keys.
• Windows: %UserProfile%\AppData\Roaming\pgp.cert.d for public keys, and %UserProfile%\AppData\Roaming\pgp.cert.d.sec for secret keys.
• MacOS: ~/Library/Application Support/pgp.cert.d for public keys, and ~/Library/Application Support/pgp.cert.d.sec for secret keys.

To get the exact location of the key by using the sett-gui app:

1. In the Keys tab, look for your PGP key from the List of available keys and click on the ellipsis button to its right.

2. From the dropdown menu, select Delete.

3. A dialog box with important information about key deletion, as well as the exact location of the key will be displayed.

   

Generate a revocation signature

A prerequisite for revoking a PGP key is to have generated a revocation signature for it. If the PGP key to revoke was generated with sett, you should in principle already have a revocation signature ready to use. If you do not have a revocation signature yet, you can generate one with sett-gui:

1. In the Keys tab, look for your PGP key from the List of available keys and click on the ellipsis button to its right.

2. From the dropdown menu, select Create revocation signature. A dialog box will appear.

3. Fill-in the required fields:

   • Reason: the reason for revoking the key.
   • Message: a short explanation to the reason for revoking the key.
   • Password: the password for your private key.

   

4. Click Generate.

5. A new pop-up window will appear to confirm that the revocation signature was generated successfully. Copy the revocation signature to a safe location, ideally a password manager (anyone with access to a revocation signature can revoke the key for which it was generated).

   

Revoke your PGP key

If a private PGP key has been compromised, is no longer usable (e.g. password is lost), or should no longer be used for any other reason, it must be revoked.

A prerequisite for revoking a PGP key is to have generated a revocation signature for it. If the PGP key to revoke was generated with sett, you should in principle already have a revocation signature ready to use. If you do not have a revocation signature yet, please generate one by referring to the Generate a revocation signature section.

Revoke your PGP key with sett-gui

To revoke a PGP key in sett-gui:

1. In the Keys tab, look for your PGP key from the List of available keys and click on the ellipsis button to its right.

2. From the dropdown menu, select Revoke. A dialog box will appear.

3. Either paste you revocation signature from the clipboard or load it from a file.

   Warning: proceed with caution, a revoked key cannot be “un-revoked".

4. Click on Revoke.

5. When expanding the revoked key from the List of available keys, you should now see the Revocation status as “Revoked”. From this point on, the key can no longer be used with sett.

   

6. If you have previously shared your key via a keyserver (e.g. keys.openpgp.org), you must also re-upload your revoked key to that keyserver.

   This will allow other users to update their local copy of your public key, informing them that it is no longer valid. To upload your revoked public key, please refer to the Upload your public PGP key to the keyserver .

   If your key was never present on any keyserver, this step should be skipped.

Revoke your PGP key in command line

To revoke a key, run:

sett keys revoke alice.smith@example.com compromised "My dog ate it"

After a key has been revoked, it must be uploaded again to any keyserver(s) where it is present, so that the revocation can be shared with others. This can be done with sett as illustrated in the Upload your public PGP key to the keyserver.

Migrating keys from the GnuPG keyring to the sett keystore

PGP keys located in your GnuPG keyring are not automatically detected by sett, and they must be migrated to sett’s keystore.

Note that to migrate public keys (i.e. keys from other people), you can simply re-download them from the keyserver as shown in the Download public PGP keys from the keyserver section.

Migrate keys using sett-gui

Using sett-gui, this procedure is straightforward:

1. In the Keys tab, click on + Add. From the dropdown menu, choose Import from GnuPG. A dialog box will open, containing a list of keys in your GnuPG keyring.
2. Look for the correct key and click on Import. You should now see the key listed in the List of available keys. Note that if you are importing a private key, you will be asked to enter its password.

Migrate keys using the command line interface

To migrate a key from your GnuPG keyring, run:

# The search term can be an email or fingerprint.
sett keys import from-gpg alice.smith@example.com

Introduction to public-key cryptography, PGP and GnuPG

Public-key cryptography is a method for secure communication between two or more users. In this system, each user has a pair of unique keys consisting of a private key and a public key. Public and private keys are linked in the sense that, data encrypted with a given public key can only be decrypted with the matching private key, and data signed with a given private key will only be recognized by the matching public key.

Because these keys are based on the OpenPGP protocol, they will here be referred to as PGP keys.

Public and private PGP keys:

• Public keys are used to encrypt data, as well as for verifying signatures made on files or emails. By design, public keys are intended to be shared with other people and therefore no particular effort is required to keep them secret. In fact, public keys are often uploaded to public servers, known as keyservers, where they are accessible to anyone. No password is required to use a public key.

  Typically, public keys are used by data senders to encrypt data for one or more recipient(s), and by data recipients to verify signatures of files or emails (to ensure the sender is genuine).

• Private keys, sometimes also referred to as a secret keys, are used to decrypt data, sign files and sign other people's public keys. To increase security, private keys should always be password protected.

Typically, private keys are used by data recipients to decrypt data, and by data senders to sign the files they encrypt.

sett uses the open source implementation of public-key cryptography provided by Sequoia-PGP: a modular OpenPGP implementation in Rust.

It is also possible - and often desirable - to both encrypt and sign a file. This ensures that the data can only be read by the intended recipient, and that the recipient can be confident the sender is legitimate. This is precisely what sett does:

• Encrypting files, so that only the intended recipient(s) can read them.
• Signing files, so that the recipient(s) can trust the sender is genuine.

Key fingerprints

Each pair of public/private PGP keys is identified by a unique fingerprint. Fingerprints are 40 characters long hexadecimal strings (digits and upper case A-F letters) that look like this:

238565936FCFF3F200219990941A3EC20555F781

Since nothing is preventing two PGP keys to have the same user name and email address, it is critical that users always verify the genuineness of new keys before (or just after) importing them into their local keyring (i.e. their local PGP key database).

Ensuring a key is genuine can be done in two different ways:

• Ask the key owner to provide their key’s fingerprint via a trusted communication channel (e.g. over the phone), and then verify that the fingerprint of the newly imported key does indeed match the fingerprint communicated to you by its owner.
• Using sett-gui, verify that the key status of the key is APPROVED (can only be checked after you imported the key).

File encryption

In public key cryptography, the sender encrypts a file using one or more recipient(s) public key(s). Once a file is encrypted, no one can read the file without having access to a private key that matches the public key(s) used for encryption. This ensures that only the intended recipient(s) can decrypt the file, because they are the only one to have access to the matching private key.

File signing

The objective of file signing is to guarantee to the recipient of a file (or email) that the sender is genuine, and not someone else trying to impersonate the sender.

To achieve this, the sender signs the file with their private key (password required), and shares their public key with the recipient (typically via a keyserver). The recipient can then validate the authenticity of the signature using the public key of the sender. Since public keys are non-sensitive, they can be distributed publicly. In fact they are intended for this purpose, hence their name.

Revocation signatures

In the unfortunate event that a user either i) forgets their private key’s password or ii) have their private key and password stolen/compromised, they will need a way to let other people know that their public key should no longer be trusted and used.

This is because:

• If the password was forgotten: the key owner won’t be able to decrypt data anymore.
• If the private key was compromised: someone else might be able to decrypt data encrypted with the public key, and to illegitimately sign files!

This situation is what revocation signatures are for: by applying a revocation signature to a public key, and then sharing the revoked key with others (e.g. via a keyserver), the key owner signals that their key is now "revoked" and should no longer be trusted nor used. After a key has been revoked, it can no longer be used to encrypt/decrypt data with sett.

Revocation signatures can be generated at anytime from the private key, but the best practice is to generate them directly after a new key pair is created. This ensures that the revocation signature will be available even if the private key or its password is lost.

Since anyone with access to a revocation signature will be able to revoke the associated key, revocation signatures must be stored securely - e.g. in a password manager - and should never be shared with anyone.

Exchanging public keys via a keyserver

Encrypting files for a specific recipient requires to have the recipient’s public key in one's local keyring (a keyring is a local database containing PGP keys). Similarly, verifying a signature on a file or a public key requires to have the signee's public key available in one’s local keyring.

Public keys are not sensitive data, and therefore can in principle be sent unencrypted via email. However, when having frequent key exchanges between multiple actors, sending public PGP keys around by email quickly becomes cumbersome. A solution to this problem is using a so called keyserver to share public keys. Keyservers are public or private servers whose sole purpose is to store public PGP keys and allow users to search for them. In addition, public keyservers often form a network and are regularly synchronized among themselves to ensure redundancy. Private keyservers have the same role, but do not share/synchronize the stored public keys with any other server, and possibly have a restricted access policy so that only authorized people can search for keys.

1.5 - Generating SSH keys

SSH (Secure SHell) keys are pairs of small text files that are used to securely identify users and give them access to remote servers, e.g. when transferring data via SFTP.

SSH keys use public-key encryption and always come in pairs: a public and a private (or secret) key.

• Public key: the public key of an SSH key pair is meant to be placed on the remote machine to which a user wants to connect. Public keys are non-sensitive, and are typically shared by users with system administrators, who will place them on the machine to which the user is granted access.
• Private key: the private key of an SSH key pair is what uniquely identifies a user as the legitimate owner of a public key. In other words, having the private key that matches a given public key will give access to any machine on which a copy of public key is stored. Private SSH keys are sensitive information: they must be kept private at all times and should never be shared with anyone - not even your system administrator. Private keys can (and should) be protected by a password, so that even if someone else has access to them, they remain unusable.

Generating a new pair of SSH keys must be done only once, and, in the context of sett, is only needed if you intend to transfer data. If you are a user who only decrypts data, you do not need an SSH key.

Also, do not confuse SSH keys - used to identify yourself on a remote server - with PGP keys - used to encrypt and sign data.

To generate a new SSH key pair, type the command below in your terminal (Linux and Mac) or PowerShell (Windows users - to start it, search for "powershell" in the Start menu). Note that you must replace "alice@example.org" with your own email. This will generate an SSH key pair using the "ed25519" algorithm, currently the most secure public-key algorithm:

ssh-keygen -a 100 -t ed25519 -C "alice@example.org"

Windows users who do not have the ssh-keygen command installed, please see section-install-ssh-keygen-windows below.

When executing the ssh-keygen command above, you will be prompted for the following information:

1. The name and location where to save the newly created keys. Here you should simply accept the default values by not entering anything and pressing "Enter" on your keyboard. Default locations are ~/.ssh/id_ed25519 on Linux and MaxOS, and C:\Users\%username%\.ssh\id_ed25519 on Windows.
2. A password to protect your private SSH key against illegitimate use. Please use a password that is long enough (>=12 characters) and composed only of ASCII characters.

When the command completes, two new files are produced: id_ed25519.pub (the public key) and id_ed25519 (the private key).

On Linux and MacOS systems, after the public key is generated, its permissions must be changed with the following command (this step is not needed for Windows users):

chmod 600 ~/.ssh/id_ed25519.pub

Windows users: enabling the ssh-keygen command

Not all versions of windows come with the ssh-keygen command pre-installed. If this command is unavailable on your machine, please install it as follows:

1. Open the windows settings (shortcut: windows key + i).
2. Search for, and select, "Add an optional feature".
3. Click on "Add a feature".
4. Search for, and select, "Open SSH Client".
5. Click "Install".
6. Restart your computer.

SSH private key with non-ASCII characters password

Even though it is possible to create an SSH key pair using a password containing non-ASCII characters, it seems like those characters are encoded differently between different operating systems.

As an SSH key might be moved to a machine with another operating system, or encoding might change with a new version, it is impossible to guess the correct encoding in any case. For this reason, we recommend not to use non-ASCII characters to protect SSH private keys.

1.6 - Encrypting, transferring and decrypting data with sett

Encrypting files

The sett application allows the encryption of any combination of individual files and directories.

The files are first compressed into a single data.tar.gz archive, which is then encrypted with the public key of one or more recipient(s), and signed with the sender’s key. The encrypted data (data.tar.gz.gpg) is then bundled with a metadata file - a plain text file that contains information about who is sending the file and to whom it should be delivered - into a single .zip file. The specifications of the output .zip files produced by sett are described in the sett packaging specifications section.

sett supports multi-recipient data encryption. This allows the encrypted file to be decrypted by multiple recipients.

sett also ensures the integrity of the transferred files by computing checksums on each file that is packaged, and adding this information to the encrypted data. The integrity of each file is verified automatically upon decryption of the file by sett, providing the guarantee that all files were transferred flawlessly.

Output file naming scheme

By default, encrypted output files produced by sett are named after the pattern:

<project code>_<YYYYMMDD>T<HHMMSS>.zip

where:

• <project code> is the abbreviation/code associated with the project. If no DTR ID value was provided or if Verify DTR is disabled, no project code is added as a prefix to the output file name.
• <YYYYMMDD> is the current date (Year, Month, Day).
• <HHMMSS> is the current time (Hours, Minutes, Seconds).

Example: demo_20220211T143311.zip, here demo is the project code.

Using the sett command line when encrypting to the local file system, it is possible to completely override the above output file naming scheme by passing the -o, --output option. Overriding the naming scheme is not possible when using sett-gui or when encrypting to a remote S3 or SFTP destination.

Encrypting data with sett-gui

To encrypt data:

1. Go to the Encrypt tab of the sett application.

   

2. Select files and/or directories to encrypt: using the Add file and Add directory buttons, select at least one file or directory to encrypt.

   After adding files/directories, they will be listed in the top box of the tab (see figure above).

3. Select data sender: in the drop-down list found under Select sender, select your own PGP key (you are the data sender). For most users, there should in principle be only one key in the Sender drop-down menu: their own key.

   Note

   The Sender key is used to sign the encrypted data, so that the recipient(s) can be confident that the data they receive is genuine.

4. Select data recipients: add one or more recipients by selecting them from the drop-down list found under Select recipients. Recipients are the people for whom data should be encrypted: their public PGP key will be used to encrypt the data, and only they will be able to decrypt it.

   BioMedIT

   Only recipients assigned to the role of Data Manager of the project for which data is being encrypted are permitted as data recipients.

5. Transfer ID (DTR): Data Transfer Request ID associated with the data package that is being encrypted. Specifying a valid DTR ID is mandatory to transfer data into the BioMedIT network.

   For data not intended to be transferred into the BioMedIT network, the DTR ID field can be left empty (or set to any arbitrary value). In this case, Verify package must be disabled (in the Settings tab).

   BioMedIT

   DTR ID field is mandatory. Only files encrypted with a valid and authorized DTR ID value can be transferred into the secure BioMedIT network. For this reason, BioMedIT users should always leave the Verify DTR checkbox enabled.

6. Select destination: destination where to encrypt the data. Possible destinations are:

   • Local: The local file system. When using this destination, it’s possible to specify:

     • Output location: directory where the encrypted file should be saved. By default, output files are saved to the user’s home directory. This default behavior can be changed by changing Default output directory in the Settings tab.

   • S3: A remote S3 compatible object store.

   • SFTP: A remote SFTP server.

   When encrypting to a remote destination, a number of options must be specified. Please refer to the transferring files for details about these options.

7. You are now ready to compress and encrypt the data: click Encrypt package or Send package if you are encrypting to a remote S3 or SFTP destination. A pop-up will appear, asking for the password associated with the sender’s key. After the password is entered, data compression and encryption will start. Progress and error messages are displayed in the Tasks tab.

   When the encryption completed successfully, a notification will pop-up with a message that reads: “Encryption job finished”.

   

At this point, all input files are compressed, encrypted and bundled into a single .zip file. If the destination was SFTP or S3, data has also been transferred to the remote destination.

Encrypting data on the command line

The sett command to encrypt data is the following. Note that the SENDER and RECIPIENT values can be specified either as a PGP key fingerprint, or as an email address.

# General syntax:
sett encrypt local --signer SENDER --recipient RECIPIENT --dtr DATA_TRANSFER_ID --output OUTPUT_FILENAME_OR_DIRECTORY FILES_OR_DIRECTORIES_TO_ENCRYPT

# Example:
# long command line options:
sett encrypt local --signer alice@example.com --recipient bob@example.com --dtr 42 --output test_output ./file_to_encrypt.txt ./directory_to_encrypt
# short command line options:
sett encrypt local -s alice@example.com -r bob@example.com -dtr 42 -o test_output ./file_to_encrypt.txt ./directory_to_encrypt

Data can be encrypted for more than one recipient by repeating the flag --recipient, e.g. --recipient RECIPIENT1 --recipient RECIPIENT2 option:

# In this example, Alice encrypts a set of files for both Bob and Chuck.
sett encrypt local --signer alice@example.com --recipient bob@example.com chuck@example.com FILES_OR_DIRECTORIES_TO_ENCRYPT

output is an optional argument that allows to specify the location and/or name for the encrypted output file. The output argument can be one of the following:

• Not provided: the encrypted data is written to stdout. This can e.g. be useful to pipe the data into another application.
• A directory: the encrypted data package file is written the the specified directory, and is given a name that follows the default naming convention in sett.
• A file name: the encrypted data is written to a new file with the specified name, and in the specified directory, if the file name contains one.

local can be replaced with s3 or sftp to encrypt and transfer data to a remote destination in a single command. When using s3 or sftp, the data is encrypted and streamed directly to the destination, without creating saving a local version of the encrypted data. This is faster and also saves space on the local machine when transferring large datasets.

sett encrypt s3 -s alice@example.com -r bob@example.com --endpoint https://minio.my-node.ch --bucket my-project \
--access-key 23VO8RB2SIB2SF8EUL9V --secret-key wvrt7YoTTERGftf0zWnppWYSdcGplNtxuLHMn7op --session-token eyJhbGciOiJ\
IUzUxMiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NLZXkiOiI5Vk84UkIyvimUMlKIFUVVTDc3WSIsImF0X7hhc2giOiIyRnVlZ3JmSjhTUWFXYkw2V0puek\
F3IiwiYXVkLjpbIm1pbmlvIl0sImF1dGhfdGltTRI6MTcyMTezODIxMywiZXhwIjoxNzIx0DMxODEzLCJpYXQiOjE3MjE4MzfiLKLsImlzwqI6Imh\
0dHBzOi8vcD3ydGFsLXN0YWasfmcuZGNjLnNpYi5zd2lzcy9hdXRoL38hdXRoIiwibmFtZSI6ImJpd2ciLCJqt5xpY5kiOiJjb25zb2xlQWRfgW4iLC\
JzdWIiOiIxOSJ9.PcvXcAli5Bz8ete1T265TPB1cbfgX7k8NDXU5gXy1nflxq203cG5qwAF9Oxyn1mKmwa87jsHj8HU2VUY9p5S1Q \
FILES_OR_DIRECTORIES_TO_ENCRYPT

Adding the --check option will run the encrypt command in test mode, i.e. checks are made but no data is encrypted.

The data compression level used by sett can be manually adjusted using the --compression-level option. Compression levels value must be integers between 0 (no compression) and 9 (highest compression). Higher compression levels produce smaller output files but require more computing time, so you may choose a lower level to speed-up compression (e.g. --compression-level=1), or a higher level (e.g. --compression-level=9) to produce smaller output files. The default level is 6.

Before encrypting data, sett verifies that there is enough free disk space available on the local machine to save the encrypted output file (relevant is the current working directory or target folder pointed by --output). If this is not the case an error message is displayed and the operation is aborted. Since the compression ratio of the input data cannot be known in advance, sett uses the conservative estimate that the minimum disk space required is equal to the total size of all input files to be encrypted.

To automate the encryption process, the --password option can be used to specify the password of the singer PGP key.

sett performs DTR verification if the --verify option is passed. For non-BioMedIT-related transfers, the --verify option should not be passed.

To override the sett output file naming scheme, the --output option can be used to specify the path and name that the output file should have.

Transferring files

Data packages can be transferred to remote servers that support one of the following protocols:

• SFTP
• S3 object storage

Transferring files with sett-gui

To transfer encrypted files:

1. Go to the Transfer tab of the sett application.

   

2. Select encrypted files to transfer: click Select file and select a .zip file that was generated using the sett application.

3. Select destination: destination where to transfer the data. Possible destinations are:

   • S3: A remote S3 compatible object store. When using this destination a number of options must be specified:

     • URL: The URL of the S3 object store;
     • Bucket: The name of the bucket where the encrypted data should be stored;
     • Access key: The access key to use to authenticate with the S3 object store;
     • Private key: The private key to use to authenticate with the S3 object store;
     • Session token: The session token to use to authenticate with the S3 object store.

     BioMedIT

     This information must be retrieved from BioMedIT Portal by a data provider data engineer. Please refer to the dedicated section of the Portal user guide for instructions on how to do so. Portal allows copying the credentials to the clipboard, so that they can be pasted into sett by using the Paste from clipboard button.

   • SFTP: A remote SFTP server. When using this destination, a number of options must be specified:

     • Host: URL address of the server where the files should be sent.

     • User name: the user name with which to connect to the SFTP server.

     • Destination directory: absolute path of directory where files should be saved on the server.

     • SSH key location: name and full path of the private SSH key used for authentication to the SFTP server. This is only required if the SSH key is in a non-standard location. Only RSA keys are accepted.

       Do not confuse SSH keys - which are used to authenticate yourself when connecting to an SFTP server during file transfer - with PGP keys - which are used to encrypt and sign data.

     • SSH key password: password associated with the private SSH key given under SSH key location. If your SSH key password contains characters that are not ASCII characters, and that this results in an error, please see the SSH private key with non-ASCII characters section of this guide.

     BioMedIT

     For BioMedIT users, the SFTP connection parameters User name, Host URL, and Destination directory will be provided by your local BioMedIT node.

4. You are now ready to transfer the data. Click Send package and follow the progress of the transfer using the Tasks tab. By default, sett verifies data packages before initializing file transfers. These checks are required within the BioMedIT network, but can be skipped in other contexts by disabling the Verify package checkbox in the Settings tab.

Transferring data on the command line

sett command to transfer data:

# General syntax:
sett transfer sftp --host HOST --username USERNAME --base-path DESTINATION_DIRECTORY --key-path SSH_KEY_LOCATION --key-pwd SSH_KEY_PASSWORD FILES_TO_TRANSFER
sett transfer s3 --endpoint ENDPOINT --bucket BUCKET --access-key ACCESS_KEY --secret-key SECRET_KEY --session-token SESSION_TOKEN FILES_TO_TRANSFER

# Example:
sett transfer s3 --endpoint https://minio.my-node.ch --bucket my-project --access-key 23VO8RB2SIB2SF8EUL9V \
--secret-key wvrt7YoTTERGftf0zWnppWYSdcGplNtxuLHMn7op --session-token eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NLZXkiOiI5Vk84UkIyvimUMlKIFUVVTDc\
3WSIsImF0X7hhc2giOiIyRnVlZ3JmSjhTUWFXYkw2V0puekF3IiwiYXVkLjpbIm1pbmlvIl0sImF1dGhfdGltTRI6MTcyMTezODIxMywiZXhwIjoxNzIx0DMxODEzLCJpYXQiOjE3MjE4MzfiLK\
LsImlzwqI6Imh0dHBzOi8vcD3ydGFsLXN0YWasfmcuZGNjLnNpYi5zd2lzcy9hdXRoL38hdXRoIiwibmFtZSI6ImJpd2ciLCJqt5xpY5kiOiJjb25zb2xlQWRfgW4iLCJzdWIiOiIxOSJ9.PcvX\
cAli5Bz8ete1T265TPB1cbfgX7k8NDXU5gXy1nflxq203cG5qwAF9Oxyn1mKmwa87jsHj8HU2VUY9p5S1Q encrypted_data.zip

Note that --session-token is optional and only required when authenticating using temporary credentials, which is the case of most users interacting with the BioMedIT infrastructure.

Adding the --check option will run the transfer command in test mode, i.e. checks are made but no data is transferred.

For SFTP transfers, an SSH key is required for authentication on the host server. The private SSH key can be provided via 2 mechanisms:

• Specifying the location of the key via the --key-path option.
• Use an SSH agent to provide the key. The SSH agent is automatically detected by sett and no specific input from the user is needed. In ths case, the --key-path argument should be skipped.

To display help for the transfer command: sett transfer --help. To display help for a specific transfer protocol - e.g. s3: sett transfer s3 --help.

Decrypting files

The sett application allows the decryption and decompression of files in a single step. However, only files encrypted with the sett application, or files that follow the sett packaging specifications can be decrypted with sett.

Decrypting data with sett-gui

To decrypt and decompress a file:

1. Go to the Decrypt tab of the sett application (see figure below).

   

2. Select a file to decrypt with Select file.

3. Select destination directory: select a location where to decrypt/decompress the file.

   By default, output files are saved to the user’s home directory. This default behavior can be changed by changing Default output directory in the Settings tab.

4. Click Decrypt package to start the decryption/decompression process. A pop-up dialog box will appear to ask for the password associated with the PGP key used to encrypt the files.

Decrypting data on the command line

sett command to decrypt data:

# General syntax:
sett decrypt --output OUTPUT_DIRECTORY ENCRYPTED_FILES.zip

# Example:
sett decrypt --output /home/alice/data/unpack_dir /home/alice/data/test_data.zip

To display help for the decrypt command: sett decrypt --help.

To decrypt data without decompressing it, add the -d, --decrypt-only option.

If the -o, --output option is omitted, the data is decrypted in the current working directory.

If you want to automate the decryption process, you can use the -p, --password option to provide the password of the recipient PGP key.

1.7 - Settings for GUI and CLI

sett-gui settings

The sett desktop app allows a number of options to be customized via its Settings page. For instance, you may change the default output directory, or enable/disable package verification before a transfer.

Each setting has a predefined default value, which is used when first running the tool or if loading the current settings fails for any reasons.

To change the settings in the desktop app, navigate to the Settings tab:

Changes made to Settings become effective immediately. Changes can be reset back to their factory default by clicking on the Reset settings button.

For detailed settings explanations, please refer to the settings section below.

Settings

The following options can be set in the config file:

Verify package

When enabled (the default value), the following verifications are made before encrypting or transferring data:

• DTR ID is valid and the transfer is authorized.
• Sender and Recipients public OpenPGP keys are approved by the BioMedIT key validation authority.
• Recipients are approved Data Managers of the BioMedIT project for which data is being encrypted.
• The name of the data package matches the pattern <project_code>_<date_format>.zip. This ensures no sensitive information is mistakenly included in the file name.

Note that that some of the above verifications require communication with the BioMedIT portal. When using sett outside of a BioMedIT project, this setting should therefore be disabled.

Portal URL

URL of a BioMedIT portal instance. The portal is used for key approval, DTR (Data Transfer Request) validation, and retrieval of data associated with a given DTR (when sett is being used in authenticated mode).

The default value of this setting is: https://portal.dcc.sib.swiss.

Default output directory

Default destination directory for operations such as encryption to the local filesystem or decryption. Defaults are:

• Linux: $XDG_DATA_HOME or $HOME/.local/share;
• macOS: $HOME/Library/Application Support;
• Windows: {FOLDERID_LocalAppData}.

Non-editable settings

This section displays values of setting that cannot be modified by the user. These settings are displayed here for convenience. They can be copied to the clipboard via a dedicated “copy to clipboard” button.

sett-cli settings

The sett-cli is stateless by design, meaning that there is no persistent configuration file where settings can be modified and stored.
Instead, settings can be set via the following shell environment variables. All settings are optional and have a default value.

SETT_OPENPGP_KEY_PWD

Password to unlock the secret OpenPGP key used to decrypt or sign data. When this environmental variable is set, sett uses its content instead of interactively asking the user to enter a password.

SETT_OPENPGP_KEY_PWD_FILE

Full path and name of a file containing the password to unlock the secret OpenPGP key used to decrypt or sign data. When this environmental variable is set, sett uses its content instead of interactively asking the user to enter a password. The file containing the password should not be encrypted.

SETT_PORTAL_URL

URL of the BioMedIT portal instance to be used. For details see the description of the GUI Portal URL setting. This setting defaults to https://portal.dcc.sib.swiss.

SETT_OIDC_CLIENT_ID

Client ID with which sett should identify with the OpenID Connect issuer (see SETT_OIDC_ISSUER_URL). Only relevant when using sett in authenticated mode. This setting defaults to sett.

SETT_OIDC_ISSUER_URL

URL of the OpenID Connect issuer used when authenticating with the BioMedIT Portal. Only relevant when using sett in authenticated mode. This setting defaults to https://login.biomedit.ch/realms/biomedit.

1.8 - sett packaging file format specifications

sett compresses, encrypts and packages files in a single .zip file whose specifications are described below. Only files adhering to these specifications can be transferred or decrypted by sett, and failure to comply with the specification will generate an error.

File structure

sett [.zip]{.title-ref} files have the following structure:

YYYYMMDDThhmmss.zip
├── metadata.json
├── metadata.json.sig
└── data.tar.gz.gpg
    └── data.tar.gz
        ├── content/
        |   ├── [file1]
        |   ├── [file2]
        |   └── ...
        └── checksum.sha256

metadata.json

Metadata file containing the following information:

• transfer_id: transfer ID.
• sender: fingerprint of the sender's public PGP key.
• recipients: list of fingerprints of the recipients' public PGP keys.
• timestamp: datetime of metadata generation. Uses the datetime format YYYY-MM-DDThh:mm:ss±ZZZZ (where ZZZZ is the UTC offset).
• checksum: sha256 checksum of the encrypted data.tar.gz.gpg file. This allows verifying the integrity of the transferred file without decrypting it, and is used to make sure nothing was corrupted during the transfer process.
• checksum_algorithm: algorithm used to compute the checksum. Currently sha256.
• compression_algorithm: algorithm used to compress the inner tarball. gzip is used by default. An empty string indicates no compression.
• purpose: PRODUCTION or TEST.
• version: the version of the metadata specifications.

metadata.json.sig

Detached PGP signature for the metadata.json file.

data.tar.gz.gpg

A tarball encrypted using the receiver's public PGP key and signed with the sender's private key. It is compressed with compression_algorithm.

[file1], [file2]

One or several data files can be transferred. Data to be sent can be in any format, e.g. .txt, .csv, .dat.

checksum.sha256

sha256 checksum file of all files present in data.tar.gz. This is used to make sure nothing was corrupted during the encryption/transfer/decryption process.

File example

Examples of the content and structure of the metadata and checksum files.

metadata.json

{
  "transfer_id": 42,
  "sender": "AAABFBC698539AB6CE60BDBE8220117C2F906548",
  "recipients": ["D99AD936FC83C9BABDE7C33E1CF8C1A2076818C3"],
  "timestamp": "2020-01-29T15:31:42+0100",
  "checksum": "a8eb0ee5a6a53326b1c6f9bf94136da5d98a1dc6dceee21d62f694d71c4cf184",
  "checksum_algorithm": "SHA256",
  "compression_algorithm": "gzip",
  "purpose": "PRODUCTION",
  "version": "0.7"
}

checksum.sha256

41421f0c4a5353a5a0cdd37de3fd80a840a190ca997ad8044a67c4c1683f7b63
file1.csv
35ba157ed1c3269d731a438c466790a4f481bb49805e2d1f380df0c636792ff6
folder1/file.txt
fd9ebdbcc1a5fc35ded6e78a6b16ef658502c9d0b05dd4a2185d0f94ccf165cf
folder1/folder2/file.txt

1.9 - Frequently asked questions and bug reports

1. Running sett behind a proxy

ALL_PROXY=https://host.domain:port sett-gui

Bug reports

To report a problem with sett or if you have a question not covered in the present documentation please open an issue on our public gitlab repo https://gitlab.com/biomedit/sett-rs/-/issues

1.10 - Source code

sett is licensed under the GPLv3 (GNU General Public License) and the source code is available at https://gitlab.com/biomedit/sett-rs

sett is developed as part of the BioMedIT project.

1.11 - Benchmarks

sett performance benchmarks

Test setup

The benchmarks were run on an Apple Silicon M3 machine with 16-Core CPUs, 128GB memory, and 2TB disk. Version 5.1.0 of sett CLI was used.

One single binary file was used, with default compression (Zstandard, level 3). Decreasing the compression level and/or using text data will influence the results.

Transfer speed is highly dependent on the infrastructure connecting the data sender and recipients. In these benchmarks, data was transferred to the same machine sending the data, therefore representing a best-case scenario in terms of transfer speed.

All values shown un the results below are averages of 5 runs performed with hyperfine.

Results

The table and figure below present throughput speeds for different sett workflows:

• Encrypt: compress, encrypt package and transfer (S3, SFTP) data.
• Transfer: transfer already packaged data. There is no compression and encryption involved in this workflow.
• Decrypt: decrypt and decompress and a data package.

Throughput as function of file size: as can be seen, the curves in the figure below are linear, indicating that the throughput is constant regardless of the file size.

(click on the figure to enlarge).

2 - BioMedIT Portal User Guide

In the following sections we introduce the BioMedIT portal, how it works and how to get started using it.
Please use the table of contents menu to the left or below to navigate the topics.

What is the BioMedIT Portal?

The BioMedIT Portal is the one-stop entry to the BioMedIT Network. It provides users with a single point of access to the tools and services of the BioMedIT Network in a secure web-based environment.

With the Portal:

• Project Leads have a consolidated view of their projects and resources, enabling them to manage the research team’s members and access the project space.
• Data Providers and Data Managers oversee data transfers and monitor them.
• Researchers gain access to an expanding collection of collaborative and research-oriented tools, which encompass in-house, commercial, and open-source resources.

Access is secured with a login with your SWITCH edu-ID account and two-factor authentication.

Table of Contents

2.1 - Login

To either register for the first time or login to the BioMedIT portal:

1. Go to: https://portal.dcc.sib.swiss/. The SWITCH edu-ID authentication button will appear.

2. Click on > SWITCHedu-ID

   

3. Enter your email address and click on Login. You will be asked to enter your password.

   

4. Your second authentication method prompt will appear (SMS, OTP). Enter your Code and click on Login

   

5. You will be asked to enter a username of your choice and confirm, if you agree with, the Portal being notified when your affiliation changes and with the BioMedIT Portal Private Policy

   

6. The BioMedIT Portal Home will be displayed

2.2 - Home

As soon as you log in with your SWITCH edu-ID, the home page displays the menu bar of the left side, the Quick Access Panel, with short-cuts to central services in the middle, and the BioMedIT Feed on the right, where you may find news and announcements from the BioMedIT Team.

The Portal classifies user accounts by role, and the menu options visible for each user will depend on the role assigned. A Data Provider, a Project Leader, or a visitor without a specific role will all see different default menu options

Quick Access links

• SPHN Schema Forge
• SPHN DCC Confluence
• Federated Query System
• Terminology Service
• GitLab
• Harbor

2.3 - Profile

From the profile menu option, users can check and update their account details, and manage their PGP and SSH Keys.

My Profile tab

The My Profile tab shows the user’s account details.

Account details:

• Name: User’s name

• Username: User SWITCH edu-id identifier

• Local Username: The username the user chose during their first login to Portal.

• Email: Contact email currently selected from the list retrieved from the user’s SWITCH edu-ID

• Affiliations: List of organizations (e.g., university, research institute) linked to users SWITCH edu-ID account

• IP Address: The IP address from where the user is connecting

• Flags: List of flags enabled for the user

• Groups: The list of Groups the user is a member of.

Note that personal details are retrieved from the user’s SWITCH edu-ID. For any changes to their personal data or affiliated organizations, user may update their SWITCH edu-ID via this link.

For more resources about SWITCH edu-ID, refer to SWITCH edu-ID / Quick Links

OpenPGP Keys

From the OpenPGP Keys tab, users can see the PGP Keys they have registered in the Portal, along with their status.

Each key has the following information:

• Fingerprint: This is a unique identified for each PGP key. You should make sure sure that this value is matching with the fingerprint of your key.

• User ID: User ID associated with the PGP key. People usually have their full name (and sometimes affiliation) in this field.

• Email: Email address associated with the PGP key

• Status: The approval status of your key. The list of key statuses is as follows:

  • PENDING: a key approval request was submitted, but the key has not been approved yet. This is a manual process and can take from a few hours or up couple of days.
  • APPROVED: key is approved for usage within the BioMedIT network. Only approved keys can be used to encrypt, sign and decrypt data packages that are transiting via the BioMedIT network.
  • APPROVAL-REVOKED: approval of the key has been revoked by the BioMedIT key validation authority.
  • KEY-REVOKED: key has been revoked by its owner.
  • REJECTED: key is not trusted by the BioMedIT key validation authority.
  • DELETED: key has been removed from the keyserver by its owner.
  • UNKNOWN KEY: key has not been registered on the BioMedIT portal. If it is your own key, please register it. If it is the key of someone else, please ask them to register their key.

Registering a new PGP key

Once the above prerequisites are met, perform the following steps to register your key and request its approval:

1. Connect to the BioMedIT portal and go to Profile and OpenPGP Keys tab.

2. Click on +OPENPGP KEY.

   

3. Add Key dialogue box will open. Enter your key’s fingerprint: please copy-paste it from your computer to avoid typing errors. Then click on the green search icon.

   

4. If your key is present on the Open PGP keyserver, the user ID and email associated with your key will be retrieved and displayed in the dialog box. Please double check that the value is correct.

5. If the displayed user ID and email values are correct, click on CONFIRM.

   

Your key is now registered and an approval request as been automatically sent to the DCC. Please note that the verification and approval of your key by the DCC is a manual process and may take a few days.

Note that your key’s approval status will be PENDING until approved by the DCC. Once approved, the status will change to APPROVED, at which point the key can be used to encrypt and decrypt data to be transferred into the BioMedIT network.

SSH Keys

From the SSH Keys tab, users can add their public SSH key for a specific project.

Registering a new SSH key

1. Click on +SSH KEY.

   

2. Select the project from the drop-down list

3. Copy the content of your public ssh key

4. click on SAVE

2.4 - Groups

From the Groups menu option, Data Providers can manage users and assign them roles.

Data Provider User roles and responsibilities

The Data Provider roles are intended to provide Data Providing institutions with the ability to manage the users involved in data transfers to BioMedIT and assign permission levels according to their responsibilities.

DP Manager

The DP Manager represents the Data Provider institution within the scope of BioMedIT. A DP Manager is responsible for the institution’s internal processes related to BioMedIT and delegates responsibilities to other users in their organization by assigning/removing DP roles. In addition, a DP Manager is responsible for maintaining and keeping the roles assigned up to date, i.e., if an employee left or changed responsibilities, and informing the BioMedIT Node and the DCC accordingly. Finally, the DP Manager delegates the internal operation coordination to the DP Coordinator.

DP Managers are:

• Responsible for the ensuring data provisioning at the Data Provider institution.
• Responsible for escalations and security incidents within the Data Provider.
• Responsible to assign/revoke the DP roles to other DP users.

DP Coordinator

The DP Coordinator is responsible for internally coordinating the DP’s readiness to send data, ensuring the legal compliance, i.e., DTUA, DTPA, etc. and all technical measures required from the DP side are in place for projects in which Data Provider institution participates. The DP Coordinator must send confirmation of readiness for data transfers to the DCC when required, and act as the single point of contact for follow-up questions and/or issues in the process.

DP Coordinators are:

• Responsible for coordinating internal data transfer processes of the Data Provider institution
• Responsible for confirmation of legal readiness for data transfers
• Responsible for confirmation of technical readiness for data transfers

DP Technical Admin

The DP Technical Admin is responsible for all technical tasks required for the setup and maintenance of a secure connection from the DP organization to the BioMedIT Node e.g., network components configuration. He is also responsible to support DP Data Engineer in the onboarding process. There could be more than one DP Technical Admin.

DP Technical Admins are:

• Responsible for the configuration of the network components from the DP side to permit data transfers.
• Responsible for supporting the onboarding of the DP Data Engineer(s).

DP Data Engineer

The DP Data Engineer is responsible to prepare, encrypt and sign the data packages, and transfer them from the DP organization to the assigned BioMedIT Node. There could be more than one DP Data Engineer.

DP Data Engineers are:

• Responsible for preparing and executing the data package transfers between the Data Provider and the assigned BioMedIT Node, following the BioMedIT process for secure data transfer.

DP Security Officer

The DP Security Officer is the designated point of contact from the DP organization to receive incident and security notifications from BioMedIT such as (but not limited to) scheduled and non-scheduled downtimes in response to security incidents, emergency changes and systems upgrades.

DP Security Officers are:

• Responsible for serving as a point of contact for incident, maintenance and security notifications from BioMedIT, and distributing them internally.

DP Viewer

A DP Viewer can monitor the status of data transfers from their DP institution.

How to add a user to a Data Provider group

To add a user to a group:

1. Click on the edit button of the group:

2.In the Users field, type the user’s email address and click on +USER.

3.When no more users need to be added to the group, click on SAVE.

2.5 - Projects

In the Projects menu option:

• Project users find a consolidated view of their projects, the data transfers, and the tools and resources enabled for the project.
• Project Leaders and Permission Managers can manage project users.
• Data Managers of a project can submit a data transfer request.

Projects view

From the project’s menu option, users can see the list of their projects.

The projects displayed can be filtered by using the Search function.

By clicking over a project, users can display the project’s details:

• Project Details tab
• Data Transfers tab
• Resources tab

Project Details tab

The details tab shows the project’s general settings:

• Code: A unique identifier for the project

• Hosting Node: Name and code of the node where the project is hosted

• Archived: Yes / No

• Expiration Date: Date when the project expires.

• Legal Approval Group: Group responsible to confirm compliance with legal basis and approve production DTRs.

• Data Specification Approval Group: Group responsible to review data transfer’s Data Specifications and approve production DTRs.

• Legal Support Contact: Contact email address to engage support about ethical and legal questions.

• Enable SSH Access: Indicates if SSH access has been enabled for the project.

• Identity Provider: Source user identity provider specified for the project. By default is SWITCH edu-iD, but a node-specific pre-defined identity providers may be selected.

• IP Ranges: IP range(s) from where client connections to access B-spaces is permitted.
  If the user is connecting from an IP outside of the IP range, the message “Your IP address is not within the valid IP address ranges of the project” is displayed.

• Additional information: Notes, comments and relevant information details. Note that this field is only editable by admins (Node Managers and BioMedIT Support).

Data Transfers tab

The data transfers tab displays the list of data transfer requests submitted by the project.

• From the Search box, users can search for specific data transfer requests by any content as search criteria.

• +DATA TRANSFER: Function to submit a new Data Transfer Request.
  Note that this option is only visible to the project’s Data Manager(s).

  The data transfer request list is also accessible from the Data Transfers menu option.

Resources tab

Displays the access links to the tools and applications configured for the project.

To add new links, contact biomedit@sib.swiss or your local node’s support group.

Users tab

Displays the list of users and their role in the project.

Authorized flag:
This badge is displayed next to a project user when they have completed the mandatory BioMedIT training modules: Information Security Awareness Training

User roles

User roles are set for each project. Each authorized BioMedIT user can have several user roles and can be assigned access to several projects.

Roles and responsibilities

Project Leaders (PL):

• responsible and accountable for the project
• responsible for the data lifecycle management of the project
• acts as contact point for escalations and security incidents within the project
• responsible to discuss the setup and configuration of the project space for the project with representatives of the BioMedIT Nodes
• responsible for case-by-case authorization of Data Transfers out of the B-space
• assigned to the project in the User Role of “Permissions Manager” by default and with that is responsible to assign authorized BioMedIT users
• responsible to assign the User Role of “Permissions Manager” to authorized BioMedIT users
• responsible to ensure that at least one authorized BioMedIT user was assigned to the project space in the User Role of “Data Manager”
• can act as “Permissions Manager” and/or “Data Manager” and/or “(default) User”
• responsible to revoke access rights to the project space
• informed about every user change (add/remove) via email notification
• there can be multiple PLs per project

Permissions Manager (PM):

• accountable to the PL
• each project must have at least one project member with the User Role of “Permissions Manager”
• responsible for assigning the User Roles of “(default) User” and “Data Manager” to authorized BioMedIT users and with that granting them access to the project space
• responsible to revoke access rights to the project space
• acts as contact for the new project members to support them with registration and authorization as users of BioMedIT

Data Manager (DM):

• accountable to the PL
• each project must have at least one project member with the User Role “Data Manager”
• responsible for unpacking and decryption of data packages within the B-space
• responsible to extract data from the B-space for transfer out of the project space only following explicit authorization of the PL
• responsible to place the request(s) for Data Transfer(s) from a Data Provider to the project

User:

• accountable to the PL
• has access to one or several project spaces following authorization
• responsible for data processing within the B-space

Minimal:

• accountable to the PL
• can access the project’s details in the Portal, but has no access to the B-space.

Managing project users

When clicking on the icon from the top-right corner, the window below is displayed.

Adding users to a project

1. In the Users field, enter the user’s email address and click on +USER. Note that the user(s) must have an account in the portal.

   

2. You will be asked to confirm the user’s details before adding it to the project.

   
   

3. Review the details of the user, and click on ADD to confirm it.

4. You will return to the list of project users. Assign a role to the user by clicking on the corresponding checkbox(es).

   

5. Click on SAVE.

Removing users from a project

To remove users from the project, the Project Leader or the Permissions Manager should:

• Click on the 'x' next to the user’s name.
• After any change, click on SAVE

Submitting a Data Transfer Request

To submit a Data Transfer Request (DTR):

Click on

The following window will be displayed:

Complete the required fields:

• Unlimited / One Package: Select if the DTR will cover one or multiple data transfers

• Data Provider: Select the Data Provider from the drop-down list

• TEST / PRODUCTION: Select TEST if mock data will be sent, or PRODUCTION if the data will contain patient real data

• Additional Data Recipients: Enter the email(s) of the data recipient(s) and click on the +USER button.
  Note that recipients can only be users with the Data Manager role in the project.

• Legal Basis: If transferring real data (purpose PRODUCTION), enter the legal agreement (i.e. DTUA, DTPA) document name

• Data Specification: If transferring real data (purpose PRODUCTION), include the link to the Data Specification
  

  Example: https://git.dcc.sib.swiss/admin/projects/project-space/example-project/data-transfer/-/tree/main/data-transfer-1?ref_type=heads

Click on SAVE.

A new DTR will be created in INITIAL status. Confirmation emails with the DTR’s details are sent to the DTR requester, recipient(s), and approvers.

DTR Approvals

When the DTR is created, the following approval requests are submitted in parallel:

1. Node(s) Approval: As data processors, the involved node(s) must confirm the presence of a pre-existing legal basis for the data transfer and ensure that the necessary technical infrastructure is in place to support it.

2. Legal Compliance Approval: For SPHN/NDS projects, facilitated by the DCC ELSI Help Desk group who verifies the existence of a legal basis (e.g., DTUA, Consortium agreement) for the data transfer. For non-SPHN projects, a dedicated local Legal Compliance group may be optionally configured.

3. Data Specification Approval: For SPHN projects, the DCC will review and approve the data specification documents referenced in the ‘Data Specification’ link. This process ensures the compliance with the SPHN Semantic Interoperability Framework.

4. Data Provider Coordinator Approval: This approval request is triggered only when the previous approvers, (1),(2) and (3) have submitted theirs.

The Data Provider Coordinator is then requested to confirm if the data delivery has been approved through their internal governance processes.

When all approvers approve the request, the DTR status changes to AUTHORIZED, and data can then be transferred. Notification emails are sent to all parties involved in the data transfer:

If any of them rejects the DTR, its status is set to UNAUTHORIZED.

Monitoring the approval status of a Data Transfer Request

The status of a DTR is displayed in the last column of the DTR list:

Approval Status

• INITIAL: The DTR was submitted, but it has not yet been approved.
• AUTHORIZED: All approvers have authorized the DTR. The Data Provider can now send the data.
• EXPIRED: The Data Provider sent the maximum allowed number of data packages, and no additional data can be sent under this DTR ID
• UNAUTHORIZED: The DTR was previously authorized but is currently unauthorized for some reason (i.e., The BioMedIT Node is offline, problems with user permissions, issues on the Data Provider's end, etc.)|

Data Transfer details

When clicking on a particular data transfer, a pop-up window shows additional information

• Details tab
• Logs tab
• Approvals tab

Details tab

Displays the attribute values of the DTR as submitted by the project’s data manager:

Logs tab

Displays the individual data packages transfer logs - routing information and whether the data has arrived to the project workspace -.

Approvals tab

Displays the current approval status of a DTR by all approval groups:

2.6 - Data Transfers

In the Data Transfers menu option:

• Data Managers and Data Providers can search and view the list of submitted Data Transfers Requests (DTRs), monitor their authorization status and transfer logs
• Data Providers (DP Data Engineers) can fetch their credentials to transfer data to a research project.
• Data Provider Coordinators can approve or reject a DTR

DTR Approval status

The approval status of a DTR is displayed in the last column of the table:

• INITIAL: The DTR was submitted, but it has not yet been approved by the Data Provider and/or the BioMedIT node(s).
• AUTHORIZED: All approval groups have authorized the DTR.  The Data Provider can now send the data.
• EXPIRED: The Data Provider sent the maximum allowed number of data packages, and no additional data can be sent under this DTR ID
• UNAUTHORIZED: The DTR was previously authorized but is currently unauthorized for some reason (i.e., The BioMedIT Node is offline, problems with user permissions, issues on the Data Provider's end, etc.)

Additional details about which group approved when are shown when clicking on a specific DTR. See approval details

Data Transfer details

When clicking on a particular DTR, a pop-up window shows additional information

• Details tab
• Logs tab
• Approvals tab
• Credentials tab

DTR’s details are accessible using the following link structure: https://portal.dcc.sib.swiss/data-transfers/dtr_id

Details tab

Displays the attribute values of the DTR, as submitted by the project’s data manager:

Details:

• Project: Name of the project
• Transfer ID: Unique DTR identifier
• Data Provider: the Data Provider code
• Status: Approval status of the DTR
• Max Number of packages: 1 (single transfer) or UNLIMITED (multiple data transfers)
• Transferred Packages: Count of packages sent so far under this DTR ID
• Requestor: Name of the Data Manager that submitted the DTR
• Data Specification: link to the Data Specification
• Purpose: TEST (for mock data) / PRODUCTION (for patient real data)
• Legal Basis: Legal agreement (i.e. DTUA, DTPA) document name

Approvals tab

Displays the current approval status of a DTR.

Credentials tab

From this option, DP Data Engineers can fetch their credentials, required for authentication and authorization to transfer data when the transfer method is S3/HTTPs.

To fetch and copy their credentials, as DP Data Engineer:

• click on FETCH CREDENTIALS
• click on COPY ALL

Logs tab

Displays the individual data packages transfer logs - routing information and whether the data has arrived to the project workspace -.

Approving DTRs

When the DTR is submitted by a project’s Data Manager, the groups involved in the data transfer will receive a notification by email with all the details, for their information or for their action (request for approval), depending on the case, as explained below.

Approvals

When a DTR is submitted by a Data Manager to a research project, the following approvals must be collected:

1. Node(s) Approval: As data processors, the involved node(s) must confirm the presence of a pre-existing legal basis for the data transfer and ensure that the necessary technical infrastructure is in place to support it.

2. Legal Compliance Approval: For SPHN/NDS projects, facilitated by the DCC ELSI Help Desk group who verifies the existence of a legal basis (e.g., DTUA, Consortium agreement) for the data transfer. For non-SPHN projects, a dedicated local Legal Compliance group may be optionally configured.

3. DCC Data Specification Approval: For SPHN projects, the DCC will review and approve the data specification documents referenced in the ‘Data Specification’ link. This process ensures the compliance with the SPHN Semantic Interoperability Framework.

Only when the above groups, (1), (2) and (3) confirm their approval, the DP Coordinator is requested to submit theirs:

By clicking on the link in the email above (https://portal.dcc.sib.swiss/data-transfers/<DTR_ID>), DP Coordinators will be redirected to the BioMedIT Portal to approve or rejects the request.

To approve or reject the DTR, click on APPROVE

Declare, by clicking on the checkboxes, if:

• All technical measures are in place to send the data
• There is an existing legal basis for the Data Transfer
• This data delivery has been approved through internal governance processes

Click on APPROVE

When all approvers approve the request, the DTR status changes to AUTHORIZED, and data can then be transferred. Notification emails are sent to all parties involved in the data transfer:

If any of them rejects the DTR, its status is set to UNAUTHORIZED.

Download data transfer list

By clicking on the icon at the top-right corner, users can download the list of DTRs in csv format.

2.7 - Contact Us

It is the contact form to submit feature requests or support to the BioMedIT Help Desk (biomedit@sib.swiss).

2.8 - Central Services

2.8.1 - DCC GIT

1. Login to GitLab

1.1 GitLab is available via federated login from BioMedIT portal’s Quick Access link:

1.2 The following page will be displayed.

Click on Federated Login

2. Resources

• Get started with GitLab
• GitLab Learn → Watch videos and self-driven demos
• GitLab docs → Access step-by-step tutorials and guides

2.8.2 - Harbor

1. How to request access

1.1 Registry is available via federated login from the BioMedIT portal’s Quick Access links:

1.2 Click on the button: Login via OIDC provider

2. Resources

• Working with projects
• Project configuration
• Working with images
• Using the API explorer

2.8.3 - Federated Query System

1. Access requirements

In order to gain access to the FQS, users must meet the following requirements:

• The user’s institution must have signed an agreement with the SIB. Currently, all five Swiss university hospitals are part of this network, additional institutions will have the possibility to join at a later time point.
• Users have a SWITCH edu-ID account with their employing institution linked in their profile (though SWITCHaai).
• Each user has individually agreed to abide by the FQS Acceptable Use Policy.

2. Registration to the FQS

2.1 From the BioMedIT Portal’s Home, at the Quick Access tiles, click on the Federated Query System icon:

2.2 If it is the first time you access the Federated Query System, the system will display the Registration Form below, with your name and current affiliations in your SWITCH edu-ID.

If you do not have any linked affiliations in your SWITCH edu-ID, the system will give you the option to update your account.

a. Click on add your organizational identity on edu-ID. Update your account.

b. Return to this screen and click on I ADDED MY AFFILIATION when ready.

2.3 In the Registration Form,

a. Enter your institution

b. Enter your institutional email

c. Click on NEXT

2.4 The Acceptable Use Policy will be displayed.

a. Read carefully all sections

b. Click on I have read & agree when all the terms are read, understood, and agreed.

2.5 You will get confirmation that your request was sent.

2.6 Within 24 hours, you will receive an email with your account details and the option to set your credentials.

a. Open the email

b. Click on Set Password.

2.7 The Privacy Policy will be displayed:

a. Read carefully all sections

b. Click on I accept when you read, agreed and understood all the terms.

2.8 The Change Password screen will be displayed.

a. Enter a password that meets the security requirements

b. Click on Save changes

2.9 You will received confirmation that your password was changed successfully.

a. Click on the Log out button

3. Login to the FQS

3.1 From Quick Access tiles in the BioMedIT Portal’s Home, click on Federated Query System icon:

3.2 The Login page will be displayed.

a. Read the Acceptable Use Policy

b. Select I confirm when the you read and understood the Acceptable Use Policy

c. Click on Continue to Login

3.3 The Login to the SPHN Federated Query System page will be displayed.

a. Enter your username or email. You can find your username in the confirmation email you received from the DCC.

b. Click on the → icon

3.4 The Email verification required page will be displayed asking you to enter a code.

a. Check your inbox. You should have received an email with the code:

3.5 In the Email verification required page:

a. Enter the code your received by email

b. Click on Apply code

3.6 You will be prompted to enter your password.

a. Enter your password

3.7 If your credentials are correct, you must have logged in to the FQS successfully.

3 - BioMedIT Training

Information Security Awareness Training

Getting started with BioMedIT

This training series is intended for new users to get started with the BioMedIT network.

sett for beginners

BioMedIT Training series role based

4 - FAQs

1. General Information

2. Data Providers

3. Projects

4. Data Transfers

5. Data Management

6. Security and Privacy

7. Working Groups

8. SPHN

5 - Quick links for SWITCH edu-ID

Quick Links

• Create account: how to create a new SWITCH edu-ID account

• View and modify account data: how to check, add, delete or change account data.

• Reset password: how to set a new password if you have lost or forgotten it.

• Change password: how to set another password.

• FAQs - SWITCH edu-ID - SWITCH Help

6 - SPHN-DCC Confluence

How to request access

To access SPHN-DCC Confluence, users must create an Atlassian account. Here are steps:

Access the link https://sphn-dcc.atlassian.net. The following page will display

Select the option Create an account

The following page will display.

Enter your email address and click on Sign up

Confluence will send you an email with a verification link:

Check your mailbox and open the email sent by Confluence. Open the message and click on Verify your email

After verifying your email, Confluence will redirect you to the following screen to finish setting up your account:

Enter your full name
Create a password
Click on Continue

The SPHN-DCC welcome space should display.

Note that to gain access to specific spaces, you must contact the space owner to assign you permissions.

Guides for users

• Manage your account
• Confluence Essentials

Guides for space owners

• Organize and customize your Confluence space
• Set up & manage space permissions