This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

BioMedIT Blog

What's new in sett 5.6.0? 🚀

We’re excited to announce the release of sett version 5.6.0, packed with bug fixes and new features to enhance your experience.

For a complete list of changes, check out the changelogs:

To download the latest version, visit the download page and select your preferred distribution.

New Features & Improvements ✨

Authenticated Decryption from S3 in TUI 🔓

Decryption from the S3 object store is now available in the Terminal User Interface (TUI)!

BioMedIT users can now seamlessly decrypt data directly from an S3 object store. Simply log in to the Portal via the User tab, then switch to the Decrypt tab. In the S3 package selection window, choose a DTR ID from the list and enter the package name.

Tip: To launch the TUI, just run sett in your terminal.

File Size and Count Display in GUI 📊

The Encrypt tab now displays the total size and number of selected files. This enhancement helps ensure the selected data matches expectations before proceeding.

Need Help or Have Feedback? ❤️

We’d love to hear from you! If you have any questions or feedback, reach out to us at biomedit@sib.swiss.

The BioMedIT Central Team

What's new in Portal 9.1.0? 🚀

We are ecstatic to announce the release of BioMedIT Portal 9.1.0, which is packed with new features and improved usability.

You may find the full list of changes in the: CHANGELOG.

New Project’s User Role History tab 📜

We have added a new tab to the project view, which allows privileged project users to see the history of users and their roles in the project, including who added/removed the user, and when. This feature is useful for easily auditing and tracking changes in the project’s user roles.

For more details, see the dedicated section in the Portal user guide.

Refurbished Data Transfer Log 📬

We have refurbished the data transfer log tab, to make it easier to read and understand. The tab now shows a tabular overview of all data packages transferred within the current data transfer, as well as a detailed view of each single data package - available by clicking on the data package in the tabular view. The log now also includes information about whether the data package was deleted, and when.

Support and feedback ❤️

If you have any questions or would like to give us feedback, please feel free to contact us at biomedit@sib.swiss.

The BioMedIT Central Team

New BioMedIT Data Transfer Architecture

Over 2024, we transitioned our federated platform and architecture to a more secure and streamlined model

In its initial conception, BioMedIT linked every data provider for research projects to a single BioMedIT node, regardless of where the project was hosted. This snowflake design was chosen to minimize onboarding efforts for each data provider. Each provider had one landing zone, always at a designated node, where encrypted and signed data packages were sent—typically via Secure File Transfer Protocol (SFTP) from whitelisted IP addresses. From these landing zones, the data was internally transferred to the BioMedIT node hosting the project receiving the data.

As the network grew and evolved, our development and infrastructure teams designed a more streamlined architecture.

Key changes

1. A More secure transfer method: HTTPS vs. SFTP

HTTPS/S3 offers enhanced security compared to SFTP, leveraging robust encryption protocols and standard web ports to reduce potential vulnerabilities. Unlike SFTP, HTTPS simplifies network configurations, ensuring seamless integration with modern systems.

2. Simplified Data Providers onboarding

The onboarding process for new data providers is now more straightforward. HTTPS/S3 eliminates the need for complex SFTP configurations or the establishment of separate landing zones for each data provider. Any BioMedIT node hosting a project requiring data can onboard a provider with minimal effort.

3. Improved user onboarding

Granting permissions to new users to transfer data to a research project has been simplified significantly. With HTTPS, there is no longer a need to exchange SSH keys for authentication. Data Provider Managers can autonomously grant permissions to other users within their organization via the BioMedIT Portal, provided they perform the data transfer from the authorized institutional IP range.

4. Direct connections without transfer nodes

By enabling direct connections between Data Providers and BioMedIT nodes, the architecture no longer requires transfer nodes. This change not only simplifies the architecture but also reduces the number of data processors involved in the legal agreements. With fewer nodes acting as data processors, legal agreements are streamlined, saving time and effort for all involved parties.

How does the process looks like now?

Despite the architectural change, the data transfer process remains unchanged. Here is an overview of the workflow:

  1. A user within the Data Provider’s Data Engineer group launches sett (sett-gui or sett-cli) and authenticates using their SWITCH edu-ID.

  2. sett retrieves the following information from the BioMedIT Portal:

    • The Data Engineer’s role.
    • The list of approved data transfer requests for their organization.
    • The necessary connection details to perform the transfer, including:
      • URL of the S3 object store.
      • S3 bucket name: This is where the data should be uploaded. The bucket is named <project_code> and was created by the sysadmins as part of the B-space setup.
      • Access credentials: These include the access key ID, secret key, and write-only STS credentials.

  3. Using sett, the Data Engineer selects the data transfer request, encrypts the data with the Data Manager’s public PGP key, and signs it with their own private PGP key.

  4. When S3 is selected as the destination, sett uploads the encrypted and compressed data package to the correct destination as a new object. The object name follows this format:
    <project_code><YYYYMMDD>T<HHMMSS><optional suffix>.zip, which is the default of sett’s output file naming scheme.

  5. Once the transfer is complete:

    • The data package becomes available in the B-space.
    • The Data Manager (data recipient) is notified.
    • A log entry is created in the BioMedIT Portal.

  6. In the B-space, the Data Manager decrypts the data using sett. During this process, the sender’s signature and the checksum of all files are verified.

architecture

What's new in sett 5.5.0? 🚀

We are happy to announce the release of sett version 5.5.0, which brings - among others - a new interactive terminal-based user interface.

For a full list of changes, please refer to the changelogs:

To download the new version, please go to the download page, and choose your distribution.

Highlights of the New Release

Terminal User Interface (TUI) ⌨️

We are excited to introduce a brand-new interactive Terminal User Interface (TUI) for command-line users! This intuitive interface enables you to encrypt, transfer, and decrypt packages, with or without authenticated mode.

To launch the TUI, simply run:

sett

Please note that the TUI is a work in progress. While most features are available , some functionalities, such as authenticated decryption and key management, are currently missing but will be added in future updates.

Please note that, all previously available CLI subcommands (encrypt, decrypt, keys, etc.) remain fully supported and functional.

GUI: Removal of the “Paste from Clipboard” button 🗑️

In the latest Portal release, the S3 credentials tab has been removed. Consequently, the “Paste from Clipboard” button has also been removed from sett.

For transferring packages within the BioMedIT framework, please use the authenticated mode.

Support and feedback ❤️

If you have any questions or would like to give us feedback, please feel free to contact us at biomedit@sib.swiss.

The BioMedIT Central Team

What's new in Portal 9.0.0? 🚀

We are hyped to announce the release of BioMedIT Portal 9.0.0, which is packed with new features and improved usability.

You may find the full list of changes in the: CHANGELOG.

Bye bye credentials tab! 👋

We have removed the credentials tab from the data transfer view. This was a temporary solution to allow data engineers to fetch STS credentials for S3-based data transfers, while we worked on a more secure, user-friendly solution in sett.

With the advent of authentication in sett, sett is now able to fetch STS credentials on behalf of the user. Learn more about the topic by reading documentation about:

One data provider, multiple nodes 🏘️

BioMedIT has discontinued SFTP and the snowflake architecture as a supported data transfer protocol. Now each data provider can directly send data via HTTPs/S3 to each of the nodes.

Portal supports this new scenario by allowing each data provider to be linked to any number of nodes. This also means that node personnel will be able to access information about users of data providers associated to their node.

OpenAPI documentation for humans 🤖

Portal now offers an OpenAPI documentation for its API that is easier to access and read. You can find it at the following URL: https://portal.dcc.sib.swiss/schema.html.

What's new in sett 5.4.0? 🚀

We are happy to announce the release of sett version 5.4.0, which brings - among others - authenticated decryption from S3 object stores to BioMedIT users.

For a full list of changes, please refer to the changelogs:

To download the new version, please go to the download page, and choose your distribution.

Highlights of the new release

Authenticated decryption from S3 in sett GUI and CLI 🔓

Decryption from an S3 object store is now available in authenticated mode!

To decrypt data directly from an S3 object store, BioMedIT users can now simply login to portal from the sett GUI app (“User Profile” tab), and enter the name of the package to decrypt and its DTR ID number.

CLI users are not left behind, as authenticated decryption is also available on the command line:

# Authenticated data decryption from portal.
 sett decrypt s3-portal --dtr DTR_ID DATA_PACKAGE_NAME.zip

Additional package details displayed in the decryption tab 🔍

When loading a package for decryption, whether from the local filesystem or a remote S3 object store, additional details about the package are now displayed:

  • Data Transfer ID
  • Timestamp
  • Custom metadata (if any)

This is in addition to the information that was already (and remains) displayed:

  • Location of package
  • Sender and recipients

Support and feedback ❤️

If you have any questions or would like to give us feedback, please feel free to contact us at biomedit@sib.swiss.

The BioMedIT Central Team

What's new in Portal 8.3.0? 🚀

We are excited to announce the release of BioMedIT Portal 8.3.0, which is packed with new features and improved usability.

You may find the full list of changes in the: CHANGELOG - Portal 8.3.0

/new and /edit pages for projects 📝

Projects have evolved a lot since the first versions of the BioMedIT Portal. They now carry so much information, that creating or editing a project from a dialog window was not enough anymore. We have introduced dedicated pages for creating and editing projects, so that information is displayed more clearly. You may find them at:

https://portal.dcc.sib.swiss/projects/new

and

https://portal.dcc.sib.swiss/projects/<project_id>/edit

Enable/disable resources and services for a project 🛠️

You can now enable or disable resources and services for a project. This will also determine whether the dedicated tabs will be rendered, ensuring your users are not presented with unnecessary information. You can change this setting from the project edit page:

https://portal.dcc.sib.swiss/projects/<project_id>/edit

MINIMAL project role becomes NO ROLE 🪪

The MINIMAL role for project has been renamed to NO ROLE. This change is to better reflect the fact that the user is part of the project, but has no role assigned. In fact, the NO ROLE is now also mutually exclusive with other roles.

Support ❤️

We thank you for your continued support and feedback. If you have any questions, please feel free to reach out to us at biomedit@sib.swiss.

The BioMedIT Central Team

What's new in sett 5.3.0? 🚀

We are happy to announce the release of sett version 5.3.0, which brings cool new features.

For a full list of changes, please refer to the changelogs:

To download the new version, please go to the download page, and choose your distribution.

Decryption from S3 in sett GUI 📂

You can now use sett GUI to stream and directly decrypt data from an S3 object store, without creating any temporary files in the local file system.

Update expiration date for OpenPGP keys 📅

You can now update the expiration date for OpenPGP keys using sett GUI or sett CLI. Both tools will also warn you, starting from three months ahead of the expiration date, when you try to use that key for signing or decryption.

Easier UserID definition for new OpenPGP keys 🗝️

When generating a new OpenPGP key in sett CLI, you can now specify the name and email separately, which makes it easier to define the UserID for the key:

sett keys generate --name "Alice Smith" --email alice.smith@example.com

.deb packages for sett CLI 📦

sett CLI is now available as a .deb package for Debian-based distributions!

Support ❤️

We thank you for your continued support and feedback. If you have any questions, please feel free to reach out to us at biomedit@sib.swiss.

The BioMedIT Central Team

What's new in Portal 8.2.0? 🚀

We are pleased to announce the release of BioMedIT Portal 8.2.0, which is packed with new features requested by the BioMedIT nodes, along with interface updates, improved usability, and enhanced performance.

You may find the full list of changes in the: CHANGELOG - Portal 8.2.0

New Features ✨

  • PGP Key Information: You can now view PGP key details directly in the user profile.
  • Data Providers: Additional tabs display the data transfers list for the specific Data Provider and the Data Provider users.
  • Nodes: Node users have been moved to a separate tab.
  • Projects: New “Additional Project Information” and “Services” tabs have been added, allowing for more relevant details to the project and tracking of which users have access to specific project services.

Improvements 🛠️

  • Navigation: Administration menu items have been bundled together.
  • Users in projects: Improved and clearer view of user roles within projects.
  • Inactive Users: The page displayed for users with “inactive” status who return to the portal has been improved to provide them clearer information.
  • Email Notifications:
    • No emails are sent for updates to users with minimal role.
    • Redundant emails for when users are added or deleted to projects have been removed.
    • Filenames are now included in data package emails.

Support ❤️

We thank you for your continued support and feedback. If you have any questions, please feel free to reach out to us at biomedit@sib.swiss.

The BioMedIT Central Team

What's new in sett 5.2.0? 🚀

We are excited to announce the release of sett version 5.2.0, which brings several significant enhancements designed to improve usability and streamline the encryption and transfer of data.

For a full list of changes, please refer to the changelogs:

To download the new version, please go to the download page, and choose your distribution.

Simplified encryption and transfer GUI ✨

The “Encrypt” and “Transfer” tabs have been combined into a single tab, making the interface cleaner and easier to navigate. You can still choose to encrypt files first, and transfer them later.

BioMedIT Portal Integration 🔐

Users can now authenticate through the BioMedIT Portal, unlocking additional features:

  • When creating a package, you can now select eligible approved data transfer requests from a drop-down menu, rather than manually entering the DTR number.
  • For S3/HTTPS transfers, credentials are automatically retrieved—no need to switch back to the Portal to fetch them.

OpenPGP key expiration date 📅

It is now possible to set an expiration date when generating an OpenPGP certificate both from sett GUI and sett CLI.

Additional settings ⚙️

The “Settings” tab of sett GUI now displays more information, such as the location of your public and private key stores.

CLI for simultaneous decryption and transfer 📬

Users can now decrypt packages from an S3 object store by specifying the corresponding arguments.

Performance and other improvements 🏎️

Additionally, this version includes several bug fixes, as well as security and performance improvements.

Support ❤️

We thank you for your continued support and feedback. If you have any questions, please feel free to reach out to us at biomedit@sib.swiss.

The BioMedIT Central Team

Porting sett from Python to Rust

sett stands for “Secure Encryption and Transfer Tool” and is an application that facilitates and automates data packaging, encryption, and transfer in BioMedIT.

Over the past two years BIWG has ported sett from Python to a more modern technology stack, composed of Rust and Svelte. This port was performed because the Python implementation was showing some design limitations, which made the life of developers and users difficult. The list of limitations included:

  • A somewhat complicated installation process, requiring multiple steps and the usage of the command line;
  • A requirement on a local installation of gpg: a tool which is very problematic to handle programmatically, especially on some operating systems;
  • Python’s performances: good, but definitely a bottleneck when compared to the performance of a compiled language like Rust can provide;
  • A Python-based GUI where implementing complex behaviors and styling can be challenging.

Before diving deeper, it’s worth mentioning that the transition was carried out gradually. The Rust codebase was initially designed as a Rust library with Python bindings, which allowed BIWG to gradually replace Python core features—i.e. encryption, transfer, decryption, key management—with their Rust equivalents. These Rust-based core features have been running in production since months, nicely encapsulated in a Python-based shell. The last steps of this port were carried out more recently, when:

  • On 22nd April 2024, the first Rust-only version was released both for the GUI and the CLI;
  • On 30th June 2024, the Python version was discontinued.

sett-gui-architecture

Usability improvements

  • Simplified installation process. Both the CLI and the GUI can be downloaded as standalone executables, with no dependency on external software;
  • More modern looking GUI, with step-by-step user guidance (i.e. fields to fill are shown to the user as needed), and simplified CLI interface (i.e. less options);
  • sett no longer relies solely on the local gpg keyring but has now an internal key store, implemented using Sequoia PGP—a utility to migrate keys from the gpg keyring to the internal key store is provided by the tool;
  • Support for parallel compression and encryption;
  • Support for simultaneous encryption and transfer;
  • Support for simultaneous decryption and transfer;
  • Use of Zstandard as default compression algorithm. Faster than the previously used gzip algorithm, and multi-threaded;
  • Authentication with BioMedIT Portal, enabling an improved user experience—e.g. auto-retrieval of S3 credentials (no need to go to portal and copy/paste credentials anymore);
  • Auto-updater for the GUI application;
  • Code signing certificates, provided by:
    • Apple, for Mac binaries;
    • SSL.com, for Windows binaries.

Performance improvements

Performances are noticeably improved, according to BIWG’s investigation and as shown in the table that follows.

Legacy sett sett 5.2.0
Encryption 50 MB/s 249 MB/s
Decryption 122 MB/s 259 MB/s

Discontinued features

  • Support for Liquidfiles as a protocol for transferring data packages;
  • Support for persisted settings/configuration for the CLI;
  • Support for SSH jump host for transfer over SFTP.

What’s coming next ?

  • An interactive terminal user interface (TUI), alternative to the classical way of running sett CLI and especially useful in the authenticated mode;
  • Support for signing public keys, useful e.g. when the users don’t rely on a central trusted authority for key verification and approval;
  • More features based on authentication with BioMedIT Portal—e.g. better feedback to the users on the status of the BioMedIT network, improved data package tracing;
  • Support for simultaneous decryption and transfer from the GUI.

Fine details on the status of present and future developments are available in BIWG roadmap.