Data package specification

sett compresses, encrypts, and packages files in a single .zip file whose specification is described below. Only files adhering to these specifications can be transferred or decrypted by sett, and failure to comply with the specification will generate an error.

File structure

sett .zip files have the following structure:

YYYYMMDDThhmmss.zip
├── metadata.json
├── metadata.json.sig
└── data.tar.gz.gpg
    └── data.tar.gz
        ├── content/
        |   ├── [file1]
        |   ├── [file2]
        |   └── ...
        └── checksum.sha256
metadata.json

Metadata file containing the following information:

  • transfer_id: numeric ID associated to each named data transfer. In authenticated mode, or if verify package is activated, sett checks via Portal that the sender is associated with the given data transfer and verifies the data transfer is valid and authorized.
  • sender: fingerprint of the sender’s public OpenPGP key, a 40-character hexadecimal string.
  • recipients list of fingerprints of the recipients’ public OpenPGP keys, i.e. a list of 40-character hexadecimal strings.
  • timestamp: point in time when a metadata file was generated. Uses the RFC 3339 format, e.g. 2024-09-03T14:06:32.675879Z
  • checksum: checksum/hash value computed on data.tar.gz.gpg, the file containing the compressed and encrypted data. This checksum allows to verify the integrity of the data.tar.gz.gpg file without having to decrypt and unpack it.
  • checksum_algorithm: algorithm used to compute checksum. Currently SHA256 is used, where SHA stands for Secure Hash Algorithm 256 bits (64 characters). The checksum of any data package – regardless of its size – is therefore always 64 characters.
  • compression_algorithm: algorithm used to compress the data before encryption in order to decrease a data package’s size and ultimately speed up the transfer time. sett uses zstandard by default, which is currently the fastest compression algorithm available. Other options are gzip and stored (no compression).
  • purpose (optional): intended use for the data package. Values can be PRODUCTION, TEST, or null (optional).
  • version: version of the sett package specifications (not to be confused with the version of sett itself). The sett application is designed to be backwards compatible with older sett packages.
  • extra (optional): additional, custom information regarding this data package. Must be given in a key-value format (see example below). Both keys and values are strings. To be able to add additional information to a sett package, you need to [x] Enable extra metadata in the sett settings before encrypting data.
metadata.json.sig

Detached PGP signature for the metadata.json file.

data.tar.gz.gpg

Compressed and encrypted tarball containing the data. This file is produced by performing the following steps (in this order):

  1. Bundle the files to encrypt into a single .tar archive file.
  2. Compress the .tar file with compression_algorithm.
  3. Encrypt the compressed data using the receiver’s public OpenPGP key and sign it with the sender’s private key.

To improve speed and minimizes disk space usage, sett performs the above operations as a continuous stream, without creating any intermediate files.

[file1], [file2], …

Data files to be encrypted. Data files can be of any format, e.g. .txt, .csv, .dat.

checksum.sha256

sha256 checksum file of all files present in data.tar.gz. This is used to make sure nothing was corrupted during the encryption/transfer/decryption process.

File example

Examples of the content and structure of the metadata and checksum files.

metadata.json

{
  "transfer_id": 42,
  "sender": "AAABFBC698539AB6CE60BDBE8220117C2F906548",
  "recipients": ["D99AD936FC83C9BABDE7C33E1CF8C1A2076818C3"],
  "timestamp": "2020-01-29T15:31:42+0100",
  "checksum": "a8eb0ee5a6a53326b1c6f9bf94136da5d98a1dc6dceee21d62f694d71c4cf184",
  "checksum_algorithm": "SHA256",
  "compression_algorithm": "gzip",
  "purpose": "PRODUCTION",
  "version": "0.7",
  "extra": {
    "key1": "value1",
    "key2": "value2"
  }
}

checksum.sha256

41421f0c4a5353a5a0cdd37de3fd80a840a190ca997ad8044a67c4c1683f7b63 file1.csv
35ba157ed1c3269d731a438c466790a4f481bb49805e2d1f380df0c636792ff6 folder1/file.txt
fd9ebdbcc1a5fc35ded6e78a6b16ef658502c9d0b05dd4a2185d0f94ccf165cf folder1/folder2/file.txt