Data package specification

sett compresses, encrypts, and packages files in a single .zip file whose specification is described below. Only files adhering to these specifications can be transferred or decrypted by sett, and failure to comply with the specification will generate an error.

File structure

sett .zip files have the following structure:

YYYYMMDDThhmmss.zip
├── metadata.json
├── metadata.json.sig
└── data.tar.gz.gpg
    └── data.tar.gz
        ├── content/
        |   ├── [file1]
        |   ├── [file2]
        |   └── ...
        └── checksum.sha256
metadata.json

Metadata file containing the following information:

  • transfer_id: transfer ID.
  • sender: fingerprint of the sender's public OpenPGP key.
  • recipients: list of fingerprints of the recipients' public OpenPGP keys.
  • timestamp: datetime of metadata generation. Uses the RFC 3339 format.
  • checksum: sha256 checksum of the encrypted data.tar.gz.gpg file. This allows verifying the integrity of the transferred file without decrypting it, and is used to make sure nothing was corrupted during the transfer process.
  • checksum_algorithm: algorithm used to compute the checksum. Currently sha256.
  • compression_algorithm: algorithm used to compress the inner tarball. zstandard is used by default. Other options are gzip and stored (no compression).
  • purpose: PRODUCTION, TEST, or null (optional).
  • version: the version of the metadata specifications.
  • extra: optional field containing additional information in the key-value format. Both keys and values are strings.
metadata.json.sig

Detached PGP signature for the metadata.json file.

data.tar.gz.gpg

A tarball encrypted using the receiver's public PGP key and signed with the sender's private key. It is compressed with compression_algorithm.

[file1], [file2]

One or several data files can be transferred. Data to be sent can be in any format, e.g. .txt, .csv, .dat.

checksum.sha256

sha256 checksum file of all files present in data.tar.gz. This is used to make sure nothing was corrupted during the encryption/transfer/decryption process.

File example

Examples of the content and structure of the metadata and checksum files.

metadata.json

{
  "transfer_id": 42,
  "sender": "AAABFBC698539AB6CE60BDBE8220117C2F906548",
  "recipients": ["D99AD936FC83C9BABDE7C33E1CF8C1A2076818C3"],
  "timestamp": "2020-01-29T15:31:42+0100",
  "checksum": "a8eb0ee5a6a53326b1c6f9bf94136da5d98a1dc6dceee21d62f694d71c4cf184",
  "checksum_algorithm": "SHA256",
  "compression_algorithm": "gzip",
  "purpose": "PRODUCTION",
  "version": "0.7",
  "extra": {
    "key1": "value1",
    "key2": "value2"
  }
}

checksum.sha256

41421f0c4a5353a5a0cdd37de3fd80a840a190ca997ad8044a67c4c1683f7b63 file1.csv
35ba157ed1c3269d731a438c466790a4f481bb49805e2d1f380df0c636792ff6 folder1/file.txt
fd9ebdbcc1a5fc35ded6e78a6b16ef658502c9d0b05dd4a2185d0f94ccf165cf folder1/folder2/file.txt