Quartz 4

Codex Encryption Design

4 min read

As summarized in the proposal, we:

  • use a freshly generated random master key (at least 256 bits) per dataset (generated and kept on the user’s machine)
  • derive a new encryption key and also an IV for each block from the master key and the block index
  • use for example AES192-CBC

For example, we could have

key = SHA256( MASTER_KEY || block_index ), truncated to 192 bits
IV  = SHA256( MASTER_IV  || block_index ), truncated to 128 bits

where both MASTER_KEY and MASTER_IV are 256 bit random numbers, and || denotes concatenation.

If storing 512 bits (as opposed to a 256 bit minimum) of key material is a problem, we could derive both by the same key, for example as

blockKEY = SHA256( MASTER_KEY || 0x01 || block_index ), truncated to 192 bits
blockIV  = SHA256( MASTER_KEY || 0x02 || block_index ), truncated to 128 bits

In our implementation, we will be using the second scheme - starting with a random master key, and then for each block deriving a block level key (blockKEY) and block level initialization vector (blockIV).

For some introduction and examples on BearSSL, please consult:

In Codex, we use a slightly modified AES-CBC mode of operation:

  1. we first generate a random MASTER_KEY - this key will be returned to the user,
  2. from the MASTER_KEY, for each block, we derive the corresponding block level encryption key blockKEY and block level initialization vector blockIV as shown in the proposal above,
  3. using the derived blockKEY and blockIV, we then encrypt the block using the BearSSL encryption primitives as demonstrated above.

As we see above, the block index is used in the process of the key and initialization vector derivation. For this reason we also need to remember to convert the block index to a byte representation - we use big-endian ordering. For this conversion a very simple function can be used:

func toBytes[T: SomeInteger](value: T): seq[byte] =
  let v =
    if system.cpuEndian == bigEndian: value
    else: swapBytes(value)
  
  result = newSeq[byte](sizeof(T))
  copyMem(addr result[0], unsafeAddr v, sizeof(T))

or, we can just use the endians2.toBytes function from nim-stew: toBytes(blockIndex.uint32, bigEndian)

Below we show the code for the derivation of the block level key and initialization vector, followed by the encryption and decryption of a block:

import std/sequtils
import bearssl/[blockx, hash]
import stew/[byteutils, endians2]
 
import ./rng
import ./hash
 
let masterKey = newSeqWith(32, Rng.instance.rand(uint8.high).byte)
 
let blockIndex = 1.uint32  
let blockIndexArray = toBytes(blockIndex, bigEndian)
 
const 
  KEY_SIZE = 24 # 192 bits for AES-192
  IV_SIZE = 16  # 128 bits
  KeyDerivationIdentifier = "aes192_block_key".toBytes
  IvDerivationIdentifier = "aes192_block_iv".toBytes
  DefaultBlockSize* = uint 1024 * 64 # as used in Codex
 
let keyForBlock = hash(addr sha256Vtable, masterKey & KeyDerivationIdentifier & blockIndexArray.toSeq)[0 ..< KEY_SIZE]
let ivForBlock = hash(addr sha256Vtable, masterKey & IvDerivationIdentifier & blockIndexArray.toSeq)[0 ..< IV_SIZE]
 
var plaintext = newSeqWith(DefaultBlockSize.int, Rng.instance.rand(uint8.high).byte)
 
let key = keyForBlock
let ive = ivForBlock
let encBuffer = plaintext
 
# encryption
var encCtx: AesBigCbcencKeys
aesBigCbcencInit(encCtx, addr key[0], key.len.uint)
aesBigCbcencRun(encCtx, addr ive[0], addr encBuffer[0], encBuffer.len.uint)
 
assert encBuffer != plaintext, "Encryption failed, output should differ from input!"
 
# decryption
let ivd = ivForBlock
 
var decCtx: AesBigCbcdecKeys
aesBigCbcdecInit(decCtx, addr key[0], key.len.uint)
aesBigCbcdecRun(decCtx, addr ivd[0], addr encBuffer[0], encBuffer.len.uint)
 
assert encBuffer == plaintext, "Decryption failed, output should match input!"

where rng and hash are defined as shown in How to generate a random number using BearSSL and How to create a hash using BearSSL.

In a similar way we will proceed with the following, successive blocks.

API

All data uploaded to Codex will, be default, be encrypted.

The standard upload API - /api/codex/v1/data - after successful upload, returns the content identifier of the uploaded content, followed by the hex encoded master encryption key, separated by a single comma character :, eg:

curl -X POST \
  http://localhost:8001/api/codex/v1/data \
  -H 'Content-Type: application/octet-stream' \
  -H 'Content-Disposition: filename="enc.txt"' \
  -w '\n' \
  -T enc.txt
zDvZRwzm22eSYNdLBuNHVi7jSTR2a4n48yy4Ur9qws4vHV6madiz:541cc266b2ef426486cd981f2d7429347c68113bda2a80d21c9f18e250bfdaff

When downloading, the user is allowed to either retrieve the encrypted content, and then use an external utility (to be provided) to decrypt it, or the user can provide the encryption master key, which then will be used by the Codex client to decrypt the content, which then will be returned to the user:

export CID="zDvZRwzm22eSYNdLBuNHVi7jSTR2a4n48yy4Ur9qws4vHV6madiz"
export KEY="541cc266b2ef426486cd981f2d7429347c68113bda2a80d21c9f18e250bfdaff"
curl "http://localhost:8001/api/codex/v1/data/${CID}/network/stream?key=${KEY}" -o "dec.txt"

Notice that when no encryption key is provided, the returned content size will always be multiply of the block size.

Graph View

Backlinks