QUESTION: Encryption support thoughts

Hi,

I would like to have an idea on how encryption at the application level together with rom can be supported. We have something working but it requires some hacks because of the usage of dry types. So i am just reaching out here to see if what we are doing might be done in a better way. Let me try to explain what we did and see how this can be made better.

So we are mainly using event sourcing (https://github.com/RailsEventStore/rails_event_store) using the rom gem and from our event we create projections with simple read models. Those read models are stored in postgres (using rom sql) by projecting our events using a repo and dry struct. The pseudo code for that is as follows:

module ReadModels
  class FileRepo < ROM::Repository[:files]
    auto_struct false

    def all
      files.to_a.map(&ReadModels::File.method(:deserialize))
    end

    def by_id(id)
      ReadModels::File.deserialize(files.by_pk(id).one!)
    end

    def rebuild(id)
      file_data = project_for(id)
      file_data = ReadModels::File.serialize(ReadModels::File.new(file_data))

      upsert!(file_data)
    end
  end
end

module Types
  include Dry.Types()
end

module ReadModels
  class File < Dry::Struct
    extend AttrEncrypted

    attribute :id, Types::String
    attr_encrypted :file_name, key: 'This is a key that is 256 bits!!', encode: true, encode_iv: true

    def self.new(file_name:, **args)
      super(**args).tap do |file|
        file.file_name = file_name
      end
    end

    def self.deserialize(file_data)
      encrypted_file_name = file_data.delete(:encrypted_file_name)
      encrypted_file_name_iv = file_data.delete(:encrypted_file_name_iv)

      allocate.tap do |file|
        file.instance_variable_set :@attributes, {}
        file_data.each do |k,v|
          file.__attributes__[k] = v
        end

        file.encrypted_file_name = encrypted_file_name
        file.encrypted_file_name_iv = encrypted_file_name_iv
      end
    end

    def self.serialize(file_model)
      file_model.to_h.merge(
        encrypted_file_name: file_model.encrypted_file_name,
        encrypted_file_name_iv: file_model.encrypted_file_name_iv
      )
    end
  end
end

So let me explain the usage of the code above. The File struct has an attribute called file_name and that gets encrypted using a gem called attr_encrypted (https://github.com/attr-encrypted/attr_encrypted). This is just a wrapper that allows for define an attribute as encrypted and then it will then in the db create 2 columns. One which stores the encrypted filename and another which is the iv to make it random.

Now in order to work together with attr_encrypted there is some hacking needed cause of incompatible data structures underneath. The main thing the code has todo is on retrieval from the db it takes he 2 columns for the file name encryption (encrypted_file_name and encrypted_file_name_iv) and assigns them so the reader method of file_name (which attr_encrypted defines) can decrypt that value. On storing the value in the db the serialize then does the reverse and assigns the columns with the encrypted values.

I know that this is something that is specific for us and i am not asking for the rom project todo anything about that but it would be nice to know how this could be made better. Specially the part where one attribute (in this case filename) could be stored in 2 different columns (the encryption use case) and read back as a single attribute. Maybe i need to create a custom dry type attribute somehow that would be able todo it? I am not to familiar with rom so that’s why i am asking here for some help/suggestions.

If i need to explain more i can definitely do that.

Thnx already to all who reply.

Hi!

This is not the first time I see this usecase. You can handle encryption relatively easily via custom mappers. This means you could port your serialize/deserialize methods into custom mappers and use them in places where they are needed. At the repository level, I’d imagine this to look more or less like this:

module ReadModels
  class FileRepo < ROM::Repository[:files]
    auto_struct false

    def all
      deserialized_files.to_a
    end

    def by_id(id)
      deserialized_files.by_pk(id).one!
    end

    def rebuild(id)
      files.changeset(:upsert, data).commit
    end

    def files
      super.map_with(:deserializer)
    end
  end
end

The logic here is that when you use files.map_with(:deserializer) then the output data fill go through your custom mapper as the last step. Then for the rebuild you’d use a custom changeset for upserts and handle serialization via Changeset.map. You can define an upsert command and then tell you changeset class to use it like this:

class CreateFileChangeset < ROM::Changeset::Create
  # this assumes you registered your upsert command as `:upsert`
  command_type :upsert

  map do |tuple|
    # your serialization logic goes here
  end
end

Hope this makes sense :slight_smile:

Cheers!

Hi,

Thnx already for the fast response !!!

I will play around with it :slight_smile: There was a lot of terminology when i started looking into to rom and did not realise that mappers could be used for this. Will keep you posted.

So i played around with it and managed to get it to work but i do have a few questions about the possibility of cleaning up some things cause it does not feel “clean” enough. Let me me copy past the code below. Again the intend is a encrypt an attribute called file name when upserting and decrypt it when reading.

First the reading part (decrypting):

module Decryption

  def self.decrypt(source_hash, decryption_attributes)
    # here take encrypted_file_name out of the source_hash and decrypt it to decrypted_file_name
    decrypted_file_name = decrypt(source_hash[:encrypted_file_name])

    source_hash.merge({
      file_name: decrypted_file_name
    })
  end

end

module Types
  include Dry.Types()
end

module ReadModels
  class DecryptedFileRecord < Dry::Struct
    attribute :key, ::Types::String
    attribute :file_name, ::Types::String
  end

  class DecryptionMapper < ROM::Transformer
    relation :files, as: :decryption_mapper

    import :decrypt, from: Decryption, as: :decrypt

    map do
      decrypt :file_name
      constructor_inject DecryptedFileRecord
    end
  end
end

As you can see i am using the mapper like you pointed out. I had to look into Dry::Transformer on how to write my own mapping methods (it seems transproc has been migrated to dry transformer). Now in my file repo i do this to be able to get all of by key, which is working fine:

  class FileRepo < ROM::Repository[:files]
    auto_struct false

    def all
      decrypted.to_a
    end

    def by_key(key)
      decrypted.by_pk(key).one!
    end

    def decrypted
      files.map_with(:decryption_mapper)
    end
  end

Now for upserting new data i kinda do the same with the changeset approach:

module Encryption
  def self.encrypt(data, encryption_attributes = {})
    encrypted_file_name, encrypted_file_name_iv = encrypt(data.delete(:file_name))
    data.merge(
      encrypted_file_name: encrypted_file_name,
      encrypted_file_name_iv:encrypted_file_name_iv
    )
  end
end

class FileChangeset < ROM::Changeset::Create
    map do |tuple|
      Encryption.encrypt(tuple)
    end
  end
end


class UpsertFile < ROM::SQL::Commands::Postgres::Upsert
  relation :files
  register_as :create_or_update
  result :one

  constraint :files_pkey
  update_statement  encrypted_file_name: Sequel.qualify(:excluded, :encrypted_file_name),
                    encrypted_file_name_iv: Sequel.qualify(:excluded, :encrypted_file_name_iv)
end

class FileRepo < ROM::Repository[:files]
  auto_struct false

  # previous other methods ...

  def rebuild(key)
    # get data from my event store
    file_data = project_for(key)

    file_data = files.changeset(FileChangeset, file_data).to_h
    files.command(:create_or_update).call(file_data)
  end
end

So when i do FileRepo.new(DB).rebuild(‘a_key’) it works fine.

Now my questions:

  • I was not able to use the files.changeset(:upsert, data).commit like you mentioned. So i my case i tried files.changeset(:create_or_update, file_data).commit cause that was the name i named my command but then it errors with ArgumentError: +:create_or_update+ is not a valid changeset type. Must be one of: [:create, :update, :delete]. I also tried to use create instead and leave the command_type :create_or_update but that threw another error that the column value for encrypted_file_name was null so i guess the changeset did not went to the mapping at all. If you could explain me what i am doing wrong here that would be nice so i can just use the commit of the changeset before calling to_h and manually calling the command. => after realizing i need to give my custom changeset as the argument to changeset it worked :slight_smile: eg: files.changeset(FileChangeset, file_data).commit , then the only thing i had todo is to match the command_type in the changeset with the register_as in the command

  • Like you see in the decryption mapper (import :decrypt …) i use the import statement on the mapper to add my own mapper function. I expect that i could do the same on the Changeset but it seems that is not possible and the map function in the changeset only has a limited set of transform functions (hash only it seems). Is this by design and could it not be interesting that both would work the same way? Or am i missing something? Cause it would be nice if i could do import :encrypt, from: Encrypt, as: :encrypt and cleanup the code by doing the same as with the decryption and do => see below for explanation but i managed to solve it by importing my transform function on the PipeRegistry

    class FileChangeset < ROM::Changeset::Create
    
      import :encrypt, from: Encryption, as: :encrypt
    
      map do |tuple|
        encrypt :file_name
      end
    end
    

But thnx already for making me move forward :slight_smile:

In waiting for a reply i discovered two potential routes I could take to implement the thing that is working in a nicer way but would like to have some feedback first cause i am not to familiar with the rom way of working:

  • I see that a changeset can except command options where i would be able to pass in a mapper using command_options on the changeset class. But it seems that is only applied on the result and i would not be able to use that to do the encryption part but only the decryption after i have send the upsert to the database? => figured this out it was indeed only for the result of the upsert command

  • I see that a changeset can use plugins as well and maybe that is the way for me to do the encryption part on the changeset level? Can i use plugins for a custom mapper inheriting from ROM::Transformer? => looking from the code i could use plugin on either the changeset or command level to be able to hide the map part in the changeset but since it is similar to the one in the mapper i would keep the map for now for readability

  • It seems to me that it would still be nice to allow to import custom tranformations functions on the changeset level (with import) like we can do with ROM::Transformer. I see the docs at https://rom-rb.org/learn/changeset/5.2/mapping/ saying that it “support all transformation functions from transproc” and some additions exist. => see my reply below, by adding it to the pipeline i am able to use it, but still would be nice to have it in the same way as the transformer/mapper but can live with it for now

Thnx

Going fast here, sorry :slight_smile:

Managed to get the encrypt function working inside the map of the changeset by import it inside the Changeset::PipeRegistry. Seems a bit hacky but works :slight_smile:

module ROM
  class Changeset
    module PipeRegistry
      import :encrypt, from: Encryption, as: :encrypt
    end
  end
end

  class FileChangeset < ROM::Changeset::Create
    map do
      encrypt :file_name
    end
  end

So i don’t really need any feedback anymore cause i have a working system. Would still like to now if it would make sense to allow the ability of importing custom transform functions directly on a changeset.

First of all thanks for sharing all of this, it’s really helpful :slight_smile:

Yeah so THE PLAN is to expose a nice API for adding custom mapping functions. This will be available in rom 6.0.0 later this year. If you have some API ideas, please let me know!

I’m glad that you figured things out on your own :slight_smile: We can and we will make things like that simpler in the near future.

Glad this is planned and will definitely watch out for it :slight_smile:

I always hate it when i find tickets or questions in either github or other systems that are designed for support and people explaining their issue but actually never share the solution so that’s why i tend to be verbose about what i am doing.

I think an interesting approach to help explain all the powerful features in rom is to have more examples by use cases (like the encryption on that i have, or for example how ruby event store uses rom) that demonstrate all the possibilities of rom. I did understand the concepts (changeset, mappers, repo, …) by looking at the docs on the website but how all the things work together was a bit of a challenge :slight_smile:

Anyways i hope this thread can help others that might have similar questions.

1 Like

Yes this has been a great challenge. There’s a “Guides” section on the website which is meant to include “use cases” explaining how to do something specific using rom. Unfortunately nobody has found the time to actually fill it with content :cry: We’ll get there eventually though :slightly_smiling_face: