I would like to have an idea on how encryption at the application level together with rom can be supported. We have something working but it requires some hacks because of the usage of dry types. So i am just reaching out here to see if what we are doing might be done in a better way. Let me try to explain what we did and see how this can be made better.
So we are mainly using event sourcing (https://github.com/RailsEventStore/rails_event_store) using the rom gem and from our event we create projections with simple read models. Those read models are stored in postgres (using rom sql) by projecting our events using a repo and dry struct. The pseudo code for that is as follows:
module ReadModels
class FileRepo < ROM::Repository[:files]
auto_struct false
def all
files.to_a.map(&ReadModels::File.method(:deserialize))
end
def by_id(id)
ReadModels::File.deserialize(files.by_pk(id).one!)
end
def rebuild(id)
file_data = project_for(id)
file_data = ReadModels::File.serialize(ReadModels::File.new(file_data))
upsert!(file_data)
end
end
end
module Types
include Dry.Types()
end
module ReadModels
class File < Dry::Struct
extend AttrEncrypted
attribute :id, Types::String
attr_encrypted :file_name, key: 'This is a key that is 256 bits!!', encode: true, encode_iv: true
def self.new(file_name:, **args)
super(**args).tap do |file|
file.file_name = file_name
end
end
def self.deserialize(file_data)
encrypted_file_name = file_data.delete(:encrypted_file_name)
encrypted_file_name_iv = file_data.delete(:encrypted_file_name_iv)
allocate.tap do |file|
file.instance_variable_set :@attributes, {}
file_data.each do |k,v|
file.__attributes__[k] = v
end
file.encrypted_file_name = encrypted_file_name
file.encrypted_file_name_iv = encrypted_file_name_iv
end
end
def self.serialize(file_model)
file_model.to_h.merge(
encrypted_file_name: file_model.encrypted_file_name,
encrypted_file_name_iv: file_model.encrypted_file_name_iv
)
end
end
end
So let me explain the usage of the code above. The File struct has an attribute called file_name and that gets encrypted using a gem called attr_encrypted (https://github.com/attr-encrypted/attr_encrypted). This is just a wrapper that allows for define an attribute as encrypted and then it will then in the db create 2 columns. One which stores the encrypted filename and another which is the iv to make it random.
Now in order to work together with attr_encrypted there is some hacking needed cause of incompatible data structures underneath. The main thing the code has todo is on retrieval from the db it takes he 2 columns for the file name encryption (encrypted_file_name and encrypted_file_name_iv) and assigns them so the reader method of file_name (which attr_encrypted defines) can decrypt that value. On storing the value in the db the serialize then does the reverse and assigns the columns with the encrypted values.
I know that this is something that is specific for us and i am not asking for the rom project todo anything about that but it would be nice to know how this could be made better. Specially the part where one attribute (in this case filename) could be stored in 2 different columns (the encryption use case) and read back as a single attribute. Maybe i need to create a custom dry type attribute somehow that would be able todo it? I am not to familiar with rom so thatās why i am asking here for some help/suggestions.
If i need to explain more i can definitely do that.
This is not the first time I see this usecase. You can handle encryption relatively easily via custom mappers. This means you could port your serialize/deserialize methods into custom mappers and use them in places where they are needed. At the repository level, Iād imagine this to look more or less like this:
module ReadModels
class FileRepo < ROM::Repository[:files]
auto_struct false
def all
deserialized_files.to_a
end
def by_id(id)
deserialized_files.by_pk(id).one!
end
def rebuild(id)
files.changeset(:upsert, data).commit
end
def files
super.map_with(:deserializer)
end
end
end
The logic here is that when you use files.map_with(:deserializer) then the output data fill go through your custom mapper as the last step. Then for the rebuild youād use a custom changeset for upserts and handle serialization via Changeset.map. You can define an upsert command and then tell you changeset class to use it like this:
class CreateFileChangeset < ROM::Changeset::Create
# this assumes you registered your upsert command as `:upsert`
command_type :upsert
map do |tuple|
# your serialization logic goes here
end
end
I will play around with it There was a lot of terminology when i started looking into to rom and did not realise that mappers could be used for this. Will keep you posted.
So i played around with it and managed to get it to work but i do have a few questions about the possibility of cleaning up some things cause it does not feel ācleanā enough. Let me me copy past the code below. Again the intend is a encrypt an attribute called file name when upserting and decrypt it when reading.
First the reading part (decrypting):
module Decryption
def self.decrypt(source_hash, decryption_attributes)
# here take encrypted_file_name out of the source_hash and decrypt it to decrypted_file_name
decrypted_file_name = decrypt(source_hash[:encrypted_file_name])
source_hash.merge({
file_name: decrypted_file_name
})
end
end
module Types
include Dry.Types()
end
module ReadModels
class DecryptedFileRecord < Dry::Struct
attribute :key, ::Types::String
attribute :file_name, ::Types::String
end
class DecryptionMapper < ROM::Transformer
relation :files, as: :decryption_mapper
import :decrypt, from: Decryption, as: :decrypt
map do
decrypt :file_name
constructor_inject DecryptedFileRecord
end
end
end
As you can see i am using the mapper like you pointed out. I had to look into Dry::Transformer on how to write my own mapping methods (it seems transproc has been migrated to dry transformer). Now in my file repo i do this to be able to get all of by key, which is working fine:
class FileRepo < ROM::Repository[:files]
auto_struct false
def all
decrypted.to_a
end
def by_key(key)
decrypted.by_pk(key).one!
end
def decrypted
files.map_with(:decryption_mapper)
end
end
Now for upserting new data i kinda do the same with the changeset approach:
module Encryption
def self.encrypt(data, encryption_attributes = {})
encrypted_file_name, encrypted_file_name_iv = encrypt(data.delete(:file_name))
data.merge(
encrypted_file_name: encrypted_file_name,
encrypted_file_name_iv:encrypted_file_name_iv
)
end
end
class FileChangeset < ROM::Changeset::Create
map do |tuple|
Encryption.encrypt(tuple)
end
end
end
class UpsertFile < ROM::SQL::Commands::Postgres::Upsert
relation :files
register_as :create_or_update
result :one
constraint :files_pkey
update_statement encrypted_file_name: Sequel.qualify(:excluded, :encrypted_file_name),
encrypted_file_name_iv: Sequel.qualify(:excluded, :encrypted_file_name_iv)
end
class FileRepo < ROM::Repository[:files]
auto_struct false
# previous other methods ...
def rebuild(key)
# get data from my event store
file_data = project_for(key)
file_data = files.changeset(FileChangeset, file_data).to_h
files.command(:create_or_update).call(file_data)
end
end
So when i do FileRepo.new(DB).rebuild(āa_keyā) it works fine.
Now my questions:
I was not able to use the files.changeset(:upsert, data).commit like you mentioned. So i my case i tried files.changeset(:create_or_update, file_data).commit cause that was the name i named my command but then it errors with ArgumentError: +:create_or_update+ is not a valid changeset type. Must be one of: [:create, :update, :delete]. I also tried to use create instead and leave the command_type :create_or_update but that threw another error that the column value for encrypted_file_name was null so i guess the changeset did not went to the mapping at all. If you could explain me what i am doing wrong here that would be nice so i can just use the commit of the changeset before calling to_h and manually calling the command.=> after realizing i need to give my custom changeset as the argument to changeset it worked eg: files.changeset(FileChangeset, file_data).commit , then the only thing i had todo is to match the command_type in the changeset with the register_as in the command
Like you see in the decryption mapper (import :decrypt ā¦) i use the import statement on the mapper to add my own mapper function. I expect that i could do the same on the Changeset but it seems that is not possible and the map function in the changeset only has a limited set of transform functions (hash only it seems). Is this by design and could it not be interesting that both would work the same way? Or am i missing something? Cause it would be nice if i could do import :encrypt, from: Encrypt, as: :encrypt and cleanup the code by doing the same as with the decryption and do=> see below for explanation but i managed to solve it by importing my transform function on the PipeRegistry
class FileChangeset < ROM::Changeset::Create
import :encrypt, from: Encryption, as: :encrypt
map do |tuple|
encrypt :file_name
end
end
In waiting for a reply i discovered two potential routes I could take to implement the thing that is working in a nicer way but would like to have some feedback first cause i am not to familiar with the rom way of working:
I see that a changeset can except command options where i would be able to pass in a mapper using command_options on the changeset class. But it seems that is only applied on the result and i would not be able to use that to do the encryption part but only the decryption after i have send the upsert to the database?=> figured this out it was indeed only for the result of the upsert command
I see that a changeset can use plugins as well and maybe that is the way for me to do the encryption part on the changeset level? Can i use plugins for a custom mapper inheriting from ROM::Transformer?=> looking from the code i could use plugin on either the changeset or command level to be able to hide the map part in the changeset but since it is similar to the one in the mapper i would keep the map for now for readability
It seems to me that it would still be nice to allow to import custom tranformations functions on the changeset level (with import) like we can do with ROM::Transformer. I see the docs at https://rom-rb.org/learn/changeset/5.2/mapping/ saying that it āsupport all transformation functions from transprocā and some additions exist.=> see my reply below, by adding it to the pipeline i am able to use it, but still would be nice to have it in the same way as the transformer/mapper but can live with it for now
Managed to get the encrypt function working inside the map of the changeset by import it inside the Changeset::PipeRegistry. Seems a bit hacky but works
module ROM
class Changeset
module PipeRegistry
import :encrypt, from: Encryption, as: :encrypt
end
end
end
class FileChangeset < ROM::Changeset::Create
map do
encrypt :file_name
end
end
So i donāt really need any feedback anymore cause i have a working system. Would still like to now if it would make sense to allow the ability of importing custom transform functions directly on a changeset.
First of all thanks for sharing all of this, itās really helpful
Yeah so THE PLAN is to expose a nice API for adding custom mapping functions. This will be available in rom 6.0.0 later this year. If you have some API ideas, please let me know!
Iām glad that you figured things out on your own We can and we will make things like that simpler in the near future.
Glad this is planned and will definitely watch out for it
I always hate it when i find tickets or questions in either github or other systems that are designed for support and people explaining their issue but actually never share the solution so thatās why i tend to be verbose about what i am doing.
I think an interesting approach to help explain all the powerful features in rom is to have more examples by use cases (like the encryption on that i have, or for example how ruby event store uses rom) that demonstrate all the possibilities of rom. I did understand the concepts (changeset, mappers, repo, ā¦) by looking at the docs on the website but how all the things work together was a bit of a challenge
Anyways i hope this thread can help others that might have similar questions.
Yes this has been a great challenge. Thereās a āGuidesā section on the website which is meant to include āuse casesā explaining how to do something specific using rom. Unfortunately nobody has found the time to actually fill it with content Weāll get there eventually though
Weāll get there eventually though 