How to use Mapper with Changeset?

Issues
1

My database looks nothing like my entities, and so my setup is fairly more complex than normal, and I am ending up needing to customize seemingly every layer.

I am able to create data, but part of the default insert is doing a query to return the inserted data, and when that happens it is not making use of my configured mapper, and since the columns don’t match it blows up. I’ve created a basic example showing the problem.

# !/usr/bin/env ruby

# This is an example of a complete ROM setup in one file.
# Useful for reproducing problems and posting to StackOverflow, forums, bug trackers, etc.
require 'bundler/inline'

gemfile do
  source 'https://rubygems.org'
  gem 'rom'
  gem 'rom-sql'
  gem 'sqlite3'
end

require "rom/transformer"

module Marv
  module Entities
    Types = ROM::Types
    class Base < ROM::Struct
      transform_keys(&:to_sym)
    end

    class Watch < Base
      attribute(:id, Types::Integer)
      attribute(:name, Types::String)
      attribute(:created, Types::JSON::DateTime)
      attribute(:updated, Types::JSON::DateTime)
    end
  end

  module SqlRelations
    Types = ROM::SQL::Types
    class Base < ROM::Relation[:sql]
      struct_namespace ::Marv::Entities
      auto_struct false
    end

    class Watches < Base
      schema(:watches) do
        attribute(:id, Types::Serial)
        attribute(:watch_type, Types::String)
        attribute(:created_at, Types::DateTime)
        attribute(:updated_at, Types::DateTime)
      end
    end
  end

  # This mapper never gets used.  How can I force it?
  class WatchMapper < ROM::Transformer
    relation :watches, as: :watch_mapper
    map do
      symbolize_keys
      rename_keys({
        watch_type: :name,
        created_at: :created,
        updated_at: :updated,
      })
    end
  end

  class NewWatchChangeset < ROM::Changeset::Create
    map do
      symbolize_keys
      rename_keys({
        name: :watch_type,
        created: :created_at,
        updated: :updated_at,
      })
    end
  end

  class WatchRepo < ROM::Repository[:watches]
    # by default always map with watch mapper to return Watch objects
    # Doesn't work yet with a Root repo as super. See:
    # https://rom-rb.zulipchat.com/#narrow/stream/191800-general/topic/repository.20default.20mapper
    # But also, doesn't appear to work at all, period.
    def root
      super.map_with(:watch_mapper)
    end
    struct_namespace Marv::Entities
    commands :create, update: :by_pk, delete: :by_pk
  end

  CONT = ROM.container(:sql, 'sqlite::memory') do |conf|
    conf.default.create_table(:watches) do
      primary_key :id
      column :watch_type, String
      column :created_at, Time
      column :updated_at, Time
    end

    conf.register_relation(Marv::SqlRelations::Watches)
    conf.register_mapper(Marv::WatchMapper)
  end
end

WATCH_REPO = Marv::WatchRepo.new(Marv::CONT)
model = WATCH_REPO.watches.mapper.model
queried = WATCH_REPO.watches.to_a
now = Time.now
watch_data = { name: "i-am-a-puppy", created: now, updated: now }

puts <<HEAD
model: #{model}

schema: #{model.schema}

watch_data: #{watch_data}

queried: #{queried}
HEAD

changeset = WATCH_REPO.watches.changeset(Marv::NewWatchChangeset, watch_data)

puts <<HEAD
changeset: #{changeset.to_h}
HEAD

# Explodes regardless of which type of commit/create is run:
# result = changeset.commit
result = WATCH_REPO.create(changeset)
# And `map_with` isn't available on either WATCH_REPO or changeset.

# Separately, there are no examples of how `map_to` is expected to work in the real world... do I need it?
queried = WATCH_REPO
            .watches
            .map_to(Marv::Entities::Watch)
            .map_with(:watch_mapper)
            .to_a

puts <<HEAD
result: #{result}

queried: #{queried}
HEAD

The output with error is:

$ ruby example.rb
model: Marv::Entities::Watch

schema: #<Dry::Types[Constructor<Schema<key_fn=.to_sym keys={id: Nominal<Integer meta={primary_key: true, alias: nil, source: :watches}> name: Nominal<String> created: Constructor<Nominal<DateTime> fn=Dry::Types::Coercions::JSON.to_date_time> updated: Constructor<Nominal<DateTime> fn=Dry::Types::Coercions::JSON.to_date_time> watch_type: Nominal<String meta={alias: nil, source: :watches}> created_at: Nominal<DateTime meta={alias: nil, source: :watches}> updated_at: Nominal<DateTime meta={alias: nil, source: :watches}>}> fn=Kernel.Hash>]>

watch_data: {:name=>"i-am-a-puppy", :created=>2023-07-06 10:28:33.600305 -0700, :updated=>2023-07-06 10:28:33.600305 -0700}

queried: []
changeset: {:watch_type=>"i-am-a-puppy", :created_at=>2023-07-06 10:28:33.600305 -0700, :updated_at=>2023-07-06 10:28:33.600305 -0700}
Traceback (most recent call last):
        30: from example.rb:122:in `<main>'
        29: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-repository-5.3.0/lib/rom/repository/class_interface.rb:128:in `block in define_command_method'
        28: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-changeset-5.3.0/lib/rom/changeset/stateful.rb:204:in `commit'
        27: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-core-5.3.0/lib/rom/commands/composite.rb:19:in `call'
        26: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-sql-3.6.1/lib/rom/sql/commands/error_wrapper.rb:18:in `call'
        25: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-core-5.3.0/lib/rom/command.rb:278:in `call'
        24: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-sql-3.6.1/lib/rom/sql/commands/create.rb:33:in `execute'
        23: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-sql-3.6.1/lib/rom/sql/commands/create.rb:49:in `insert'
        22: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-core-5.3.0/lib/rom/relation.rb:361:in `to_a'
        21: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-core-5.3.0/lib/rom/relation.rb:361:in `to_a'
        20: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-core-5.3.0/lib/rom/relation.rb:361:in `each'
        19: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-core-5.3.0/lib/rom/relation.rb:221:in `each'
        18: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-core-5.3.0/lib/rom/mapper.rb:97:in `call'
        17: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-core-5.3.0/lib/rom/mapper.rb:97:in `reduce'
        16: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-core-5.3.0/lib/rom/mapper.rb:97:in `each'
        15: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-core-5.3.0/lib/rom/mapper.rb:97:in `block in call'
        14: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/transproc-1.1.1/lib/transproc/function.rb:49:in `call'
        13: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/transproc-1.1.1/lib/transproc/function.rb:49:in `call'
        12: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/transproc-1.1.1/lib/transproc/array.rb:44:in `map_array'
        11: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/transproc-1.1.1/lib/transproc/array.rb:44:in `map'
        10: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/transproc-1.1.1/lib/transproc/array.rb:44:in `block in map_array'
         9: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/transproc-1.1.1/lib/transproc/function.rb:49:in `call'
         8: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/transproc-1.1.1/lib/transproc/function.rb:49:in `call'
         7: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/transproc-1.1.1/lib/transproc/class.rb:32:in `constructor_inject'
         6: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/dry-struct-1.6.0/lib/dry/struct/class_interface.rb:264:in `new'
         5: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/dry-types-1.7.1/lib/dry/types/constructor.rb:81:in `call_unsafe'
         4: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/dry-types-1.7.1/lib/dry/types/schema.rb:60:in `call_unsafe'
         3: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/dry-types-1.7.1/lib/dry/types/schema.rb:341:in `resolve_unsafe'
         2: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/dry-types-1.7.1/lib/dry/types/schema.rb:377:in `resolve_missing_keys'
         1: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/dry-types-1.7.1/lib/dry/types/schema.rb:377:in `each'
/Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/dry-types-1.7.1/lib/dry/types/schema.rb:386:in `block in resolve_missing_keys': :name is missing in Hash input (Dry::Types::MissingKeyError)
        30: from example.rb:122:in `<main>'
        29: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-repository-5.3.0/lib/rom/repository/class_interface.rb:128:in `block in define_command_method'
        28: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-changeset-5.3.0/lib/rom/changeset/stateful.rb:204:in `commit'
        27: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-core-5.3.0/lib/rom/commands/composite.rb:19:in `call'
        26: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-sql-3.6.1/lib/rom/sql/commands/error_wrapper.rb:18:in `call'
        25: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-core-5.3.0/lib/rom/command.rb:278:in `call'
        24: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-sql-3.6.1/lib/rom/sql/commands/create.rb:33:in `execute'
        23: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-sql-3.6.1/lib/rom/sql/commands/create.rb:49:in `insert'
        22: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-core-5.3.0/lib/rom/relation.rb:361:in `to_a'
        21: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-core-5.3.0/lib/rom/relation.rb:361:in `to_a'
        20: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-core-5.3.0/lib/rom/relation.rb:361:in `each'
        19: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-core-5.3.0/lib/rom/relation.rb:221:in `each'
        18: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-core-5.3.0/lib/rom/mapper.rb:97:in `call'
        17: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-core-5.3.0/lib/rom/mapper.rb:97:in `reduce'
        16: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-core-5.3.0/lib/rom/mapper.rb:97:in `each'
        15: from /Users/pboling/.asdf/installs/ruby/2.7.8/lib/ruby/gems/2.7.0/gems/rom-core-5.3.0/lib/rom/mapper.rb:97:in `block in call'

I am so lost. :frowning:

2

It seems that my problem stems from here, in create.rb:

        def insert(tuples)
          pks = tuples.map { |tuple| relation.insert(tuple) }
          relation.where(relation.primary_key => pks).to_a
        end

which eventually gets here, in mapper.rb:

    def call(relation)
      transformers.reduce(relation.to_a) { |a, e| e.call(a) }
    end

And the mapper isn’t my mapper (though it appears it should be?) and thus my transformer isn’t being used, even though that is the only way it could possibly work.

3

This works:

# !/usr/bin/env ruby

# This is an example of a complete ROM setup in one file.
# Useful for reproducing problems and posting to StackOverflow, forums, bug trackers, etc.
require 'bundler/inline'

gemfile do
  source 'https://rubygems.org'
  gem 'byebug'
  gem 'rom', '~> 5.2'
  gem 'rom-sql', '~> 3.0'
  gem 'sqlite3'
end

require "rom/transformer"

module Marv
  module Entities
    Types = ROM::Types

    class Watch < ROM::Struct
      transform_keys(&:to_sym)

      attribute(:id, Types::Integer)
      attribute(:name, Types::String)
      attribute(:created, Types::JSON::DateTime)
      attribute(:updated, Types::JSON::DateTime)
    end
  end

  module SqlRelations
    class Watches < ROM::Relation[:sql]
      auto_struct false

      schema(:watches) do
        attribute(:id, Types::Serial)
        attribute(:watch_type, Types::String)
        attribute(:created_at, Types::DateTime)
        attribute(:updated_at, Types::DateTime)
      end
    end
  end

  # This mapper never gets used.  How can I force it?
  class WatchMapper < ROM::Transformer
    relation :watches

    map do
      symbolize_keys

      rename_keys({
        watch_type: :name,
        created_at: :created,
        updated_at: :updated,
      })
    end
  end

  class NewWatchChangeset < ROM::Changeset::Create
    map do
      symbolize_keys

      rename_keys({
        name: :watch_type,
        created: :created_at,
        updated: :updated_at,
      })
    end
  end

  class WatchRepo < ROM::Repository[:watches]
  end

  CONT = ROM.container(:sql, 'sqlite::memory') do |conf|
    conf.default.create_table(:watches) do
      primary_key :id
      column :watch_type, String
      column :created_at, Time
      column :updated_at, Time
    end

    conf.register_relation(Marv::SqlRelations::Watches)
    conf.register_mapper(Marv::WatchMapper)
  end
end

WATCH_REPO = Marv::WatchRepo.new(Marv::CONT)

now = Time.now
watch_data = { name: "i-am-a-puppy", created: now, updated: now }

changeset = WATCH_REPO.watches.changeset(Marv::NewWatchChangeset, watch_data)
result = changeset.commit

puts result.inspect

queried = WATCH_REPO.watches.to_a

puts queried.inspect
1 Like
4

In general, I would recommend using database to create views with renamed columns. This would simplify your ROM setup quite a bit.

1 Like
5

Thanks for the working example! i’m going to diff it with what I had now.

The mapping in SQL could possibly be done by a SQL expert, but it is far more than simple column name changes. I don’t even know how to do the complex mappings in ROM yet, but I know Ruby, so I hope I can figure it out. We are taking a simple table which was abusing a JSON column and exploding it into a very complex hierarchy of discrete data structures and adding lots of new entities. (It will continue to abuse the JSON column under the hood until all internal clients are upgraded to use the new entities from the new API backed by ROM, at which point we can finally change the data structure in the database.)

This example code was hyper-simplified to not distract from the most important thing I needed to solve. i’ll have a lot more posts here I think as I wade through.

In trying to figure this out I did rewrite the example code using ROM 6.0.0.alpha1 & friends, and I got confused there as well. As it is unfinished I’ll just stick to ROM 5.3 for now.

6

After comparing it looks like I won’t be able to use the Base classes I was hoping to. Not the end of the world, and I understand that’s something that will change somewhat with v6. Is that correct?

7

The result of your version does not use my entities in the returned objects. Instead it uses the generic ROM::Struct. That’s fine, I suppose, so long as I can turn what gets returned into the right shape. I am not interested at all in the DB shape.

#<ROM::Struct::Watch id=1 watch_type="i-am-a-puppy" created_at=2023-07-07 09:46:45.492007 -0700 updated_at=2023-07-07 09:46:45.492007 -0700>
[#<ROM::Struct::Watch id=1 watch_type="i-am-a-puppy" created_at=2023-07-07 09:46:45.492007 -0700 updated_at=2023-07-07 09:46:45.492007 -0700>]

Is there a way to automatically have it map to my Entity? If not is there a proper way to map to the entity from the ROM::Struct?
The goal of what I am working on is to essentially hide the database, and fully disconnect from it past a certain layer of code, so that we can retool against the higher level code using the right shapes, and then eventually fix the database. This is in a zero downtime system, so it is impossible to turn off the system to upgrade the code and database together. The upgrades must roll out in steps.

8

Adding back the struct_namespace results in the original error coming back.

Thinking about this some more, and I realize that your solution isn’t using the Mapper at all. I suppose the original question still remains then. How can we use a Changeset with a Mapper?

9

The objects being returned do not have the shape that would result from the mapper.

10

It is now clear that this is the exact same issue as discussed in the other thread. Seems there is no simple solution currently.

11

They way to string together a Changeset with a Mapper and end up with a custom Entity is like this:

# !/usr/bin/env ruby

# This is an example of a complete ROM setup in one file.
# Useful for reproducing problems and posting to StackOverflow, forums, bug trackers, etc.
require 'bundler/inline'

gemfile do
  source 'https://rubygems.org'
  gem 'rom', '~> 5.2'
  gem 'rom-sql', '~> 3.0'
  gem 'sqlite3'
end

require "rom/transformer"

module Marv
  module Entities
    Types = ROM::Types

    class Watch < ROM::Struct
      transform_keys(&:to_sym)

      attribute(:id, Types::Integer)
      attribute(:name, Types::String)
      attribute(:created, Types::DateTime)
      attribute(:updated, Types::DateTime)
    end
  end

  module SqlRelations
    class Watches < ROM::Relation[:sql]
      # NOTE: This has no effect at all.
      # struct_namespace Marv::Entities
      auto_struct false

      schema(:watches) do
        attribute(:id, Types::Serial)
        attribute(:watch_type, Types::String)
        attribute(:created_at, Types::DateTime)
        attribute(:updated_at, Types::DateTime)
      end
    end
  end

  # This mapper never gets used.  How can I force it?
  class WatchMapper < ROM::Transformer
    relation :watches, as: :watch_mapper

    map do
      symbolize_keys

      rename_keys({
        watch_type: :name,
        created_at: :created,
        updated_at: :updated,
      })
    end
  end

  class NewWatchChangeset < ROM::Changeset::Create
    map do
      symbolize_keys

      rename_keys({
        name: :watch_type,
        created: :created_at,
        updated: :updated_at,
      })
    end
  end

  class WatchRepo < ROM::Repository[:watches]
    # NOTE: This will break everything if set!
    #   Changesets are unaware of custom Mappers.
    #   Thus, Changeset is incompatible with struct_namespace for custom objects,
    #     if they need a mapper to transform the structure.
    # struct_namespace Marv::Entities
  end

  CONT = ROM.container(:sql, 'sqlite::memory') do |conf|
    conf.default.create_table(:watches) do
      primary_key :id
      column :watch_type, String
      column :created_at, Time
      column :updated_at, Time
    end

    conf.register_relation(Marv::SqlRelations::Watches)
    conf.register_mapper(Marv::WatchMapper)
  end
end

WATCH_REPO = Marv::WatchRepo.new(Marv::CONT)

now = Time.now
watch_data = { name: "i-am-a-puppy", created: now, updated: now }

changeset = WATCH_REPO.watches.changeset(Marv::NewWatchChangeset, watch_data)
result = changeset.commit

puts result.inspect

queried = WATCH_REPO.watches.to_a

puts queried.inspect

query_mapped = WATCH_REPO.watches.by_pk(1).map_with(:watch_mapper).map_to(Marv::Entities::Watch).to_a

puts query_mapped.inspect
# => [#<Marv::Entities::Watch id=1 name="i-am-a-puppy" created=2023-07-07 11:41:06.764457 -0700 updated=2023-07-07 11:41:06.764457 -0700>]
12

@pboling That’s what I’ve done, custom Changeset classes. I’m pulling data from a number of APIs, and updating local db records which look nothing like the incoming data. I’m not using any of the predefined mapping methods. Instead I use map do |data| within the changeset, and occasionally refer to the original object (which is pretty cool, though I haven’t used it with a multi-record update yet, not sure how that would work).

class CustomChangeset < ROM::Changeset::Update[:line_items]
  map do |data|
    out = {}
    out[:first_column]   = data[:whatever]
    out[:second_column]  = data[:something_else] if original[:second_column] ! ''
    out
  end
end

line_items.by_pk(id).changeset(CustomChangeset, data).commit

I don’t know if I’m using this feature correctly, but so far it works well. One thing I haven’t figured out yet is why the custom changeset map method gets called twice. I think it has something to do with the diff’ing process. It doesn’t seem to be a problem, except when I have side-effects that need to trigger from the changeset.

2 Likes