In which I successfully debug associations which wouldn't set foreign keys, and ponder documentation improvements and sanity checks

I had a problem with my project. I didn’t want help fixing it, though, I wanted understanding how to debug it and fix it myself because I anticipated that I would encounter similar problems in the future.

Herein follows a tale of woe. Interested parties may be interested in following it to understand how someone not familiar with the software approaches the software. Other interested parties trying to figure out associations might also benefit. After the tale of woe I comment on how documentation could be improved to help others avoid this trouble.

… This material will be split as “new users can only put two links in a post”. Curse you, Discourse. All my links are to Github under the rom-rb project namespace.

Tale of Woe

I have a dry-web-roda template application, and I’m playing around with some domain-driven design, so my application itsuki has a couple of tables like bookings and booking_requests. The idea is that the booking itself has a top-level object, and various requests are made to put it in a certain state, like the user making the booking requesting certain dates and the administrator user approving them.

I’ve gotten migrations all sorted, the SQL tables look the way I want them to (for the moment), I’ve got a couple of relation classes describing them and setting defaults on fields like created_at, and now I’m trying to write an operation that represents the initial request, where a user submits a form and the application creates a booking with a booking_request associated with it.

My operation class looks somewhat like this:

     booking_repo.transaction do
        booking = bookings_repo.new_booking
        STDERR.puts booking.inspect

        booking_request = booking_requests_repo.changeset(
          message: params['message'],
          user_initiated: true,
          state: 0,
        )
        STDERR.puts booking_request.inspect

        assoc = booking_request.associate(booking, :booking)
        STDERR.puts assoc.inspect

        assoc.commit
      end

When this code runs, I get output something like so:

#<ROM::Struct::Booking id=8 active=true reference="B-2017-6DLD8X" created_at=2017-07-23 13:34:02 +0100>
#<ROM::Changeset::Create relation=ROM::Relation::Name(booking_requests) data={:message=>"", :user_initiated=>true, :state=>0}>
#<ROM::Changeset::Associated:0x007faddef44168 @__options__={:left=>#<ROM::Changeset::Create relation=ROM::Relation::Name(booking_requests) data={:message=>"", :user_initiated=>true, :state=>0}>, :associations=>{:booking=>#<ROM::Struct::Booking id=8 active=true reference="B-2017-6DLD8X" created_at=2017-07-23 13:34:02 +0100>}}, @left=#<ROM::Changeset::Create relation=ROM::Relation::Name(booking_requests) data={:message=>"", :user_initiated=>true, :state=>0}>, @associations={:booking=>#<ROM::Struct::Booking id=8 active=true reference="B-2017-6DLD8X" created_at=2017-07-23 13:34:02 +0100>}>
ROM::SQL::NotNullConstraintError: PG::NotNullViolation: ERROR:  null value in column "booking_id" violates not-null constraint
DETAIL:  Failing row contains (5, null, 2017-07-23 12:34:02.412665, t, null, USD, 0, ).

In short, the foreign key booking_id has not been set on the booking_request object.

Code-reading suggests that what’s going on here is the Create changeset is having #with_association() called with the booking request association as a parameter, and then .call is being called on the result. I’m not 100% sure which case of with_association is being used here, but I can drop in some STDERR.puts debugging in the gem to help me figure things out. In rom-sql land I find def associate and do some utterly deplorable in-place gem modification to STDERR.puts the contents of output_tuples:

[{:message=>"", :user_initiated=>true, :state=>0, :booking_id=>13}]

Huh, so it IS finding the booking. That’s good, at least. Unhack my code. What’s going on here? Let’s go back to rom-repostitory for Rom::Changeset::Associated#command and print out what it’s generating inside that reduce loop:

#<ROM::SQL::Commands::Create[:booking_requests]relation=#<Persistence::Relations::BookingRequests name=booking_requests dataset=#<Sequel::Postgres::Dataset: "SELECT \"id\", \"message\", \"created_at\", \"user_initiated\", \"state\" FROM \"booking_requests\" ORDER BY \"booking_requests\".\"id\"">> options={:relation=>#<Persistence::Relations::BookingRequests name=booking_requests dataset=#<Sequel::Postgres::Dataset: "SELECT \"id\", \"message\", \"created_at\", \"user_initiated\", \"state\" FROM \"booking_requests\" ORDER BY \"booking_requests\".\"id\"">>, :source=>#<Persistence::Relations::BookingRequests name=booking_requests dataset=#<Sequel::Postgres::Dataset: "SELECT \"id\", \"message\", \"created_at\", \"user_initiated\", \"state\" FROM \"booking_requests\" ORDER BY \"booking_requests\".\"id\"">>, :result=>:one, :input=>#<Dry::Types::Constructor type=#<Dry::Types::Hash::Schema primitive=Hash options={:member_types=>{:id=>#<ROM::SQL::Attribute[Integer] primary_key=true name=:id source=ROM::Relation::Name(booking_requests)>, :message=>#<ROM::SQL::Attribute[String] name=:message source=ROM::Relation::Name(booking_requests)>, :created_at=>#<ROM::SQL::Attribute[DateTime] name=:created_at source=ROM::Relation::Name(booking_requests)>, :user_initiated=>#<ROM::SQL::Attribute[TrueClass | FalseClass] name=:user_initiated source=ROM::Relation::Name(booking_requests)>, :state=>#<ROM::SQL::Attribute[Integer] name=:state source=ROM::Relation::Name(booking_requests)>}} meta={}>>, :curry_args=>[#<ROM::Changeset::Create relation=ROM::Relation::Name(booking_requests) data={:message=>"", :user_initiated=>true, :state=>0}>], :before=>[{:associate=>{:assoc=>#<ROM::SQL::Association::ManyToOne source=ROM::SQL::Association::Name(booking_requests) target=ROM::SQL::Association::Name(bookings as booking) result=:one>, :keys=>{#<ROM::SQL::QualifiedAttribute dataset=:booking_requests attribute=:booking_id>=>#<ROM::SQL::QualifiedAttribute dataset=:bookings attribute=:id>}, :parent=>#<ROM::Struct::Booking id=16 active=true reference="B-2017-AH2YAU" created_at=2017-07-23 14:17:06 +0100>}}], :after=>[:finalize], :associations=>{:booking=>{:parent=>#<ROM::Struct::Booking id=16 active=true reference="B-2017-AH2YAU" created_at=2017-07-23 14:17:06 +0100>}}, :configured_associations=>[:booking]}>

Oh lord, that’s a lot to wade through, and the relevant part appears to be nearish the end:

:before=>[{
  :associate=>{
    :assoc => #<ROM::SQL::Association::ManyToOne
      source=ROM::SQL::Association::Name(booking_requests)
      target=ROM::SQL::Association::Name(bookings as booking)
      result=:one>,
    :keys=>{
      #<ROM::SQL::QualifiedAttribute
         dataset=:booking_requests
         attribute=:booking_id>
        => #<ROM::SQL::QualifiedAttribute
             dataset=:bookings
             attribute=:id>
     }, # end keys
     :parent => #<ROM::Struct::Booking
        id=16
        active=true
        reference="B-2017-AH2YAU"
        created_at=2017-07-23 14:17:06 +0100>
      }# end associate }] # end before

This is a mess but suggests to me that the create command is supposed to do something called associate, and run this before its main execution, connecting booking_requests.booking_id with bookings.id. Gee, it sure looks like everything I ever expected it to. What the heck is going wrong?

(I’m replying too quickly. Please wait 12 seconds. Curse you, Discourse. I will hate you forever.)

Discourse tells me, “Consider replying to several posts at once”. No, Discourse. You’re missing the point, and I still hate you.

I suppose I can look into the Create command. Apparently it inherits from ROM::Commands::Create which must be where the :before stuff is handled. No, wait, that’s a boring class, Rom::Command must be where it’s at. Sure enough, that’s it, in command.call, which appears to have generic hook-application framework in apply_hooks:

    def apply_hooks(hooks, tuples, *args)
      hooks.reduce(tuples) do |a, e|
        if e.is_a?(Hash)
          hook_meth, hook_args = e.to_a.flatten(1)
          __send__(hook_meth, a, *args, **hook_args)
        else
          __send__(e, a, *args)
        end
      end
    end

Ugh, what’s going on here? I guess a is set to some tuples and e.is_a?(Hash) so we’re going to associate(bunch-o-tuples, {assoc: keys: parent: ...})? Great, where on the command is this method defined? … must be over here in rom/sql/plugin/associates which … is where we were already, and saw meaningful output.

Discourse has apparently decided that I am no longer allowed to post links to GITHUB. Curse you Discourse. Broken links are attached below.

But what CONSUMES this output? apply_hooks … ah, it must be returning the result of the reduce operation and then assigning it to prepared (i’m not used to assignment from if/else like that but it works). So it must be somewhere in execute that things go wrong. Well, let’s hack a trace in there. Back in rom-sql land: ht tps://github. com/rom-rb/rom-sql/blob/v1.3.3/lib/rom/sql/commands/create.rb - we can omghax Create.execute.

I have the right tuples still, right?
{:message=>"", :user_initiated=>true, :state=>0, :booking_id=>17}

Sweet. And insert_tuples?
[{:message=>"", :created_at=>#<DateTime: 2017-07-23T15:08:10+01:00 ((2457958j,50890s,752423000n),+3600s,2299161j)>, :user_initiated=>true, :state=>0}]

AHA! Caught you. So what the heck is input[tuple] and why is it returning the wrong thing? … is it THIS? https:// github . com/rom-rb/rom-sql/blob/7630d0819a791ff80ca356791057a2a625ff0354/lib/rom/sql/type_dsl.rb#L21 ? okay… sure, so, then, what’s this constructor about?

#<Dry::Types::Constructor 
type=#<Dry::Types::Hash::Schema 
primitive=Hash
options={
  :member_types=>{
  :id=>#<ROM::SQL::Attribute[Integer] primary_key=true name=:id source=ROM::Relation::Name(booking_requests)>, 
  :message=>#<ROM::SQL::Attribute[String] name=:message source=ROM::Relation::Name(booking_requests)>,
  :created_at=>#<ROM::SQL::Attribute[DateTime] name=:created_at source=ROM::Relation::Name(booking_requests)>,
  :user_initiated=>#<ROM::SQL::Attribute[TrueClass | FalseClass] name=:user_initiated source=ROM::Relation::Name(booking_requests)>,
  :state=>#<ROM::SQL::Attribute[Integer] name=:state source=ROM::Relation::Name(booking_requests)>
}} meta={}>>

Well, that doesn’t look like it has a booking_id. Progress!

Going to bet that this comes from where I do

    class BookingRequests < ROM::Relation[:sql]
      schema(:booking_requests) do 

You know, I have this as part of the schema:

        associations do
          belongs_to :booking, foreign_key: :booking_id
        end

But that’s not working, so, why don’t I just try something where I define the field as well as the attribute?

        attribute :booking_id, Types::Strict::Int
        associations do
          belongs_to :booking, foreign_key: :booking_id
        end

OH THANK GOD EVERYTHING WORKS NOW.

(I’m still “replying too quickly” and must wait 18 seconds to continue my story. Discourse, I wish for you to experience utter commercial failure and be replaced by a far worthier successor.)

Why wasn’t this obvious?

So let’s go back and look at the documents and see why this wasn’t obvious… What does Changeset Transactions say in its example?

  conf.relation(:users) do
    schema(infer: true) do
      associations do
        has_many :tasks
      end
    end
  end

  conf.relation(:tasks) do
    schema(infer: true) do
      associations do
        belongs_to :user
      end
    end
  end

Ah haha, it must be inferring everything automatically.

And what of the Associations page? Yep, everything is infer: true there as well.

Thoughts

Documentation Proposal

While infer: true is handy for examples, I would propose that we should include notes and an example with an explicit schema, instead of an inferred schema, because this pitfall must be quite easy to fall into when coming from an ActiveRecord codebase where the example code looks more like this:

class Customer < ActiveRecord::Base
  has_many :orders, :dependent => :destroy
end

class Order < ActiveRecord::Base
  belongs_to :customer
end

without any explicit definition of the field.

Sanity checking

Is there any case where it makes sense for there to be an association without a field for that association to be stored in? Would it make sense to introduce sanity checking for this to run at some point of the relation definition process?