Skip to content

Refactor of COPY code #2509

Open
Open

Description

@joto

The code in src/db-copy-mgr.hpp and src/db-copy.[ch]pp could do with some refactoring. Here are the reasons:

  • The db_copy_mgr_t class mixes two things: It handles the formatting of the data as used in the COPY command.
    And it handles the actual copy, the commit/rollback, and the interfacing with the processor. You can see the
    split when looking at where the m_processor member ist used. This mixture makes it hard to unit test the
    COPY format code.
    This formatting part is extended in the new binary COPY format code in Use binary COPY for flex and middle tables #2503 making this even more awkward.
  • Making all of this dependend on the DELETER type doesn't look right to me. It seems overkill to use a
    template parameter for the DELETER code. The db_deleter_by_id_t and db_deleter_by_type_and_id_t classes
    really differ only in the delete_rows() function. Storing the type for each entry in m_deletables
    (in item_t) isn't so expensive that we couldn't jst do it in both cases. So maybe all of this should
    just be a flag somewhere to choose the right code in delete_rows().
  • In parallelize the COPY phase #2426 it was proposed to run multiple COPYs in parallel. If we split the code doing the actual COPY from
    the code that prepares the COPY data, this would be easier to implement.

I am not sure how all of this can look like, it needs some experimenting, but here are some ideas:

  • Refactoring the DELETER code would make db_copy_mgr_t::m_current not depend on a template parameter any more
  • This opens up the possiblity to move the COPY formatting code out of db_copy_mgr_t into its own class that
    is then only used in db_copy_mgr_t.
  • Then db_copy_mgr_t can be redesigned to work with parallel COPYs.

These could hopefully be done in several small steps.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions