Skip to content

skip gforce() setup when j is only .N - #7920

Merged
ben-schwen merged 3 commits into
masterfrom
gforce-dotN
Oct 19, 2026
Merged

ben-schwen merged 3 commits into
masterfrom
gforce-dotN

Conversation

@m-muecke

Copy link
Copy Markdown
Contributor

On small tables data.table seems to be a bit slower for DT[, .N, by = g] than dplyr. AFAICT, this is due to overhead from gforce() that is not needed. An alternative approach would've been returning a GForce_N element in the output list of .optimize_gforce() and checking with that.

master

library(data.table, warn.conflicts = FALSE)

set.seed(1)
n = 1e7
bench::press(
  groups = c(10, 1e6),
  {
    DT = data.table(g = sample.int(groups, n, TRUE), v = runif(n))
    df = as.data.frame(DT)
    bench::mark(
      data.table = DT[, .N, by = g],
      dplyr = dplyr::count(df, g),
      check = FALSE,
      iterations = 20,
      filter_gc = FALSE
    )
  }
)
#> Running with:
#>    groups
#> 1      10
#> 2 1000000
#> # A tibble: 4 × 7
#>   expression  groups      min   median `itr/sec` mem_alloc `gc/sec`
#>   <bch:expr>   <dbl> <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl>
#> 1 data.table      10  37.33ms   38.1ms    22.8     268.8MB    22.8 
#> 2 dplyr           10  22.75ms   26.3ms    38.7      80.9MB     9.68
#> 3 data.table 1000000 107.77ms  112.4ms     8.01    297.6MB     8.01
#> 4 dplyr      1000000    3.67s     3.8s     0.260   219.7MB    13.9

Created on 2026-10-13 with reprex v2.1.1

PR

library(data.table, warn.conflicts = FALSE)

set.seed(1)
n = 1e7
bench::press(
  groups = c(10, 1e6),
  {
    DT = data.table(g = sample.int(groups, n, TRUE), v = runif(n))
    df = as.data.frame(DT)
    bench::mark(
      data.table = DT[, .N, by = g],
      dplyr = dplyr::count(df, g),
      check = FALSE,
      iterations = 20,
      filter_gc = FALSE
    )
  }
)
#> Running with:
#>    groups
#> 1      10
#> 2 1000000
#> # A tibble: 4 × 7
#>   expression  groups      min   median `itr/sec` mem_alloc `gc/sec`
#>   <bch:expr>   <dbl> <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl>
#> 1 data.table      10   17.8ms   18.5ms    52.5      39.9MB    18.4 
#> 2 dplyr           10   22.4ms   23.7ms    39.1      80.9MB    23.5 
#> 3 data.table 1000000   94.5ms   97.6ms    10.1      68.7MB     4.02
#> 4 dplyr      1000000     3.7s     3.9s     0.255   219.7MB    14.6

Created on 2026-10-13 with reprex v2.1.1

@codecov

codecov Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.08%. Comparing base (9693c56) to head (d4db51a).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #7920   +/-   ##
=======================================
  Coverage   99.08%   99.08%           
=======================================
  Files          88       88           
  Lines       17418    17420    +2     
=======================================
+ Hits        17259    17261    +2     
  Misses        159      159           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jangorecki
jangorecki requested a review from ben-schwen October 13, 2026 21:19
@github-actions

Copy link
Copy Markdown
  • HEAD=gforce-dotN slower P<0.001 for memrecycle regression fixed in #5463
  • HEAD=gforce-dotN slower P<0.001 for Date-IDate regression in #7213

Comparison Plot

Generated via commit c26d25f

Download link for the artifact containing the test results: ↓ atime-results.zip

Task Duration
R setup and installing dependencies 1 minutes and 45 seconds
Installing different package versions 13 minutes and 38 seconds
Running and plotting the test cases 5 minutes and 39 seconds

@ben-schwen ben-schwen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, TY

@ben-schwen

Copy link
Copy Markdown
Member

Minor thing: moving the tests into optimize.Rraw keeps the GForce and lapply tets together

@ben-schwen
ben-schwen merged commit 7b7f63b into master Oct 19, 2026
11 of 12 checks passed
@ben-schwen
ben-schwen deleted the gforce-dotN branch October 19, 2026 13:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants