The R Ibex package brings the Ibex query engine to R by providing a dplyr-compatible backend. Ibex can also be used as a standalone language through a REPL, notebooks or compiled to C++. The R Ibex package is significantly faster than dplyr and faster (more than an order of magnitude faster in 5/33 tests) than data.table on the majority of tasks benchmarked. The performance gap grows with data size. The script is in the repo and the paired results per task are in the final plot.

Ibex mainly wins on analytical queries and remains competitive on simple data updating operations. Ibex is still in development so further performance gains are on the cards. For the specific queries for which Ibex is slower, I’m aiming for parity, on those it might be hard to beet data.table as it is of course quite fast and well-integrated into R. The distribution of speedups and the performance over the benchmarked sizes for various task groups is shown below.

How

The wins are obtained by lazy by default queries and typed data frames, both of which enable algorithmic wins. Ibex uses more space-efficient data structures and has many specializations for specific query shapes. The package avoids interacting with R during query execution which would give all performance gains away again. The results below are from an 8 core, 64GB machine on AWS (r7i.2xlarge).

Algorithmic wins

One example of a win over data.table that Ibex can obtain is the following code: head(dt[order(x)], 100). In this statement, data.table will first sort and then take the top 100 rows, this algorithm is O(n log n) in general. The equivalent Ibex code tbl[order x, head 100] or tbl[order x][head 100] is analyzed before running and an efficient O(n log k) algorithm is employed.

Caveats and plans

This is still an experimental package, there will be bugs and there will be performance regressions. The Ibex package falls back on dplyr if it can’t handle a query.

Mixing queries with arbitrary R functions is not supported well yet. At the moment, I’m looking at efficiently falling back to R vectorized calls during Ibex query execution for functions that are defined in R.

Lazy execution of the queries will allow optimized reading and decoding of parquet files depending on which rows and columns are required, this has been implemented in Ibex proper but the R package does not yet take advantage of this fusion.

All observations

The plot below shows each paired observation of Ibex vs data.table. The performance benefits it gives cover many use cases and can be substantial. For the cases where Ibex loses, the loss is often minimal.