The R Ibex package brings the Ibex query engine to R by providing a dplyr-compatible backend. Ibex can also be used as a standalone language through a REPL, notebooks or compiled to C++. The R Ibex package is significantly faster than dplyr and faster (more than an order of magnitude faster in 5/33 tests) than data.table on the majority of tasks benchmarked. The performance gap grows with data size. The script is in the repo and the paired results per task are in the final plot.
Ibex mainly wins on analytical queries and remains competitive on simple data updating operations. Ibex is still in development so further performance gains are on the cards. For the specific queries for which Ibex is slower, I’m aiming for parity, on those it might be hard to beet data.table as it is of course quite fast and well-integrated into R. The distribution of speedups and the performance over the benchmarked sizes for various task groups is shown below.
How
The wins are obtained by lazy by default queries and typed data frames, both of which enable algorithmic wins. Ibex uses more space-efficient data structures and has many specializations for specific query shapes. The package avoids interacting with R during query execution which would give all performance gains away again. The results below are from an 8 core, 64GB machine on AWS (r7i.2xlarge).
Algorithmic wins
One example of a win over data.table that Ibex can obtain is the following code: head(dt[order(x)], 100). In this statement, data.table will first sort and then take the top 100 rows, this algorithm is O(n log n) in general. The equivalent Ibex code tbl[order x, head 100] or tbl[order x][head 100] is analyzed before running and an efficient O(n log k) algorithm is employed.
Caveats and plans
This is still an experimental package, there will be bugs and there will be performance regressions. The Ibex package falls back on dplyr if it can’t handle a query.
Mixing queries with arbitrary R functions is not supported well yet. At the moment, I’m looking at efficiently falling back to R vectorized calls during Ibex query execution for functions that are defined in R.
Lazy execution of the queries will allow optimized reading and decoding of parquet files depending on which rows and columns are required, this has been implemented in Ibex proper but the R package does not yet take advantage of this fusion.
All observations
The plot below shows each paired observation of Ibex vs data.table. The performance benefits it gives cover many use cases and can be substantial. For the cases where Ibex loses, the loss is often minimal.
Ibex is a new table focused language for analysis of tabular data and time-series. Its goal is to combine the conciseness of data.table with speed, obtained by native code generation, a fast interpreter and lots of attention for efficient algorithms and code. It aims to make exploratory queries quick to write and fast to execute. The goal is to make data analysis feel smooth and fast. A demo of a quant pipeline gives an idea of what is possible.
Since my previous post on Ibex a lot has happened. Ibex gained useful features, speed and benchmarks (these numbers are before multithreading support) and cross platform binaries. The new features are
Rich join support, with a basic approach to join planning.
Vectorization and efficient algorithms of common table operations.
Date and Timestamps are efficiently stored as signed counts since epoch.
Time Series support: windowing, rolling, grouped rolling, resampling, as-of joins.
Fast, vectorized RNG using Zorro (xoshiro256++ based). Purpose-built for Ibex and split off.
Multithreading in active development but for OHLCV analysis I have some promising results already. However, I only looked at this narrow use case of writing.
Some integration work for Python and R but still alpha.
Plugins for I/O and data analytics.
Some exploration in how an effect system can work for Ibex. The syntax supports it but the Ibex optimizer doesn’t use the information yet. Long term goal is to allow Ibex to efficiently use code in plugins, for example by reordering function calls if it can be proven this is correct based on the given effects.
Ergonomics
I started working on Ibex with a focus on ergonomics. It should be easy to write quick analysis without too much special characters or using the shift key. OHLCV looks like this:
ticks[ select {
open = first(price),
high = max(price),
low = min(price),
close = last(price),
volume_sum = sum(volume) },
by symbol,
resample 10s,
order symbol ]
Ticks is a table (TimeFrame) with columns timestamp, symbol, price and volume. The expression selects the data it wants in the select clause, by symbol instructs Ibex to group the rows by symbol, resample 10s uses a duration literal to resample the data in 10 second long buckets and the final clause orders by symbol. The ordering by timestamp is implicit. Short, sweet and easy to write. The code is succinct but there is no need to do any code golf to get to that low count. select can be omitted if you want. The syntax borrows heavily from both R’s data.table and SQL keywords. Compared to libraries such as Polars, Pandas and data.table, there is no need to quote column names. The Ibex syntax is specialized for tables and will look up names in the table context first, then in the outer scope. Compared to SQL, the order of the clauses is free so it is possible to write queries in the order that feels natural for the problem at hand. Of course, I’m biased but I would use the Ibex syntax for its convenience alone.
Joins
Joins are easy to write as well, and quite fast. Using the nycflights13 data set we can get the name of the carrier of the flight using a join and then do a group by and count. Very little boilerplate required. Every word or symbol in the expression is contributing something meaningful to the semantics of the query.
The TimeFrame is a unique feature to Ibex. A TimeFrame is like a regular DataFrame that is always has one chosen timestamp column as the final in an ordering. It is what makes the resample keyword in the snippet above work. This simplifies writing time series analysis such as windowing and resampling and reduces the likelihood of errors.
Performance
Being convenient might not be enough to convince everybody though. I have done some benchmarking and am still doing more. Ibex performs very well on one core but lags more mature engines when the core count goes up. Often, on a single core it is competitive with multithreaded Polars and now that multithreaded Ibex is in development, I’m confident that it will be be beating Polars, Clickhouse and DuckDB in many real world use cases in the coming months. Fully replicating the PDS benchmark ran by Polars is on the short term roadmap. An excerpt from the in-memory benchmarks, single threaded Ibex against the multithreaded competition (8 cores) on 32 million rows:
Deep dives in why Ibex performs as well as it does will be published in follow-up blog posts.
Plugins and interop
Ibex supports plugins to allow interop and more general programming. Through these plugins Ibex can read and write parquet and CSV. For data analysis, there is a k-means, pca plugin. For generating some fake data, you can use the data_gen plugin:
The important table operations have been implemented and perform quite well on a single core. Improving multithreading support and further development of the plugins. My early experiments have shown that on the small PDS set (SF-1) Ibex is competitive on a single core, it loses when core counts go up. Making Ibex work efficiently on huge machines will take some effort, probably a lot.
Conclusion
Ibex is an ergonomic and fast alternative for the well known analytical data processors such as Polars and DuckDB. In my benchmarking it beats the alternatives on a single core and the syntax is both easy to read and write. Efficient threading is actively being worked on and initial experiments suggest that the single core performance holds up in a multithreaded setting.
Anthropic used a swarm of Claude Opus 4.6 agents to build a C compiler for about $20,000 in API costs with minimal intervention. It did make me realise that while I have been a happy user of R’s data.table for quite some time, the time to build something better is here. Third party packages like data.table but also pandas, Polars, the Tidyverse are bolted on an existing scripting language. These packages are marvels of engineering but the required hacks do have both pros and cons. They can lead to surprising behaviour (often: bugs) and when performance matters, a deep understanding of what happens under the hood is needed. The trade-off seems structural. A language focused on data frames might be a much more powerful tool.
So I, together with my team of LLMs started building a new language. The result is Ibex. It is far from done but it does do some useful work already:
CSV and Parquet reading
Select / Update / Filter / Group / Aggregate
Regular and table variables
Function definitions
External C++ interop
Basic type inference
Basic joins
It can be run through a REPL and also transpile to C++ so the generated code can be used in larger projects. An example
All of these run quite fast, practically instantly on my machine and in a similar range as single threaded data.table.
What’s next?
Ibex is still incomplete, data frame manipulation needs to be extended and tuned. Broadcasting operators are a must have. Dates are represented by strings still. Time series and windowing support seem like a good idea. Multithreading support will yield significant speed ups. Graphing straight from the REPL is good to have. Since C++ interop is quite straightforward, I want to add numerical methods by importing a BLAS and support for random number generation and statistical tests.
Feedback and contributions are welcome. The repository is on GitHub.
