Skip to content

Commit

Permalink
fix: use the fair spill pool instead of the greedy spill pool (#2126)
Browse files Browse the repository at this point in the history
Right now we use the greedy spill pool when we use spilling for btree
index training. This is because the btree index plan is:

```
Input Stream -> Sort    -> SortPreservingMerge -> Output
Input Stream (old data) -/
```

My thought was that `Sort` was the only spillable operator and we could
use the greedy pool. It turns out that `SortPreservingMerge` is also
spillable and we can't get away with the greedy pool.
  • Loading branch information
westonpace authored Mar 28, 2024
1 parent 7e114ee commit cf9c5c5
Showing 1 changed file with 3 additions and 4 deletions.
7 changes: 3 additions & 4 deletions rust/lance-datafusion/src/exec.rs
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ use datafusion::{
execution::{
context::{SessionConfig, SessionContext, SessionState},
disk_manager::DiskManagerConfig,
memory_pool::GreedyMemoryPool,
memory_pool::FairSpillPool,
runtime_env::{RuntimeConfig, RuntimeEnv},
TaskContext,
},
Expand Down Expand Up @@ -175,9 +175,8 @@ pub fn execute_plan(
let mut runtime_config = RuntimeConfig::new();
if options.use_spilling {
runtime_config.disk_manager = DiskManagerConfig::NewOs;
runtime_config.memory_pool = Some(Arc::new(GreedyMemoryPool::new(
options.mem_pool_size as usize,
)));
runtime_config.memory_pool =
Some(Arc::new(FairSpillPool::new(options.mem_pool_size as usize)));
}
let runtime_env = Arc::new(RuntimeEnv::new(runtime_config)?);
let session_state = SessionState::new_with_config_rt(session_config, runtime_env);
Expand Down

0 comments on commit cf9c5c5

Please sign in to comment.