In the rapidly evolving landscape of R, the line between "script" and "orchestration" has never been thinner. For years, if you needed to run tasks in parallel, manage complex dependencies, or scale a workflow beyond the limits of your local memory, you reached for packages like future , foreach , or targets .
But crew (which stands for oordinated R esource E xecution W orker) isn't just another entry in the parallel-processing catalog. Created by William Landau, the author of the targets package, crew is a fundamental rethink of how R should talk to background jobs. the crew pkg
But the real magic happens when you pair crew with targets . In a _targets.R file, changing the controller is a one-line edit: In the rapidly evolving landscape of R, the
Furthermore, crew requires that your worker sessions be fully self-contained. Any library, function, or data object must be loaded or passed explicitly. There is no "magic" global environment inheritance. crew is the industrial-grade conveyor belt that the R ecosystem has been missing. It doesn't try to be the flashiest parallel package; instead, it focuses on being the most reliable . Created by William Landau, the author of the
controller <- crew_controller_local(workers = 8) controller$start() for (file in all_files) { controller$push( name = file, command = process_file(file) ) } results <- list() while (controller$pop()$name != "done") { Crew auto-replaces crashed workers results <- c(results, controller$pop()$result) }
For HPC users: Replace crew_controller_local() with crew_controller_slurm() and define your job submission template. The API remains identical.
With crew :