DEV3 Technologies home

DEV3 Developer Forum

chunk(), lazy() or cursor() for processing a million rows in Laravel?

Laravel helpSolved0 views
dev3 ·

I need to export or update about 1.5 million rows and Model::all() runs out of memory. I see chunk, chunkById, lazy, lazyById and cursor in the docs. Which one should I pick and what are the pitfalls?

#laravel#eloquent#performance

0

1 reply

dev3 · Accepted answer

all() loads every model into memory. Pick one of the batching tools.

chunkById (default choice for updates)

User::where('active', false)->chunkById(1000, function ($users) {
    foreach ($users as $user) {
        $user->update(['archived_at' => now()]);
    }
});

chunk() uses offset/limit, so if your callback changes the column used in the where clause, rows get skipped. chunkById() paginates by primary key and is safe for updates and deletes.

lazyById (nice API, constant memory)

foreach (Order::lazyById(2000) as $order) {
    fputcsv($out, [$order->id, $order->total]);
}

Returns a LazyCollection that loads chunks behind the scenes, so you can use map, filter and each as if it were one collection.

cursor()

Runs one query and hydrates one model at a time. Lowest memory in PHP but the DB driver may still buffer the full result set (the MySQL PDO driver uses buffered queries by default), and you cannot eager load relationships, so it invites N+1. Prefer lazy/lazyById unless you know why.

Bulk SQL when possible

User::where('active', false)->update(['archived_at' => now()]);

One query beats a million model updates, but skips model events.

Pitfalls

  • Eager loading works with chunk and lazy (->with('items')->lazyById()), not with cursor.
  • Run big jobs in a queue or console command with --memory monitoring; disable query log: DB::disableQueryLog().
  • Always order deterministically; chunkById does it for you.
0

Your reply

Every post is reviewed by DEV3 before it appears. Usually reviewed within 24 hours.

Markdown supported. Use ``` for code blocks. Images are not allowed.