My recent job was to write Symfony 2 command that copies and transforms enormous amount of SQL data (~70GB) from one database to another doing a lot of queries and data transformation in mean time. I knew that this script would be running for days or even weeks.

I found out that when I run my newly created command it takes a while for it to start gradually consume more and more memory.

It’s always nice to have a method like this for monitoring memory usage.

    function printMemoryUsage()
    {
        $this->output->writeln(sprintf('Memory usage (currently) %dKB/ (max) %dKB', round(memory_get_usage(true) / 1024), memory_get_peak_usage(true) / 1024));
    }

Right after the start the command used around 30MB – not much since I have 8GB right now, normally I wouldn’t even bother to optimize it more.

But after a while it reached 40MB, 50MB and kept going and since Ubuntu by default has php memory limit in CLI set to -1 (unlimited) this script could consume the whole RAM after a couple of days.

I thought that garbage collector which is enabled by default in PHP 5.3 would clear memory at some point but it doesn’t, setting memory limit to reasonable 32MB in this case by putting in code

ini_set('memory_limit', '32M');

caused standard exhausted memory limit error. Then I asked myself why garbage collector didn’t clear unused objects that I could miss?

Ok, it’s not the time to be lazy and relay on g. collector, I spent some time adding unset() for every object that was created. That still didn’t help :(

After hours of debugging I’ve found 3 issues:

1st issue, lazy Garbage Collector

If you use infinite loop like I did you should force GC to do its job by

    gc_collect_cycles();

2nd, dirty Entity Manager

Use clear() method once a while, it detaches doctrine objects that are not used any more.

    $this->em->flush();
    $this->em->clear();

3rd issue, SQL Logger, this one was the worst to find

Every time you query database SQL Logger stores information about that. Normally, it’s not a problem but in commands running infinitely every KB counts.

You can turn it off like this.

    $this->em = $this->getContainer()->get('doctrine')
                    ->getEntityManager();
    $this->em->getConnection()->getConfiguration()->setSQLLogger(null);

If this post helped you let me know by leaving a comment, it’s always nice to know that I could help someone :)