symfony-2.0-logo

How to avoid memory leaks in Symfony 2 Commands

My recent job was to write Symfony 2 command that copy and transform enormous amount of SQL data (~70GB) from one database to another doing a lot of queries and data transformation in mean time. I knew that this script will be running for days or even weeks.

I found out that when I run my newly created command for while it slowly consume more and more memory.

It’s always nice to have method like this for monitoring memory usage.

function printMemoryUsage() 
{
    $this->output->writeln(sprintf('Memory usage (currently) %dKB/ (max) %dKB', round(memory_get_usage(true) / 1024), memory_get_peak_usage(true) / 1024));
}

Right after start command used around 30MB – not much since I have 8GB right now, normally I wouldn’t even bother to optimize it more.

But after a while it reached 40MB, 50MB and kept going, since Ubuntu by default have php memory limit in CLI set to -1 (unlimited) this script could consume whole RAM after couple of days.

php-lazy-garbage-collector

I thought that garbage collector which is enabled by default in PHP 5.3 would clear memory at some point but he doesn’t, setting memory limit to reasonable 32MB in this case by putting in code

ini_set('memory_limit', '32M');

Caused standard exhausted memory limit error, I asked myself why garbage collector didn’t clear unused objects that I could missed?

Ok, it’s not time to be lazy and relay on g. collector, I’ve spent some time adding unset() for every object that was created. That still didn’t help :(

After hours of debugging I’ve found 3 issues

1st issue, lazy Garbage Collector
If you use infinite loop like I did you should force GC to do it’s job by

gc_collect_cycles();

2nd, dirty Entity Manager
use clear() method once a while, it detaches doctrine objects that are not used any more.

$this->em->flush();
$this->em->clear();

3rd issue, SQL Logger, this one was the worst to find
Every time you query database SQL Logger stores information about that, normally it’s not a problem but in infinite running commands every kB counts.

You can turn it off like this.

$this->em = $this->getContainer()->get('doctrine')
                ->getEntityManager();
$this->em->getConnection()->getConfiguration()->setSQLLogger(null);
If this post helped you let me know by leaving a comment, it’s always nice to know that I could help someone :)

13 thoughts on “How to avoid memory leaks in Symfony 2 Commands

      1. Note that prod environment isn’t enough, you need to run it in no-debug mode to stop that memory leak.

        If you look in app/console you’ll see that the Kernel is initialised with the environment you set but unlike web/app.php it does not force debug off when you’re using prod.

        Run your script like this:

        php app/console a:command –env=prod –no-debug

        and you’ll see that the logger with no longer run which causes a huge memory leak on long iterations due to the string replacements.

    1. You’re welcome Alex :)

      I keep finding issues like this almost everyday during my work but for now they stay as sticky notes on my “offline” board as items to write in future. Maybe in “next life” I will have time for stuff like this :)

  1. Thanks a lot for this! I’ve also spent quite some time inserting `unset()` whereever possible hoping that should be enough. Well, it’s not… Thanks for the new knowledge! :)

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>