Perl, Caching, and Memory Leaks

class: center, middle

# Perl, Caching, and Memory Leaks

Lee Johnson (LEEJO / leejo / lee@humanstate.com)

???

Warning! Possibly hand wavy in parts

My C is very rusty (although it seems people *want* rust rather than C these days?)

The perl internals are very deep, my knowledge of those internals is not deep

This is just the basics

---

## We've all heard of this?

--
* ### Make it work

--
* ### Make it right

--
* ### Make it fast

???
The oft missed additional clause

--
* ### Fix all the bugs you introduced by making it fast

???
Where "bugs" could be memory leaks

Immortal caches could be classified as memory leaks, if you have enough of them

More on this later

### But first a recap...

---

## Spot the bug

```c
#include <stdio.h>
#include <stdlib.h>
#include <sys/resource.h>

int main() {
	int i, *ptr; struct rusage r_usage;

getrusage(RUSAGE_SELF,&r_usage);
	printf("Start usage: %ld bytes\n",r_usage.ru_maxrss);

for ( i = 0; i < 100000; i++ ) {
		ptr = (int*) malloc(1000 * sizeof(int));
		// do something with ptr ...
	}

getrusage(RUSAGE_SELF,&r_usage);
	printf("  End usage: %ld bytes\n",r_usage.ru_maxrss);

return 0;
}
```

???

It should be obvious, this is a talk about memory leaks

---

## malloc() but no free()

```c
for ( i = 0; i < 100000; i++ ) {
*	ptr = (int*) malloc(1000 * sizeof(int));
	// do something with ptr ...
}
```

```bash
> ./a.out
Start usage: 1675264 bytes
  End usage: 3538944 bytes

```

???
Process has doubled in size

We have to deal with this in C

---

## Clean up after use

```c
for ( i = 0; i < 100000; i++ ) {
	ptr = (int*) malloc(1000 * sizeof(int));
	// do something with ptr ...

// and free up memory
*	free(ptr);
}
```

```bash
> ./a.out
Start usage: 1675264 bytes
  End usage: 1728512 bytes
```

???
Not a big deal in this trivial example

But imagine this is somewhere deep in a library, and part of a long running process

That's why we have tools like ASAN, purify, valgrind, etc

---

## What about Perl?

```perl
#!perl

use strict;
use warnings;
use feature qw/ say /;
use Unix::Getrusage;

my ( $i,$ptr );

printf("Start usage: %ld bytes\n",getrusage()->{ru_maxrss});

for ( $i = 0; $i < 100000; $i++ ) {
*    $ptr = [ ( 1 ) x 1000 ];
    # do something with ptr ...
}

printf("  End usage: %ld bytes\n",getrusage()->{ru_maxrss});

exit( 0 );
```

???
It's not exactly the same, but close enough for what I'm getting at here

---

## Perl will handle memory usage for us

```bash
> perl leak.pl
Start usage: 4177920 bytes
  End usage: 4366336 bytes
```

???

We had a "pointer" to an array of 1000 ints, done 100,000 times over

Perl handled the memory allocation and freeing for us

### In Perl we don't have to worry about memory allocation ...

### Until we do

### Memory is cheap but not infinite

???

Or at least not infinitely fast

---

## On Reference Counting

```
perldoc perlguts
```

```
Reference Counts and Mortality

Perl uses a reference count-driven garbage collection mechanism. SVs, AVs,
  or HVs (xV for short in the following) start their life with a reference
  count of 1. If the reference count of an xV ever drops to 0, then it will
  be destroyed and its memory made available for reuse.

...
```

???

This is how perl handles the memory for us.

We can inspect reference counts using Devel::Peek

---

## Devel::Peek

```perl
use Devel::Peek;

for ( $i = 0; $i < 100000; $i++ ) {
    $ptr = [ ( 1 ) x 1000 ];
    # do something with ptr ...
}

*Dump( $ptr );
```

---

## Devel::Peek

```bash
SV = IV(0x7fc9dc829350) at 0x7fc9dc829360
  REFCNT = 1
  FLAGS = (ROK)
  RV = 0x7fc9dc828958
  SV = PVAV(0x7fc9dc804e80) at 0x7fc9dc828958
    REFCNT = 1
    FLAGS = ()
    ARRAY = 0x7fc9dc600010
    FILL = 1
    MAX = 1
    FLAGS = (REAL)
    Elt No. 0
    SV = IV(0x7fc9dc8289f0) at 0x7fc9dc828a00
      REFCNT = 1
      FLAGS = (IOK,pIOK)
      IV = 1
    Elt No. 1
    SV = IV(0x7fc9dc803270) at 0x7fc9dc803280
      REFCNT = 1
      FLAGS = (IOK,pIOK)
      IV = 1
	... (repeated 1000 times)
```

???
This is the output from Devel::Peek::Dump(), I won't go into it other than to highlight (next slide)

---

## Devel::Peek

```bash
SV = IV(0x7fc9dc829350) at 0x7fc9dc829360
* REFCNT = 1
  FLAGS = (ROK)
  RV = 0x7fc9dc828958
  SV = PVAV(0x7fc9dc804e80) at 0x7fc9dc828958
*   REFCNT = 1
    FLAGS = ()
    ARRAY = 0x7fc9dc600010
    FILL = 1
    MAX = 1
    FLAGS = (REAL)
    Elt No. 0
    SV = IV(0x7fc9dc8289f0) at 0x7fc9dc828a00
*     REFCNT = 1
      FLAGS = (IOK,pIOK)
      IV = 1
    Elt No. 1
    SV = IV(0x7fc9dc803270) at 0x7fc9dc803280
*     REFCNT = 1
      FLAGS = (IOK,pIOK)
      IV = 1
	... (repeated 1000 times)
```

???
REFCNT - the reference count of each value

Notice that it's 1 for every value here

So when the top level REFCNT goes to 0 everything in this struct can be garbage collected

When does that happen (see two slides up?)

---

## How Leaks Happen

```perl
#!perl

use strict;
use warnings;
use feature qw/ say /;
use Unix::Getrusage;

my ( $i,$ptr,@arr );

printf("Start usage: %ld bytes\n",getrusage()->{ru_maxrss});

for ( $i = 0; $i < 100000; $i++ ) {
*     push( @arr,$ptr = [ ( 1 ) x 1000 ] );
    # do something with ptr ...
}

printf("  End usage: %ld bytes\n",getrusage()->{ru_maxrss});
exit( 0 );
```

???

Here's the same perl script with a subtle tweak.

Note I'm talking about subtle memory leaks here, this is a slightly contrived example to illustrate how it happens.

You can easily create death by "Out of memory!" by trying to read a 100GB file into an array, putting large amounts of data into global variables (another reason why they're bad), and so on.

---

## How Leaks Happen

```bash
> perl leak_2.pl
Start usage: 4005888 bytes
  End usage: 3261702144 bytes <--- Eeek!
```

```perl
use Devel::Peek;

for ( $i = 0; $i < 100000; $i++ ) {
*   push( @arr,$ptr = [ ( 1 ) x 1000 ] );
    # do something with ptr ...
}

*Dump( $ptr );
```

???
You probably *don't* want to run this, dumping out the tree of a 3GB struct isn't a great idea

---

## How Leaks Happen

```bash
SV = IV(0x7f976c805420) at 0x7f976c805430
  REFCNT = 1
  FLAGS = (ROK)
  RV = 0x7f976c003280
  SV = PVAV(0x7f976c004e30) at 0x7f976c003280
    REFCNT = 2
    FLAGS = ()
    ARRAY = 0x7f976bf013a0
    FILL = 1
    MAX = 1
    FLAGS = (REAL)
    Elt No. 0
    SV = IV(0x7f976c0293b0) at 0x7f976c0293c0
      REFCNT = 1
      FLAGS = (IOK,pIOK)
      IV = 1
    Elt No. 1
    SV = IV(0x7f976c028a98) at 0x7f976c028aa8
      REFCNT = 1
      FLAGS = (IOK,pIOK)
      IV = 1
	... (repeated 1000 times)
```

???

See the difference between this and the previous Dump output?

---

## How Leaks Happen

```bash
SV = IV(0x7f976c805420) at 0x7f976c805430
  REFCNT = 1
  FLAGS = (ROK)
  RV = 0x7f976c003280
  SV = PVAV(0x7f976c004e30) at 0x7f976c003280
*   REFCNT = 2
    FLAGS = ()
    ARRAY = 0x7f976bf013a0
    FILL = 1
    MAX = 1
    FLAGS = (REAL)
    Elt No. 0
    SV = IV(0x7f976c0293b0) at 0x7f976c0293c0
      REFCNT = 1
      FLAGS = (IOK,pIOK)
      IV = 1
    Elt No. 1
    SV = IV(0x7f976c028a98) at 0x7f976c028aa8
      REFCNT = 1
      FLAGS = (IOK,pIOK)
      IV = 1
	... (repeated 1000 times)
```

???

Because the global @arr points to this, so even when $ptr is overwritten in each
iteration of the loop there is still a pointer to the array(s).

So this REFCNT doesn't ever go to 0, we have a memory leak

---

## A Little Bit About Our App

![campaign](img/campaign.png)

???

A typical page on the site

This is built from a template toolkit template that uses objects passed in via the mojo controller

---

## A Little Bit About Our App

--
* We try to do as much as possible with objects

???

They're just easier to work with, provide proper abstraction and encapsulation

--
* We break out repeated code into Roles

--
* This includes constructor logic

???

We have, for example, a populateActivist method (in a Role) that will return a new instance of an Activist object given the data from the db (which is a DBIx::Class::ResultSet object)

In the Activist class:
```perl
$self->_populateActivist( $activist_rs,$self );
```

???
\_populateActivist will set all the attributes on $self here using the data from the resultset

In some other class:
```perl
my $Activist = $self->_populateActivist( $activist_rs );
```

???
\_populateActivist will return a new instance of the Activist class here, with its attributes populated

--
* We are very greedy with data

???

We try to grab as much as possible from the database (through use of prefetch in dbic) and populate as many objects as necessary ahead of time.

--
* Going over the network to the db is expensive

???
Doing one query with several joins is cheaper than several deferred single queries because 90% of the time we end up needing that information anyway.

When you're developing everything (?) is on localhost, which effectively has a zero round trip time

---

## A Little Bit About Our App

![donations](img/donations.png)

???

Built from attributes on objects

---

## A Little Bit About Our App

![donations](img/donations_objects.png)

???

Each blue dot is an attribute out of a single object, sometimes nested objects

On some pages we can end up with the *same* object more than once - for example in this case an Activist can make many donations so they could occur more than once on the same page because a Donation object has an Activist object

We can optimise that - one of the fundamental approaches to optimisations: remove duplicate work

---

## The Ephemeral Object Cache (EOC)

--
* `_populateFoo` methods use a globally scoped cache

--
* If an object *already* exists for the class return that object

???

Otherwise create the object then store it in the cache

--
* Objects are keyed in the cache by their class name and primary key

--
* They're sort of like singletons, but not

--
* The cache persists *only* for the life of the request

???

We nuke the entire cache at the start *and* end of each request

--
* In some cases this is a huge optimisation:

```bash
D> Nuking 30 EOC objects, reused 24 times
```

???
The EOC here prevented the creation of 24 duplicate objects

On some edge case pages these can go up into the hundreds, or even thousands

```bash
D> Nuking 3675 EOC objects, reused 3561 times:
Address - 2
Agency - 1
Agency::Account - 1
Agency::Category - 1
Beneficiary - 1
Contribution::History - 1201
Country - 1
Currency - 1
Payment - 1157
Payment::Record - 1205
Payment::Split - 87
Property - 17
```

???

A more in depth breakdown from the log file - the EOC makes a huge difference here.

Why so many objects? Because edge cases, users do odd things with the site, every Payment object has a Currency object, etc.

---

## Object Methods

```perl
> perl \
	-Ilib \
	-MGivenGain::Model::Activist \
	-E'say GivenGain::Model::Activist->new( id => 106579 )
		->first_name'

Lee
```

???

Objects obviously have attributes

```perl
say GivenGain::Model::Activist->new( id => 106579 )
	->donations->[0]
	->amount

100
```

???

Some of those attributes are objects

--
```perl
say GivenGain::Model::Activist->new( id => 106579 )
	->donations->[0]
	->donor
	->first_name

Lee
```

???

And those objects have attributes

---

## Round And Round And Round

A contrived example:

```perl
say GivenGain::Model::Activist->new( id => 106579 )
	->donations->[0]
	->donor
	->donations->[0]
	->donor
	->donations->[0]
	->donor
	->first_name

Lee
```

No circular references here though:

```perl
my $act = GivenGain::Model::Activist->new( id => 106579 );
say $act->first_name;
say $act;
say $act->donations->[0]->donor;'

Lee
GivenGain::Model::Activist=HASH(0x2652388)
GivenGain::Model::Activist=HASH(0x61b3dd8)
```

???
Note different instances of the class

---

## Leaks via Caching

When we turn on the EOC?

```perl
*$main::eoc = {};

my $act = GivenGain::Model::Activist->new( id => 106579 );
say $act->first_name;
say $act;
say $act->donations->[0]->donor;'

Lee
GivenGain::Model::Activist=HASH(0x4639e08)
GivenGain::Model::Activist=HASH(0x4639e08)
```

Activist object now points back at itself. A circular reference:

```perl
find_cycle( $act );

Cycle (1):
    $GivenGain::Model::Activist::A->{'donations'} => \@B
                           $B->[0] => \%GivenGain::Model::Donation::C
    $GivenGain::Model::Donation::C->{'donor'} => \%GivenGain::Model::Activist::A
```

???

As you can see, activist points to donations that points to a donor that points back at the activist.

In a long running process this will never be garbage collected = a memory leak.

More on find\_cycle in a bit

---

## Detecting Leaks

???

The problem with a large app is you need at least some idea where the leak might be before you start to use specific tools

---

## Detecting Leaks

### Brute Force Crawling

```bash
MEMORY_CHECK=4570,4571 prove -lrv t/www/010_page_load.t 2>&1 | grep grew

# /cc/putfootrally2016/donations/ grew 0.15% (PID: 4571)
# /cc/putfootrally2016/donations/page/2/ grew 0.15% (PID: 4571)
```

High level crawler script, can be told to watch memory of specific PIDs

In theory, if there are no memory leaks, when run enough times the PIDs will eventually stop growing

If there is a particular page that results in a leak the test will find it

---

## Detecting Leaks

### Brute Force Dumping

```
On Fri, 16 Mar 2018 17:39:23 +0000 Andrew Beverley wrote:
1. Once the process was significantly large in memory use, I core-dumped
it using gdb:

gdb --pid=1234
  (gdb) gcore
  (gdb) detach
  (gdb) exit

2. I ran the dump through strings to see if I could spot any patterns:

strings core.1234 | sort | uniq -c | sort -nr | less

3. This showed thousands of strings containing data from SSL
certificates. I then looked at the SSL-related modules and found this:

https://metacpan.org/changes/distribution/Net-SSLeay#L29

4. I upgraded Net::SSLeay, repeated the above process, and there was a
significant reduction in that SSL certificate data in the dump.
```

---

## Detecting Leaks

### Brute Force Destruction + Devel::Cycle

```perl
package Some::Role::Used::By::All::Classes;

use Moose::Role;
use Devel::Cycle;

# everthing uses this role, so we can create a global DEMOLISH method
# here that will be called by all objects on destruction
sub DEMOLISH {
	my ( $self ) = @_;

find_cycle( $self ) if $ENV{FIND_MEMORY_CYCLE};
}
```

Set `FIND_MEMORY_CYCLE` in DEV.

???

But those objects with circular references won't be destroyed?

Periodically restart the server, which will flush them out.

---

## Fixing Leaks

Possibly the quick and easy way, may result in bugs.

```perl
use Scalar::Util qw/ weaken /;

weaken( $ref );
```

???

Prevents a reference count being held against the thing referenced by $ref

For Moose like attributes:

```perl
has foo => (
	...
	weak_ref => 1,
);
```

This will almost certainly result in bugs.

See [https://rjbs.manxome.org/rubric/entry/1837](https://rjbs.manxome.org/rubric/entry/1837)

???

Your attribute will probably be garbage collected prematurely

You probably need to tweak your logic rather than do this

---

## Fixing Leaks

```perl
has foo => (
	...
	default => sub { ... },
);
```

Becomes:

```perl
my $_foo;

sub foo {
	return $_foo if $_foo;
	my ( $self,... ) = @_;
	$_foo = ...
	return $_foo;
);

```

???

It's not as pretty but foo is no longer referenced by $self here, we have broken the cycle

In this case we can still take advantage of the EOC

You don't have to use the package scoped variable if your caller stores the result somewhere itself

---

## Preventing Regressions

### Brute Force Destruction + Devel::Cycle

As before, a `DEMOLISH` method.

???

Really you want something more targeted?

---

## Preventing Regressions

### Test::Memory::Cycle

```perl
use Test::Memory::Cycle;

$main::eoc = {};

my $Activist = GivenGain::Model::Activist->new(
    id => 106579,

is( $Activist->first_name,'Lee','->first_name' );
is( $Activist->donations->[0]->donor->first_name,'Lee','cycle through' );

memory_cycle_ok( $Activist );
```
---

## Preventing Regressions

### Test::Memory::Cycle

```bash
t/unit/givengain/model/activist_leak.t ..
ok 1 - ->first_name
ok 2 - cycle through
not ok 3

#   Failed test at t/unit/givengain/model/activist_leak.t line 20.
# Cycle #1
#     GivenGain::Model::Activist A->{donations} => @B
#     @B->[0] => GivenGain::Model::Donation C
#     GivenGain::Model::Donation C->{donor} => GivenGain::Model::Activist A
...
```

---

## Preventing Regressions

### Test::LeakTrace

```perl
use Test::LeakTrace;

$main::eoc = {};

no_leaks_ok {
    my $Activist = GivenGain::Model::Activist->new(
        id => 106579,
    );

my $first_name = $Activist->first_name;
    my $Donor = $Activist->donations->[0]->donor;
} 'Activist does not leak';

```

---

## Preventing Regressions

### Test::LeakTrace

```bash
not ok 1 - Activist does not leak (leaks 75 <= 0)

#   Failed test 'Activist does not leak (leaks 75 <= 0)'
#   at t/unit/givengain/model/activist_leak.t line 19.
#     '75'
#         <=
#     '0'
```

???

Sits there for a while working out the tree then...

```bash
# leaked REF(0x5efd438) from lib/GivenGain/Role/Populate/Activist.pm line 46.
#   45: $act->location_id($activist->location_id);
#   46: if ($activist->location) {
#   47:         $act->location($act->_populateLocation($activist->location));
# SV = IV(0x5efd428) at 0x5efd438
#   REFCNT = 1
#   FLAGS = (ROK)
#   RV = 0x5f099d8
#     SV = PVHV(0x5be48e0) at 0x5f099d8
...
```

???

Applying the aforementioned fix makes the test pass

And also doesn't break our *other* tests

Test::LeakTrace is quite slow so perhaps not one to run all the time

But it's very thorough

In fact too thorough - finds leaks in the Test:: namespace code, DBIx::Class, etc

---

## Questions?

More info:

* [perldoc perlguts](http://perldoc.perl.org/perlguts.html)
* [perldoc perldebguts](http://perldoc.perl.org/perldebguts.html#Debugging-Perl-Memory-Usage)
* [Devel::Cycle](https://metacpan.org/pod/Devel::Cycle)
* [Test::Memory::Cycle](https://metacpan.org/pod/Test::Memory::Cycle)
* [Devel::Gladiator](https://metacpan.org/pod/Devel::Gladiator)
* [Devel::SizeMe](https://metacpan.org/pod/Devel::SizeMe)
* [Devel::MAT](https://metacpan.org/pod/distribution/Devel-MAT/lib/Devel/MAT/UserGuide.pod) [more info](https://tech.binary.com/tracing-perl-memory-leaks-with-devel-mat/)
* [Test::LeakTrace At The Rescue](https://blog.celogeek.com/201211/277/perl-memory-leaks-testleaktrace-at-the-rescue/)
* [Tim Bunce, Profiling Memory Usage LPW 2013](https://www.youtube.com/watch?v=UkEOQoOYYgQ)
* [Nicolas Rochelemagne, Perl dependencies: playing with %INC and profiling memory, TPC 2016](https://www.youtube.com/watch?v=AMa1JWgG4yY)
* [Three Kinds of Leaks](https://blog.nelhage.com/post/three-kinds-of-leaks/)

---