Monitoring Python Memory Usage: Techniques And Best Practices

Memory management is an important aspect of programming logic, especially in Python, where vast data needs to be managed due to its use in ML and AI. Memory leaks can cause a program to run out of memory after several hours of execution. To manage these leaks, memory monitoring is essential.

There are several tools available for monitoring memory usage in Python, including:

- Tracemalloc: A library module that traces every memory block in Python. It provides information about the total size, number, and average size of allocated memory blocks.

- Psutil: A Python system library used to track various system resources and their utilisation. It can be used for profiling, limiting, and managing process resources.

- Memory Profiler: A Python library module used for monitoring process memory. It uses psutil to create a decorator and get the memory distribution.

- Guppy: Allows you to profile an entire Python application and restrict it to specific elements.

- mprof: Shows memory usage over the lifetime of an application, helping to identify if memory is being cleaned up and released.

- memory_profiler: An open-source package that performs line-by-line memory profiling for Python code.

These tools can help developers optimise their code by identifying areas of high memory usage and potential memory leaks.

Using the memory_profiler package

The memory_profiler package is a Python library module used for monitoring process memory. It uses the psutil module to create a decorator, which is then used to get the memory distribution.

To install the package, use the following command:

Pip install -U memory_profiler

The memory_profiler package can be used to display the memory consumed by each line in the code. Implementation of finding the memory consumption is easy as the decorator can be directly called instead of writing a whole new code.

Python

Import the profile decorator from the memory_profiler package

From memory_profiler import profile

Instantiate the decorator

@profile

Define the function for which memory needs to be monitored

Def my_func():

X = [x for x in range(0, 1000)]

Y = [y * 100 for y in range(0, 1500)]

Del x

Return y

If __name__ == '__main__':

My_func()

The output will look something like this:

Line # Mem usage Increment Occurrences Line Contents

============================================================

2 30.5 MiB 30.5 MiB 1 @profile

3 def my_func():

4 30.6 MiB 0.1 MiB 1003 x = [x for x in range(0,1000)]

5 30.7 MiB 0.1 MiB 1503 y = [y*100 for y in range(0,1500)]

6 30.7 MiB 0.0 MiB 1 del x

7 30.7 MiB 0.0 MiB 1 return y

The first column represents the line number of the code that has been profiled, the second column (Mem usage) the memory usage of the Python interpreter after that line has been executed, the third column (Increment) represents the difference in memory of the current line with respect to the last one, and the last column (Line Contents) prints the code that has been profiled.

Using the tracemalloc library module

Tracemalloc is a library module that traces every memory block in Python. It provides the following information:

• Traceback where an object was allocated
• Statistics on allocated memory blocks per filename and per line number: total size, number, and average size of allocated memory blocks
• Compute the differences between two snapshots to detect memory leaks

To start tracing most memory blocks allocated by Python, the module should be started as early as possible by setting the PYTHONTRACEMALLOC environment variable to 1, or by using the -X tracemalloc command-line option. The tracemalloc.start() function can also be called at runtime to start tracing Python memory allocations.

By default, a trace of an allocated memory block only stores the most recent frame (1 frame). To store more frames at startup, set the PYTHONTRACEMALLOC environment variable to the desired number of frames, or use the -X tracemalloc=nframes command-line option.

Here's an example of how to use Tracemalloc to profile memory usage:

Python

Import tracemalloc

Tracemalloc.start()

Run your application ...

Snapshot = tracemalloc.take_snapshot()

Top_stats = snapshot.statistics('lineno')

Print("[ Top 10 ]")

For stat in top_stats[:10]:

Print(stat)

This code will display the top 10 files allocating the most memory.

You can also use Tracemalloc to compute differences between two snapshots:

Python

Import tracemalloc

Tracemalloc.start()

Start your application ...

Snapshot1 = tracemalloc.take_snapshot()

Call the function leaking memory ...

Snapshot2 = tracemalloc.take_snapshot()

Top_stats = snapshot2.compare_to(snapshot1, 'lineno')

Print("[ Top 10 differences ]")

For stat in top_stats[:10]:

Print(stat)

This code will display the top 10 differences in memory usage between the two snapshots.

Tracemalloc also provides a way to get the traceback of a memory block:

Python

Import tracemalloc

Store 25 frames

Tracemalloc.start(25)

Run your application ...

Snapshot = tracemalloc.take_snapshot()

Top_stats = snapshot.statistics('traceback')

Pick the biggest memory block

Stat = top_stats[0]

Print("%s memory blocks: %.1f KiB" % (stat.count, stat.size / 1024))

For line in stat.traceback.format():

Print(line)

This code will display the traceback of the biggest memory block, including the number of memory blocks and the total size in KiB.

Tracemalloc also provides functions to get the current and peak size of all traced memory blocks: tracemalloc.get_traced_memory(). You can use tracemalloc.reset_peak() to set the peak size to the current size, and tracemalloc.clear_traces() to clear all traces of memory blocks allocated by Python.

Using the psutil library

The psutil library is a powerful tool for monitoring system performance and resource utilization in Python. It offers cross-platform compatibility, supporting Windows, Linux, macOS, and more. With psutil, you can retrieve information about CPU, memory, disks, and network usage. Here's an overview of how to use psutil for memory monitoring:

Installation

First, you need to install the psutil library. You can use the following commands depending on your operating system:

Python

For Linux

Sudo pip install psutil

For Windows

Pip install psutil

Basic Usage

To get started with psutil, import the library and explore its functions:

Python

Import psutil

Monitoring Memory Usage

Psutil provides the `virtual_memory()` function to retrieve information about memory usage. This function returns a named tuple containing various memory-related metrics:

Python

Memory_usage = psutil.virtual_memory()

Print(f"Memory Usage: {memory_usage.percent}%")

The `virtual_memory()` function provides the following fields:

• `total`: Total memory excluding swap.
• `available`: Available memory for processes.
• `percent`: Memory usage as a percentage.
• `used`: The amount of memory currently in use.
• `free`: Memory that is not used and readily available.

You can access these fields directly from the `virtual_memory()` result:

Python

Print('Total Memory:', psutil.virtual_memory().total)

Print('Available Memory:', psutil.virtual_memory().available)

Advanced Usage

Psutil also offers more advanced features for memory monitoring. For example, you can monitor the memory usage of a specific process:

Python

Import os

Import psutil

Get the process ID of the current Python script

Pid = os.getpid()

Get memory information for the process

Process = psutil.Process(pid)

Mem_info = process.memory_info()

Print the memory usage in bytes

Print('Memory Usage:', mem_info.rss)

In this example, we use `os.getpid()` to get the process ID (PID) of the current Python script. Then, we create a `psutil.Process` object for that PID and retrieve its memory information using the `memory_info()` method. The `memory_info()` method returns a tuple containing various memory metrics, including the resident set size (`rss), which represents the memory usage of the process in bytes.

The psutil library is a versatile and easy-to-use tool for monitoring system performance and resource utilization in Python. It provides a wide range of functions to retrieve information about CPU, memory, disks, and network usage. By utilizing psutil, you can effectively monitor memory usage, track specific process memory consumption, and optimize your Python applications for better performance.

Using the memory profiler from PyPI

The memory profiler from PyPI is a Python library module used for monitoring process memory. It uses the psutil code to create a decorator and then uses it to get the memory distribution. To install the memory profiler, use the following command:

Bash

Pip install -U memory_profiler

The memory profiler can be used to display the memory consumed by each line in the code. Implementation of finding the memory consumption is easy using the memory profiler as we directly call the decorator instead of writing a whole new code.

Python

Import the library

From memory_profiler import profile

Instantiate the decorator

@profile

Code for which memory has to be monitored

Def my_func():

X = [x for x in range(0, 1000)]

Y = [y * 100 for y in range(0, 1500)]

Del x

Return y

If __name__ == '__main__':

My_func()

The output will look something like this:

Line # Mem usage Increment Occurrences Line Contents

============================================================

2 30.5 MiB 30.5 MiB 1 @profile

3 def my_func():

4 30.6 MiB 0.1 MiB 1003 x = [x for x in range(0,1000)]

5 30.7 MiB 0.1 MiB 1503 y = [y*100 for y in range(0,1500)]

6 30.7 MiB 0.0 MiB 1 del x

7 30.7 MiB 0.0 MiB 1 return y

The first column represents the line number of the code that has been profiled, the second column (Mem usage) the memory usage of the Python interpreter after that line has been executed. The third column (Increment) represents the difference in memory of the current line with respect to the last one. The last column (Line Contents) prints the code that has been profiled.

Using the mprof package

The mprof package can be used to generate a full memory usage report of your Python script and to plot it. The package is available on PyPI and can be installed using pip:

Bash

Pip install memory-profiler

To use mprof, run the following commands:

Bash

Mprof run script.py

Mprof plot

The first command will run your Python script and record memory usage over time. The second command will generate a plot of your script's memory usage.

You can also use mprof to profile a specific function within your script. To do this, decorate the function with the @profile decorator and run your script with the -m memory_profiler option:

Python

From memory_profiler import profile

@profile

Def my_func():

# function code here

Then, run your script with the following command:

Bash

Python -m memory_profiler script.py

This will output the line-by-line memory usage of your script, allowing you to identify which parts of your code are using the most memory.

Mprof also provides options for tracking memory usage in multiprocessing contexts and setting debugger breakpoints based on memory usage thresholds.

Frequently asked questions