Time and Memory Profiling for Python
Let's save some time and memory, coz both are expensive.
- Introduction
- Premature Optimization is evil.
- Magic functions and tools
- Timeit
- Timing Profiling with Lprun
- Memory Profiling with mprun
- Memory Leakage using pympler
- Read More
Introduction
Every Developer desires to make their code optimzied and efficient. A dream where the developers want that their code to execute faster, with no memory leakage on the production system.
Let's make this dream true...
Creating Data-processing pipeline, writing new algorithms, Deploying Machine Learning models to server million users, Scientific calculation in astropyhsics, these are few areas where when we write code we want to profile every single line of code for two things
- The amount of time it is taking to execute where our goal is to reduce the time taken a.k.a Time Complexity.
- The memory consumption for execution of that code where our goal is to reduce the memory usage a.k.a Space complexity.
There always a trade-off between both of them some time we are fine with memory consumption but not with the time it takes and vice-versa based on the needs we check for the trade-off, but the best system is where we can reduce both space and time complexity.
Premature Optimization is evil.
Early in developing Algorithms we should think less about these things because it can be counter-productive which can lead to premature optimization and its the root cause of all evil.
So first make it work then optimize it.
Magic functions and tools
While most of the data science experiments starts in Ipython Notebook. The Ipython enviroment gives us some magic functions which can be utilized to profile our code.
- %%timeit: Measuring time taken for the codeblock to run
- %lprun: Run code with the line-by-line profiler
- %mprun: Run code with the line-by-line memory profiler
For Tracing Memory Leakage we can use Pympler.
import numpy as np
number = np.random.randint(0,100,10000)
%%timeit
total = 0
for i in number:
total+=i
%%timeit
number.sum()
The difference in the execution time is evident one.
Non-vectorized code in Milliseconds, 10-3.
Vectorized code in Microseconds, 10-6.
Vectorized code is the winner here.
Timing Profiling with Lprun
line_profiler is a package for doing line-by-line timing profiling of functions.
Install using
pip install line_profiler
Python provides a builtin profiler, but we will be using Line profiler for reasons stated below.
The current profiling tools supported in Python 2.7 and later only time function calls. This is a good first step for locating hotspots in one's program and is frequently all one needs to do to optimize the program. However, sometimes the cause of the hotspot is actually a single line in the function, and that line may not be obvious from just reading the source code. These cases are particularly frequent in scientific computing. Functions tend to be larger (sometimes because of legitimate algorithmic complexity, sometimes because the programmer is still trying to write FORTRAN code), and a single statement without function calls can trigger lots of computation when using libraries like numpy. cProfile only times explicit function calls, not special methods called because of syntax. Consequently, a relatively slow numpy operation on large arrays like this,
a[large_index_array] = some_other_large_array
is a hotspot that never gets broken out by cProfile because there is no explicit function call in that statement.
LineProfiler can be given functions to profile, and it will time the execution of each individual line inside those functions. In a typical workflow, one only cares about line timings of a few functions because wading through the results of timing every single line of code would be overwhelming. However, LineProfiler does need to be explicitly told what functions to profile.
# once installed we have load the extension
%load_ext line_profiler
def some_operation(x):
x = x **2
x = x +2
x = np.concatenate([x,x,x],axis=0)
return x
Now the %lprun command will do a line-by-line profiling of any function–in this case, we need to tell it explicitly which functions we're interested in profiling:
%lprun -f some_operation some_operation(np.random.randn(100))
Timer unit: 1e-06 s
Total time: 7.7e-05 s
File: <ipython-input-30-80aca4fcfa96>
Function: some_operation at line 1
Line # Hits Time Per Hit % Time Line Contents
==============================================================
1 def some_operation(x):
2 1 24.0 24.0 31.2 x = x **2
3 1 22.0 22.0 28.6 x = x +2
4 1 30.0 30.0 39.0 x = np.concatenate([x,x,x],axis=0)
5 1 1.0 1.0 1.3 return xThe source code of the function is printed with the timing information for each line. There are six columns of information.
- Line : The line number in the file.
- Hits: The number of times that line was executed.
- Time: The total amount of time spent executing the line in the timer's units. In the header information before the tables, you will see a line "Timer unit:" giving the conversion factor to seconds. It may be different on different systems.
- Per Hit: The average amount of time spent executing the line once in the timer's units.
- % Time: The percentage of time spent on that line relative to the total amount of recorded time spent in the function.
- Line Contents: The actual source code. Note that this is always read from disk when the formatted results are viewed, not when the code was executed. If you have edited the file in the meantime, the lines will not match up, and the formatter may not even be able to locate the function for display.
Memory Profiling with mprun
This is a python module for monitoring memory consumption of a process as well as line-by-line analysis of memory consumption for python programs.
Install
pip install -U memory_profiler
The only issue mprun doesn't work on notebook rather on a python file so we will write the code in notebook %%file magic function we will write that into a file and execute mprun on it
%load_ext memory_profiler
%%file mprun.py
import numpy as np
def some_operation(x):
y = x **2
z = y +2
result = np.concatenate([x,y,z],axis=0)
return result
from mprun import some_operation
%mprun -f some_operation some_operation(np.random.randn(100000))
Line # Mem usage Increment Line Contents
================================================
2 62.5 MiB 62.5 MiB def some_operation(x):
3 63.3 MiB 0.8 MiB y = x **2
4 64.0 MiB 0.8 MiB z = y +2
5 66.3 MiB 2.3 MiB result = np.concatenate([x,y,z],axis=0)
6 66.3 MiB 0.0 MiB return result- The first column represents the line number of the code that has been profiled.
- The second column (Mem usage) the memory usage of the Python interpreter after that line has been executed.
- The third column (Increment) represents the difference in memory of the current line with respect to the last one.
- The last column (Line Contents) prints the code that has been profiled.
Memory Leakage using pympler
Pympler is a development tool to measure, monitor and analyze the memory behavior of Python objects in a running Python application.
By pympling a Python application, detailed insight in the size and the lifetime of Python objects can be obtained. Undesirable or unexpected runtime behavior like memory bloat and other “pymples” can easily be identified.
Pympler integrates three previously separate modules into a single, comprehensive profiling tool. The asizeof module provides basic size information for one or several Python objects, module muppy is used for on-line monitoring of a Python application and module Class Tracker provides off-line analysis of the lifetime of selected Python objects.
A web profiling frontend exposes process statistics, garbage visualisation and class tracker statistics.