 |
VOOZH | about |
|
VOOZH | about |
New in version 3.4.
The tracemalloc module is a debug tool to trace memory blocks allocated by Python. It provides the following information:
To trace most memory blocks allocated by Python, the module should be started as early as possible by setting the environment variable to , or by using -X command line option. The function can be called at runtime to start tracing Python memory allocations.
By default, a trace of an allocated memory block only stores the most recent frame (1 frame). To store 25 frames at startup: set the environment variable to , or use the -X command line option.
Display the 10 files allocating the most memory:
import tracemalloc tracemalloc.start() # ... run your application ... snapshot = tracemalloc.take_snapshot() top_stats = snapshot.statistics('lineno') print("[ Top 10 ]") for stat in top_stats[:10]: print(stat)
Example of output of the Python test suite:
[ Top 10 ] <frozen importlib._bootstrap>:716: size=4855 KiB, count=39328, average=126 B <frozen importlib._bootstrap>:284: size=521 KiB, count=3199, average=167 B /usr/lib/python3.4/collections/__init__.py:368: size=244 KiB, count=2315, average=108 B /usr/lib/python3.4/unittest/case.py:381: size=185 KiB, count=779, average=243 B /usr/lib/python3.4/unittest/case.py:402: size=154 KiB, count=378, average=416 B /usr/lib/python3.4/abc.py:133: size=88.7 KiB, count=347, average=262 B <frozen importlib._bootstrap>:1446: size=70.4 KiB, count=911, average=79 B <frozen importlib._bootstrap>:1454: size=52.0 KiB, count=25, average=2131 B <string>:5: size=49.7 KiB, count=148, average=344 B /usr/lib/python3.4/sysconfig.py:411: size=48.0 KiB, count=1, average=48.0 KiB
We can see that Python loaded data (bytecode and constants) from modules and that the module allocated to build types.
See for more options.
Take two snapshots and display the differences:
import tracemalloc tracemalloc.start() # ... start your application ... snapshot1 = tracemalloc.take_snapshot() # ... call the function leaking memory ... snapshot2 = tracemalloc.take_snapshot() top_stats = snapshot2.compare_to(snapshot1, 'lineno') print("[ Top 10 differences ]") for stat in top_stats[:10]: print(stat)
Example of output before/after running some tests of the Python test suite:
[ Top 10 differences ] <frozen importlib._bootstrap>:716: size=8173 KiB (+4428 KiB), count=71332 (+39369), average=117 B /usr/lib/python3.4/linecache.py:127: size=940 KiB (+940 KiB), count=8106 (+8106), average=119 B /usr/lib/python3.4/unittest/case.py:571: size=298 KiB (+298 KiB), count=589 (+589), average=519 B <frozen importlib._bootstrap>:284: size=1005 KiB (+166 KiB), count=7423 (+1526), average=139 B /usr/lib/python3.4/mimetypes.py:217: size=112 KiB (+112 KiB), count=1334 (+1334), average=86 B /usr/lib/python3.4/http/server.py:848: size=96.0 KiB (+96.0 KiB), count=1 (+1), average=96.0 KiB /usr/lib/python3.4/inspect.py:1465: size=83.5 KiB (+83.5 KiB), count=109 (+109), average=784 B /usr/lib/python3.4/unittest/mock.py:491: size=77.7 KiB (+77.7 KiB), count=143 (+143), average=557 B /usr/lib/python3.4/urllib/parse.py:476: size=71.8 KiB (+71.8 KiB), count=969 (+969), average=76 B /usr/lib/python3.4/contextlib.py:38: size=67.2 KiB (+67.2 KiB), count=126 (+126), average=546 B
We can see that Python has loaded of module data (bytecode and constants), and that this is more than had been loaded before the tests, when the previous snapshot was taken. Similarly, the module has cached of Python source code to format tracebacks, all of it since the previous snapshot.
If the system has little free memory, snapshots can be written on disk using the method to analyze the snapshot offline. Then use the method reload the snapshot.
Code to display the traceback of the biggest memory block:
import tracemalloc # Store 25 frames tracemalloc.start(25) # ... run your application ... snapshot = tracemalloc.take_snapshot() top_stats = snapshot.statistics('traceback') # pick the biggest memory block stat = top_stats[0] print("%s memory blocks: %.1f KiB" % (stat.count, stat.size / 1024)) for line in stat.traceback.format(): print(line)
Example of output of the Python test suite (traceback limited to 25 frames):
903 memory blocks: 870.1 KiB File "<frozen importlib._bootstrap>", line 716 File "<frozen importlib._bootstrap>", line 1036 File "<frozen importlib._bootstrap>", line 934 File "<frozen importlib._bootstrap>", line 1068 File "<frozen importlib._bootstrap>", line 619 File "<frozen importlib._bootstrap>", line 1581 File "<frozen importlib._bootstrap>", line 1614 File "/usr/lib/python3.4/doctest.py", line 101 import pdb File "<frozen importlib._bootstrap>", line 284 File "<frozen importlib._bootstrap>", line 938 File "<frozen importlib._bootstrap>", line 1068 File "<frozen importlib._bootstrap>", line 619 File "<frozen importlib._bootstrap>", line 1581 File "<frozen importlib._bootstrap>", line 1614 File "/usr/lib/python3.4/test/support/__init__.py", line 1728 import doctest File "/usr/lib/python3.4/test/test_pickletools.py", line 21 support.run_doctest(pickletools) File "/usr/lib/python3.4/test/regrtest.py", line 1276 test_runner() File "/usr/lib/python3.4/test/regrtest.py", line 976 display_failure=not verbose) File "/usr/lib/python3.4/test/regrtest.py", line 761 match_tests=ns.match_tests) File "/usr/lib/python3.4/test/regrtest.py", line 1563 main() File "/usr/lib/python3.4/test/__main__.py", line 3 regrtest.main_in_temp_cwd() File "/usr/lib/python3.4/runpy.py", line 73 exec(code, run_globals) File "/usr/lib/python3.4/runpy.py", line 160 "__main__", fname, loader, pkg_name)
We can see that the most memory was allocated in the module to load data (bytecode and constants) from modules: . The traceback is where the loaded data most recently: on the line of the module. The traceback may change if a new module is loaded.
Code to display the 10 lines allocating the most memory with a pretty output, ignoring and files:
import linecache import os import tracemalloc def display_top(snapshot, group_by='lineno', limit=10): snapshot = snapshot.filter_traces(( tracemalloc.Filter(False, "<frozen importlib._bootstrap>"), tracemalloc.Filter(False, "<unknown>"), )) top_stats = snapshot.statistics(group_by) print("Top %s lines" % limit) for index, stat in enumerate(top_stats[:limit], 1): frame = stat.traceback[0] # replace "/path/to/module/file.py" with "module/file.py" filename = os.sep.join(frame.filename.split(os.sep)[-2:]) print("#%s: %s:%s: %.1f KiB" % (index, filename, frame.lineno, stat.size / 1024)) line = linecache.getline(frame.filename, frame.lineno).strip() if line: print(' %s' % line) other = top_stats[limit:] if other: size = sum(stat.size for stat in other) print("%s other: %.1f KiB" % (len(other), size / 1024)) total = sum(stat.size for stat in top_stats) print("Total allocated size: %.1f KiB" % (total / 1024)) tracemalloc.start() # ... run your application ... snapshot = tracemalloc.take_snapshot() display_top(snapshot)
Example of output of the Python test suite:
Top 10 lines #1: Lib/base64.py:414: 419.8 KiB _b85chars2 = [(a + b) for a in _b85chars for b in _b85chars] #2: Lib/base64.py:306: 419.8 KiB _a85chars2 = [(a + b) for a in _a85chars for b in _a85chars] #3: collections/__init__.py:368: 293.6 KiB exec(class_definition, namespace) #4: Lib/abc.py:133: 115.2 KiB cls = super().__new__(mcls, name, bases, namespace) #5: unittest/case.py:574: 103.1 KiB testMethod() #6: Lib/linecache.py:127: 95.4 KiB lines = fp.readlines() #7: urllib/parse.py:476: 71.8 KiB for a in _hexdig for b in _hexdig} #8: <string>:5: 62.0 KiB #9: Lib/_weakrefset.py:37: 60.0 KiB self.data = set() #10: Lib/base64.py:142: 59.8 KiB _b32tab2 = [a + b for a in _b32tab for b in _b32tab] 6220 other: 3602.8 KiB Total allocated size: 5303.1 KiB
See for more options.
Clear traces of memory blocks allocated by Python.
See also .
Get the traceback where the Python object obj was allocated. Return a instance, or if the module is not tracing memory allocations or did not trace the allocation of the object.
See also and functions.
Get the maximum number of frames stored in the traceback of a trace.
The module must be tracing memory allocations to get the limit, otherwise an exception is raised.
The limit is set by the function.
Get the current size and peak size of memory blocks traced by the module as a tuple: .
Get the memory usage in bytes of the module used to store traces of memory blocks. Return an .
if the module is tracing Python memory allocations, otherwise.
See also and functions.
Start tracing Python memory allocations: install hooks on Python memory allocators. Collected tracebacks of traces will be limited to nframe frames. By default, a trace of a memory block only stores the most recent frame: the limit is . nframe must be greater or equal to .
Storing more than frame is only useful to compute statistics grouped by or to compute cumulative statistics: see the and methods.
Storing more frames increases the memory and CPU overhead of the module. Use the function to measure how much memory is used by the module.
The environment variable () and the -X command line option can be used to start tracing at startup.
See also , and functions.
Stop tracing Python memory allocations: uninstall hooks on Python memory allocators. Also clears all previously collected traces of memory blocks allocated by Python.
Call function to take a snapshot of traces before clearing them.
See also , and functions.
Take a snapshot of traces of memory blocks allocated by Python. Return a new instance.
The snapshot does not include memory blocks allocated before the module started to trace memory allocations.
Tracebacks of traces are limited to frames. Use the nframe parameter of the function to store more frames.
The module must be tracing memory allocations to take a snapshot, see the function.
See also the function.
Filter on traces of memory blocks.
See the function for the syntax of filename_pattern. The and file extensions are replaced with .
Examples:
If inclusive is (include), only trace memory blocks allocated in a file with a name matching at line number .
If inclusive is (exclude), ignore memory blocks allocated in a file with a name matching at line number .
Line number () of the filter. If lineno is , the filter matches any line number.
Filename pattern of the filter ().
If all_frames is , all frames of the traceback are checked. If all_frames is , only the most recent frame is checked.
This attribute has no effect if the traceback limit is . See the function and attribute.
Snapshot of traces of memory blocks allocated by Python.
The function creates a snapshot instance.
Compute the differences with an old snapshot. Get statistics as a sorted list of instances grouped by group_by.
See the method for group_by and cumulative parameters.
The result is sorted from the biggest to the smallest by: absolute value of , , absolute value of , and then by .
Write the snapshot into a file.
Use to reload the snapshot.
Create a new instance with a filtered sequence, filters is a list of instances. If filters is an empty list, return a new instance with a copy of the traces.
All inclusive filters are applied at once, a trace is ignored if no inclusive filters match it. A trace is ignored if at least one exclusive filter matches it.
Load a snapshot from a file.
See also .
Get statistics as a sorted list of instances grouped by group_by:
| group_by | description |
|---|---|
| filename | |
| filename and line number | |
| traceback |
If cumulative is , cumulate size and count of memory blocks of all frames of the traceback of a trace, not only the most recent frame. The cumulative mode can only be used with group_by equals to and .
The result is sorted from the biggest to the smallest by: , and then by .
Maximum number of frames stored in the traceback of : result of the when the snapshot was taken.
Traces of all memory blocks allocated by Python: sequence of instances.
The sequence has an undefined order. Use the method to get a sorted list of statistics.
Statistic difference on memory allocations between an old and a new instance.
returns a list of instances. See also the class.
Number of memory blocks in the new snapshot (): if the memory blocks have been released in the new snapshot.
Difference of number of memory blocks between the old and the new snapshots (): if the memory blocks have been allocated in the new snapshot.
Total size of memory blocks in bytes in the new snapshot (): if the memory blocks have been released in the new snapshot.
Difference of total size of memory blocks in bytes between the old and the new snapshots (): if the memory blocks have been allocated in the new snapshot.
Traceback where the memory blocks were allocated, instance.
Sequence of instances sorted from the most recent frame to the oldest frame.
A traceback contains at least frame. If the module failed to get a frame, the filename at line number is used.
When a snapshot is taken, tracebacks of traces are limited to frames. See the function.
The attribute is an instance of instance.
Format the traceback as a list of lines with newlines. Use the module to retrieve lines from the source code. If limit is set, only format the limit most recent frames.
Similar to the function, except that does not include newlines.
Example:
print("Traceback (most recent call first):") for line in traceback: print(line)
Output:
Traceback (most recent call first): File "test.py", line 9 obj = Object() File "test.py", line 12 tb = tracemalloc.get_object_traceback(f())
27.6. — Trace or track Python statement execution
28. Software Packaging and Distribution
