The problem is that pathspider is unable to finish, but also that it will not terminate after throwing the out of memory error.
Running Pathspider on a digital ocean VM with 2gb of ram and Debian 9.
Pathspider is invocated via a python script using subprocess.call().
Executing the webtest.ndjson in examples/webtest.ndjson works flawlessly.
The allocatet memory starts at around 200mb and slowly increases with time.
There has been similar problems with os.fork() that when a new proces is instanciated it allocates the same memory as the parent process in case it would need it.
See:https://stackoverflow.com/questions/1367373/python-subprocess-popen-oserror-errno-12-cannot-allocate-memory
The number of filedescriptors is always around 730. As exeeding this limit can also throw the same error.
stderr is:
WARNING: Failed to execute tcpdump. Check it is installed and in the PATH
INFO:pathspider:activating spider...
INFO:pathspider:starting pathspider
INFO:pathspider:Creating observer
INFO:pathspider:opening output file Gm8h6-fra1-ecn
ERROR:pathspider:exception occurred. terminating.
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/pathspider-2.0.0.dev0-py3.5.egg/pathspider/base.py", line 403, in exception_wrapper
target(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/pathspider-2.0.0.dev0-py3.5.egg/pathspider/sync.py", line 46, in configurator
self.configurations[config](self)
File "/usr/local/lib/python3.5/dist-packages/pathspider-2.0.0.dev0-py3.5.egg/pathspider/plugins/ecn.py", line 44, in config_ecn
stderr=subprocess.DEVNULL)
File "/usr/lib/python3.5/subprocess.py", line 266, in check_call
retcode = call(*popenargs, **kwargs)
File "/usr/lib/python3.5/subprocess.py", line 247, in call
with Popen(*popenargs, **kwargs) as p:
File "/usr/lib/python3.5/subprocess.py", line 676, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.5/subprocess.py", line 1221, in _execute_child
restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory
INFO:pathspider:terminating pathspider
INFO:observer:processed 6038714 packets (24943 dropped, 0 short, 3535 non-ip) into 1053009 flows (0 ignored)
The relevant files on the machine are:
root@ps-fra1-ecn:~# ls -l
-rw-r--r-- 1 root root 773 Sep 12 21:01 config.json
-rw-r--r-- 1 root root 252236255 Sep 13 15:31 Gm8h6-fra1-ecn
-rw-r--r-- 1 root root 191778315 Sep 12 21:01 input.ndjson
-rw-r--r-- 1 root root 2877 Sep 12 21:00 install_script.py
drwxr-xr-x 9 root root 4096 Sep 12 21:01 pathspider
drwxr-xr-x 6 root root 4096 Sep 12 21:01 python-libtrace
-rw-r--r-- 1 root root 2432 Sep 12 21:00 run_pathspider.py
-rw-r--r-- 1 root root 1432 Sep 13 15:30 stderr1.txt
and the system memory
root@ps-fra1-ecn:~# free -h
total used free shared buff/cache available
Mem: 2.0G 1.8G 75M 6.9M 113M 55M
Swap: 0B 0B 0B
Also pathspider is not terminating after the error occurs
root@ps-fra1-ecn:~# ps aux | grep pspdr
root 3356 1.9 86.0 3126620 1766620 ? Sl Sep12 24:58 /usr/bin/python3 /usr/local/bin/pspdr measure -i eth0 --input input.ndjson --output Gm8h6-fra1-ecn -w 80 ecn
root 3389 0.8 2.0 432556 42500 ? Sl Sep12 11:12 /usr/bin/python3 /usr/local/bin/pspdr measure -i eth0 --input input.ndjson --output Gm8h6-fra1-ecn -w 80 ecn
root 21682 0.0 0.0 12720 896 pts/0 S+ 18:27 0:00 grep pspdr
The problem is that pathspider is unable to finish, but also that it will not terminate after throwing the out of memory error.
Running Pathspider on a digital ocean VM with 2gb of ram and Debian 9.
Pathspider is invocated via a python script using subprocess.call().
Executing the webtest.ndjson in examples/webtest.ndjson works flawlessly.
The allocatet memory starts at around 200mb and slowly increases with time.
There has been similar problems with os.fork() that when a new proces is instanciated it allocates the same memory as the parent process in case it would need it.
See:https://stackoverflow.com/questions/1367373/python-subprocess-popen-oserror-errno-12-cannot-allocate-memory
The number of filedescriptors is always around 730. As exeeding this limit can also throw the same error.
stderr is:
The relevant files on the machine are:
and the system memory
Also pathspider is not terminating after the error occurs