{"Awaiting Registry": 1, "Failed to lookup WHOIS": 48, "Got WHOIS": 544, "Host": "172.30.0.121", "Timestamp": "2016-04-26T17:35:00.048931", "Within Known CIDR Block": 208} {"Awaiting Registry": 1, "Failed to lookup WHOIS": 49, "Got WHOIS": 552, "Host": "172.30.0.124", "Timestamp": "2016-04-26T17:35:00.046517", "Within Known CIDR Block": 243} {"Awaiting Registry": 1, "Failed to lookup WHOIS": 50, "Got WHOIS": 535, "Host": "172.30.0.143", "Timestamp": "2016-04-26T17:35:00.057408", "Within Known CIDR Block": 186} {"Awaiting Registry": 1, "Failed to lookup WHOIS": 50, "Got WHOIS": 554, "Host": "172.30.0.249", "Timestamp": "2016-04-26T17:35:00.008689", "Within Known CIDR Block": 242} {"Awaiting Registry": 1, "Failed to lookup WHOIS": 51, "Got WHOIS": 584, "Host": "172.30.0.20", "Timestamp": "2016-04-26T17:35:00.047658", "Within Known CIDR Block": 202} {"Awaiting Registry": 1, "Failed to lookup WHOIS": 54, "Got WHOIS": 481, "Host": "172.30.0.250", "Timestamp": "2016-04-26T17:35:00.058991", "Within Known CIDR Block": 212} {"Awaiting Registry": 1, "Failed to lookup WHOIS": 54, "Got WHOIS": 525, "Host": "172.30.0.238", "Timestamp": "2016-04-26T17:35:00.057304", "Within Known CIDR Block": 243} {"Awaiting Registry": 1, "Failed to lookup WHOIS": 54, "Got WHOIS": 525, "Host": "172.30.0.240", "Looking up WHOIS": 1, "Timestamp": "2016-04-26T17:35:00.033937", "Within Known CIDR Block": 253} {"Awaiting Registry": 1, "Failed to lookup WHOIS": 56, "Got WHOIS": 511, "Host": "172.30.0.236", "Timestamp": "2016-04-26T17:35:00.058570", "Within Known CIDR Block": 222} {"Awaiting Registry": 1, "Failed to lookup WHOIS": 57, "Got WHOIS": 492, "Host": "172.30.0.239", "Timestamp": "2016-04-26T17:35:00.058613", "Within Known CIDR Block": 184} {"Awaiting Registry": 1, "Failed to lookup WHOIS": 57, "Got WHOIS": 545, "Host": "172.30.0.170", "Timestamp": "2016-04-26T17:35:00.058690", "Within Known CIDR Block": 213} {"Awaiting Registry": 1, "Failed to lookup WHOIS": 57, "Got WHOIS": 569, "Host": "172.30.0.117", "Looking up WHOIS": 1, "Timestamp": "2016-04-26T17:35:00.062300", "Within Known CIDR Block": 219} {"Awaiting Registry": 1, "Failed to lookup WHOIS": 59, "Got WHOIS": 566, "Host": "172.30.0.112", "Timestamp": "2016-04-26T17:35:00.054850", "Within Known CIDR Block": 229} {"Awaiting Registry": 1, "Failed to lookup WHOIS": 60, "Got WHOIS": 515, "Host": "172.30.0.88", "Looking up WHOIS": 2, "Timestamp": "2016-04-26T17:35:00.022801", "Within Known CIDR Block": 238} {"Awaiting Registry": 1, "Failed to lookup WHOIS": 60, "Got WHOIS": 561, "Host": "172.30.0.13", "Timestamp": "2016-04-26T17:35:00.038099", "Within Known CIDR Block": 210} {"Awaiting Registry": 1, "Failed to lookup WHOIS": 61, "Got WHOIS": 520, "Host": "172.30.0.136", "Timestamp": "2016-04-26T17:35:00.057452", "Within Known CIDR Block": 237} {"Awaiting Registry": 1, "Failed to lookup WHOIS": 61, "Got WHOIS": 553, "Host": "172.30.0.71", "Looking up WHOIS": 1, "Timestamp": "2016-04-26T17:35:00.034859", "Within Known CIDR Block": 259} {"Awaiting Registry": 1, "Failed to lookup WHOIS": 8, "Got WHOIS": 89, "Host": "172.30.0.11", "Timestamp": "2016-04-26T17:35:00.047294", "Within Known CIDR Block": 54} {"Host": "172.30.0.109", "Timestamp": "2016-04-26T17:35:00.059843"} {"Host": "172.30.0.135", "Timestamp": "2016-04-26T17:35:00.058640"} {"Host": "172.30.0.15", "Timestamp": "2016-04-26T17:35:00.060111"} {"Host": "172.30.0.167", "Timestamp": "2016-04-26T17:35:00.059880"} {"Host": "172.30.0.188", "Timestamp": "2016-04-26T17:35:00.060530"} {"Host": "172.30.0.193", "Timestamp": "2016-04-26T17:35:00.028907"} {"Host": "172.30.0.45", "Timestamp": "2016-04-26T17:35:00.059367"} {"Host": "172.30.0.47", "Timestamp": "2016-04-26T17:35:00.059336"} {"Host": "172.30.0.50", "Timestamp": "2016-04-26T17:35:00.059753"} {"Host": "172.30.0.6", "Timestamp": "2016-04-26T17:35:00.059641"} {"Host": "172.30.0.62", "Timestamp": "2016-04-26T17:35:00.059618"} {"Host": "172.30.0.69", "Timestamp": "2016-04-26T17:35:00.059966"} {"Host": "172.30.0.86", "Timestamp": "2016-04-26T17:35:00.060063"} If a line didnt have at least a single "Awaiting Registry" entry then it means it hasnt even collected its initial 1,000 IP addresses..This would turn out to be a problem where the coordinator was so busy with the CIDR hit endpoint that connections to all other Django endpoints were beginning to time out..Whats worse is an early version of the workers code would raise a time out exception and then it would stop..Having this process running via Supervisor would mean Supervisor would have a go at re-starting the process and possibly add some resiliency to this process..Heres the output of top on the coordinator at one point during this exercise..top – 18:21:27 up 1:49, 1 user, load average: 4.56, 2.84, 3.14 Tasks: 119 total, 3 running, 115 sleeping, 0 stopped, 1 zombie %Cpu0 : 31.6 us, 4.4 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 64.0 st KiB Mem: 2048516 total, 1922796 used, 125720 free, 110600 buffers KiB Swap: 0 total, 0 used, 0 free..1165020 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31495 ubuntu 20 0 775088 83276 4760 S 76.9 4.1 56:28.14 /home/ubuntu/.ips/bin/python manage.py + 4896 ubuntu 20 0 1905688 196972 12428 S 4.7 9.6 2:14.61 java -Xmx1G -Xms1G -server -XX:+UseParN+ 3524 ubuntu 20 0 1323452 95312 12324 S 3.3 4.7 0:49.79 /usr/lib/jvm/java-7-oracle/bin/java -Xm+ 31517 ubuntu 20 0 392488 34884 4272 S 2.7 1.7 1:18.73 python manage.py collect_whois ….And heres the output of top on a worker..As you can see its underutilised..top – 18:26:34 up 1:46, 1 user, load average: 0.00, 0.01, 0.05 Tasks: 143 total, 2 running, 141 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.2 us, 0.0 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 8175632 total, 1938020 used, 6237612 free, 111492 buffers KiB Swap: 0 total, 0 used, 0 free..655132 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31170 ubuntu 20 0 23716 1864 1280 R 0.3 0.0 0:00.05 top 26735 ubuntu 20 0 105632 1880 896 S 0.0 0.0 0:00.00 sshd: ubuntu@notty 26737 ubuntu 20 0 11152 932 692 S 0.0 0.0 0:00.00 bash -c cd /home/ubuntu/ips &&….26742 ubuntu 20 0 121116 39464 4404 S 0.0 0.5 0:06.02 python manage.py celeryd –concurrency=30 26747 ubuntu 20 0 426108 44460 3596 S 0.0 0.5 0:00.50 python manage.py celeryd –concurrency=30 …. More details