When starting incrond now, I see zillions of incrond processes and quite soon afterwards a load of 1000 and more. I did not see such a strange behaviour with incron-0.5.10-8.el7 before.
Bodhi is disabling automatic push to stable due to negative karma. The maintainer may push manually if they determine that the issue is not severe.
$ rpm -q incron
incron-0.5.12-9.el7.x86_64
$ for i in $(seq 1 250); do date > /var/www/html/${i}.txt; done
$ for i in $(seq 1 50); do mkdir -p /var/www/html/${i}; done
$ cat /etc/incron.d/reproducer.conf
/var/www/html/ IN_CLOSE_WRITE,IN_NO_LOOP /root/bin/update.sh "$@$#"
/var/www/html/10/ IN_CLOSE_WRITE,IN_NO_LOOP /root/bin/update.sh "$@$#"
$ cat /root/bin/update.sh
sed -e 's/2019/2020/g' -i "$1"
How to trigger the zillions of incrond processes that will eat the system sooner or later?
sed -e 's/1/2/' -i /var/www/html/100.txt
Afterwards I see lots of incrond processes and when doing nothing but waiting some time, I end up with zillions of incrond processes and load 1000 and when waiting even longer, the system crashes or ends with a kernel panic of khugepaged. Nevertheless the system is dead and needs a hard reset.
Kevin: That's interesting! Moving the content from /etc/incron.d/reproducer.conf to /var/spool/incron/root (via incrontab -e) seems indeed to solve this. No more zillions of incrond processes yet! May we get the same fix also for system wide /etc/incron.d/, please?
Important: The IN_NO_LOOP option still does no longer work, the example above creates endless loops with user incrontab (even it's overall better than before).
This update has been submitted for batched by kevin.
This update has been submitted for testing by kevin.
This update has been pushed to testing.
Got bitten by this badly. This fix seems to fix it, as confirmed by not seeing hundreds of zombie processes after upgrading to it. Thanks!
When starting incrond now, I see zillions of incrond processes and quite soon afterwards a load of 1000 and more. I did not see such a strange behaviour with incron-0.5.10-8.el7 before.
Bodhi is disabling automatic push to stable due to negative karma. The maintainer may push manually if they determine that the issue is not severe.
robert: are you sure the new one was used? can you duplicate it?
This update has reached 14 days in testing and can be pushed to stable now if the maintainer wishes
Kevin: Yes, quite reliable:
How to trigger the zillions of incrond processes that will eat the system sooner or later?
Afterwards I see lots of incrond processes and when doing nothing but waiting some time, I end up with zillions of incrond processes and load 1000 and when waiting even longer, the system crashes or ends with a kernel panic of khugepaged. Nevertheless the system is dead and needs a hard reset.
robert: Can you move your incrontab from /etc/incron.d to a root user incrontab and see if the problem persists?
My theory is that this fixes only the user incrontab case, not the system wide /etc/incron.d/ case...
Kevin: That's interesting! Moving the content from
/etc/incron.d/reproducer.confto/var/spool/incron/root(viaincrontab -e) seems indeed to solve this. No more zillions of incrond processes yet! May we get the same fix also for system wide/etc/incron.d/, please?Important: The
IN_NO_LOOPoption still does no longer work, the example above creates endless loops with user incrontab (even it's overall better than before).This update has been submitted for batched by kevin.
This update has been submitted for stable by kevin.
This update has been pushed to stable.
I experienced piling up incron processes with the fixed version. I manually patched a fix to that from the original repo https://github.com/ar-/incron/pull/56/commits/0a8cb83983394c2bd36105fcae5c56f89e931686.
Seem to prevent the stacking up of tasks.
Process structure after a few minutes runtime.
/usr/sbin/incrond
_ /usr/sbin/incrond | _ /usr/sbin/incrond | _ /usr/sbin/incrond | _ /usr/sbin/incrond | _ /usr/sbin/incrond | | _ /usr/sbin/incrond | | | _ /usr/sbin/incrond | | | | _ /usr/sbin/incrond | | | _ /usr/sbin/incrond | | _ /usr/sbin/incrond | | | _ /usr/sbin/incrond | | _ /usr/sbin/incrond | | _ /usr/sbin/incrond | _ /usr/sbin/incrond _ /usr/sbin/incrond | _ /usr/sbin/incrond | _ /usr/sbin/incrond | _ /usr/sbin/incrond _ /usr/sbin/incrond _ /usr/sbin/incrond | _ /usr/sbin/incrond | _ /usr/sbin/incrond | _ /usr/sbin/incrond | | _ /usr/sbin/incrond | _ /usr/sbin/incrond | _ /usr/sbin/incrond | _ /usr/sbin/incrond _ /usr/sbin/incrond _ /usr/sbin/incrond | _ /usr/sbin/incrond | _ /usr/sbin/incrond | _ /usr/sbin/incrond | _ /usr/sbin/incrond _ /usr/sbin/incrond