My systemd service kept timing out because of how long it would take to boot up also, so this fixed it for me:
Edit your systemd file:
- For modern versions of
systemd: Runsystemctl edit --full node.service(replace "node" with your service name).- This will create a system file at
/etc/systemd/system/node.service.d/that will override the system file at/usr/lib/systemd/system/node.service. This is the proper way to configure your system files. More information about how to usesystemctl editis here.
- This will create a system file at
- Directly editing system file: The system file for me is at
/usr/lib/systemd/system/node.service. Replace "node" with your application name. However, it is not safe to directly edit files in/usr/lib/systemd/(See comments)
- For modern versions of
Use
TimeoutStartSec,TimeoutStopSecorTimeoutSec(more info here) to specify how long the timeout should be for starting & stopping the process. Afterwards, this is how my systemd file looked:[Unit] Description=MyProject Documentation=man:node(1) After=rc-local.service [Service] WorkingDirectory=/home/myproject/GUIServer/Server/ Environment="NODE_PATH=/usr/lib/node_modules" ExecStart=-/usr/bin/node Index.js Type=simple Restart=always KillMode=process TimeoutSec=900 [Install] WantedBy=multi-user.target- You can also view the current Timeout status by running any of these (but you'll need to edit your service to make changes! See step 1). Confusingly, the associated properties have a "U" in their name for microseconds. See this Github issue for more information:
systemctl show node.service -p TimeoutStartUSecsystemctl show node.service -p TimeoutStopUSecsystemctl show node.service -p TimeoutUSec
- You can also view the current Timeout status by running any of these (but you'll need to edit your service to make changes! See step 1). Confusingly, the associated properties have a "U" in their name for microseconds. See this Github issue for more information:
Next you'll need to reload the systemd with
systemctl reload node.serviceNow try to start your service with
systemctl start node.serviceIf that didn't work, try to reboot systemctl with
systemctl rebootIf that didn't work, try using the
--no-blockoption for systemctl like so:systemctl --no-block start node.service. This option is described here: "Do not synchronously wait for the requested operation to finish. If this is not specified, the job will be verified, enqueued and systemctl will wait until the unit's start-up is completed. By passing this argument, it is only verified and enqueued."- There is also the option to use
systemctl maskinstead ofsystemctl start. For more info see here.
- There is also the option to use
Updates from Comments:
TimeoutSec=infinity: Instead of using "infinity" here, put a large amount of time instead, likeTimeoutSec=900(15 min). If the application takes "forever" to exit, then it's possible that it will block a reboot indefinitely. Credit @Alexis Wilke and @JCCyC- Instead of editing
/usr/lib/systemd/system, trysystemctl editinstead or edit/etc/systemd/systemto override them instead. You should never edit service files in/usr/lib/. Credit @ryeager and @0xC0000022L
** Update from systemd source docs ** When specified "infinity" as a value to any of these timeout params, the timeout logic is disabled.
JobTimeoutSec=, JobRunningTimeoutSec=,TimeoutStartSec=, TimeoutAbortSec=
The default is "infinity" (job timeouts disabled), except for device units where JobRunningTimeoutSec= defaults to DefaultTimeoutStartSec=.
Reference: enter link description here
Similarly this logic applies to service level and laid out clearly in URL below. Reference: enter link description here
Answer from Katie on Stack ExchangeMy systemd service kept timing out because of how long it would take to boot up also, so this fixed it for me:
Edit your systemd file:
- For modern versions of
systemd: Runsystemctl edit --full node.service(replace "node" with your service name).- This will create a system file at
/etc/systemd/system/node.service.d/that will override the system file at/usr/lib/systemd/system/node.service. This is the proper way to configure your system files. More information about how to usesystemctl editis here.
- This will create a system file at
- Directly editing system file: The system file for me is at
/usr/lib/systemd/system/node.service. Replace "node" with your application name. However, it is not safe to directly edit files in/usr/lib/systemd/(See comments)
- For modern versions of
Use
TimeoutStartSec,TimeoutStopSecorTimeoutSec(more info here) to specify how long the timeout should be for starting & stopping the process. Afterwards, this is how my systemd file looked:[Unit] Description=MyProject Documentation=man:node(1) After=rc-local.service [Service] WorkingDirectory=/home/myproject/GUIServer/Server/ Environment="NODE_PATH=/usr/lib/node_modules" ExecStart=-/usr/bin/node Index.js Type=simple Restart=always KillMode=process TimeoutSec=900 [Install] WantedBy=multi-user.target- You can also view the current Timeout status by running any of these (but you'll need to edit your service to make changes! See step 1). Confusingly, the associated properties have a "U" in their name for microseconds. See this Github issue for more information:
systemctl show node.service -p TimeoutStartUSecsystemctl show node.service -p TimeoutStopUSecsystemctl show node.service -p TimeoutUSec
- You can also view the current Timeout status by running any of these (but you'll need to edit your service to make changes! See step 1). Confusingly, the associated properties have a "U" in their name for microseconds. See this Github issue for more information:
Next you'll need to reload the systemd with
systemctl reload node.serviceNow try to start your service with
systemctl start node.serviceIf that didn't work, try to reboot systemctl with
systemctl rebootIf that didn't work, try using the
--no-blockoption for systemctl like so:systemctl --no-block start node.service. This option is described here: "Do not synchronously wait for the requested operation to finish. If this is not specified, the job will be verified, enqueued and systemctl will wait until the unit's start-up is completed. By passing this argument, it is only verified and enqueued."- There is also the option to use
systemctl maskinstead ofsystemctl start. For more info see here.
- There is also the option to use
Updates from Comments:
TimeoutSec=infinity: Instead of using "infinity" here, put a large amount of time instead, likeTimeoutSec=900(15 min). If the application takes "forever" to exit, then it's possible that it will block a reboot indefinitely. Credit @Alexis Wilke and @JCCyC- Instead of editing
/usr/lib/systemd/system, trysystemctl editinstead or edit/etc/systemd/systemto override them instead. You should never edit service files in/usr/lib/. Credit @ryeager and @0xC0000022L
** Update from systemd source docs ** When specified "infinity" as a value to any of these timeout params, the timeout logic is disabled.
JobTimeoutSec=, JobRunningTimeoutSec=,TimeoutStartSec=, TimeoutAbortSec=
The default is "infinity" (job timeouts disabled), except for device units where JobRunningTimeoutSec= defaults to DefaultTimeoutStartSec=.
Reference: enter link description here
Similarly this logic applies to service level and laid out clearly in URL below. Reference: enter link description here
Running systemctl show SERVICE_NAME.service -p TimeoutStopUSec I could at least see the timeout set by systemd to my service.
I changed the script to a regular unit file one in order for it work properly.
TimeoutStartSec should be used. As @muru mentioned in the comment, the correct way to run a script that should exit is to use Type=oneshot instead of Type=simple.
For example, if your script is normally executed in under 10 seconds, your config would be:
[Service]
Type=oneshot
TimeoutStartSec=10
SyslogIdentifier=myservice
Environment='MYVAR=myvar'
User=deanresin
ExecStart=/usr/bin/python3 /home/deanresin/myscript
If the script takes more than 10 seconds to execute, it will be killed, and the service would show as failed.
Since systemd 229 (2016-02-11), the RuntimeMaxSec option can be used. This does not require (or even work with) Type=oneshot as it sets a time limit for the "active" state instead of "activating".
centos7 - How to change the systemd timeout in centos - Stack Overflow
Put time limit on systemd-tmpfiles-setup by default
how do I configure an end time for systemd timers - Stack Overflow
How to set up a systemd service to retry 5 times on a cycle of 30 seconds - Stack Overflow
Hey all, I was reading up on the systemd timeout parameters in the system.conf file (/etc/systemd/system.conf) and was curious if anyone has any input as to problems that can arise if we set the #DefaultTimeoutStopSec=90s to something like 5 seconds rather than letting it stay at 90 seconds.
This is something I do from time to time on systems if I get an SDDM timeout when shutting down the PC and never have had a problem but wondered if it could cause an issue other than the journal not logging.
systemctl daemon-reexec should solve your problem.
there are only a few properties that you can set thru "set-property" command.
you can get a version of your systemd thru command:
systemctl --version
then you need to update it to get latest commands like "cat" with:
sudo yum update systemd -y
after that you can see stuff that overrides your DefaultTimeoutStartSec=90s run:
systemctl cat <service>
Then you need to create a {service}.service file to override your settings with information from [Service] section from systemctl cat {service} command and restart daemon after that:
echo "[Service]
TimeoutSec=15min
ExecStart=/etc/rc.d/init.d/<service> start
ExecStop=/etc/rc.d/init.d/<service> stop
ExecReload=/etc/rc.d/init.d/<service> reload" > /lib/systemd/system/{service}.service
sudo systemctl daemon-reload
It is possible to terminate systemd services after a certain amount of time.
Use the RuntimeMaxSec option for Type=simple services:
[Unit]
Description=Terminate simple Service Test
[Service]
Type=simple
RuntimeMaxSec=5
ExecStart=/bin/sleep 10
Use the TimeoutStartSec for Type=oneshot services:
[Unit]
Description=Terminate oneshot Service Test
[Service]
Type=oneshot
TimeoutStartSec=5
ExecStart=/bin/sleep 10
Both services will be terminated after 5 seconds before the sleep timer reaches 10 seconds.
Details on RuntimeMaxSec and TimeoutStartSec can be found here:
https://www.freedesktop.org/software/systemd/man/systemd.service.html
I believe you can add the following to the service file's [Unit] directives:
JobTimeoutSec=43200
You could probably just write a wrapper script to terminate the process after 12 hours or use timeout 12h COMMAND [ARG]...
Partial solution. I basically set these values
StartLimitIntervalSec=1d
StartLimitBurst=5
Which limits 5 restarts per day.
You can specify
StartLimitIntervalSec=infinity
but please read the documentation for when the restart counter is reset.
Additionally it is reported that system reboot and systemctl daemon-reload also reset the counter.
To have a service restart 3 times at 90 second intervals include the following lines in your systemd service file:
[Unit]
StartLimitIntervalSec=400
StartLimitBurst=3
[Service]
Restart=always
RestartSec=90
Before systemd-230 it was called just StartLimitInterval:
[Unit]
StartLimitInterval=400
StartLimitBurst=3
[Service]
Restart=always
RestartSec=90
This worked worked for me for a service that runs a script using Type=idle. Note that StartLimitIntervalSec must be greater than RestartSec * StartLimitBurst otherwise the service will be restarted indefinitely.
It took me some time with a lot of trial and error to work out how systemd uses these options, which suggests that systemd isn't as well documented as one would hope. These options effectively provide the retry cycle time and maximum retries that I was looking for.
References: https://manpages.debian.org/testing/systemd/systemd.unit.5.en.html for Unit section https://manpages.debian.org/testing/systemd/systemd.exec.5.en.html for Service section
Some years later and with systemd 232 it dosn't work anymore as described in the question and in the answers from 2016. Option name StartLimitIntervalSec and Sections have changed. Now it has to look like this example:
[Unit]
StartLimitBurst=5
StartLimitIntervalSec=33
[Service]
Restart=always
RestartSec=5
ExecStart=/bin/sleep 6
This will do 5 restarts in 30 sec (5*6) plus one restart in 33 sec. So we have 6 restarts in 33 sec. This exceeds the limit of 5 restarts in 33 sec. So restarts will stop at 5 counts after about 31 sec.
I have a Debian 12 virtual machine with GNOME desktop. No fancy mods, no weird config, all I have in it is what the installer gave me + Wine Staging + one Windows app. The VM starts up lightning fast, and *sometimes* it shuts down lightning fast too. However, more often than not, it will get hung up while trying to run systemd stop jobs to get the system into a safe-to-shutdown state.
There's three jobs that end up "hanging", however I believe only one of them (ssh-agent.service) is the problem, since once it times out after a minute and thirty seconds, the rest of the system quickly finishes up the shutdown process. However, this means that I have to wait for a minute and thirty seconds (plus a bit) before the VM will shut down.
When the systemd unit in question doesn't get hung up, it finishes its stop job almost instantly and the system shuts down fast. When it does get hung up, I know within about two seconds that it's hung up. Is there a way for me to change the timeout for this particular unit from 90 seconds to 3 seconds? I'm fine with changing files that the package manager might overwrite, since worst case scenario it will overwrite them and then I'll have one slow shutdown before I get it fixed again.