My systemd service kept timing out because of how long it would take to boot up also, so this fixed it for me:

1. Edit your systemd file:

   • For modern versions of systemd: Run systemctl edit --full node.service (replace "node" with your service name).
     • This will create a system file at /etc/systemd/system/node.service.d/ that will override the system file at /usr/lib/systemd/system/node.service. This is the proper way to configure your system files. More information about how to use systemctl edit is here.
   • Directly editing system file: The system file for me is at /usr/lib/systemd/system/node.service. Replace "node" with your application name. However, it is not safe to directly edit files in /usr/lib/systemd/ (See comments)

2. Use TimeoutStartSec, TimeoutStopSec or TimeoutSec (more info here) to specify how long the timeout should be for starting & stopping the process. Afterwards, this is how my systemd file looked:

   [Unit]
   Description=MyProject
   Documentation=man:node(1)
   After=rc-local.service
   
   [Service]
   WorkingDirectory=/home/myproject/GUIServer/Server/
   Environment="NODE_PATH=/usr/lib/node_modules"
   ExecStart=-/usr/bin/node Index.js
   Type=simple
   Restart=always
   KillMode=process
   TimeoutSec=900
   
   [Install]
   WantedBy=multi-user.target
   

   • You can also view the current Timeout status by running any of these (but you'll need to edit your service to make changes! See step 1). Confusingly, the associated properties have a "U" in their name for microseconds. See this Github issue for more information:
     • systemctl show node.service -p TimeoutStartUSec
     • systemctl show node.service -p TimeoutStopUSec
     • systemctl show node.service -p TimeoutUSec

3. Next you'll need to reload the systemd with systemctl reload node.service

4. Now try to start your service with systemctl start node.service

5. If that didn't work, try to reboot systemctl with systemctl reboot

6. If that didn't work, try using the --no-block option for systemctl like so: systemctl --no-block start node.service. This option is described here: "Do not synchronously wait for the requested operation to finish. If this is not specified, the job will be verified, enqueued and systemctl will wait until the unit's start-up is completed. By passing this argument, it is only verified and enqueued."

   • There is also the option to use systemctl mask instead of systemctl start. For more info see here.

Updates from Comments:

• TimeoutSec=infinity: Instead of using "infinity" here, put a large amount of time instead, like TimeoutSec=900 (15 min). If the application takes "forever" to exit, then it's possible that it will block a reboot indefinitely. Credit @Alexis Wilke and @JCCyC
• Instead of editing /usr/lib/systemd/system, try systemctl edit instead or edit /etc/systemd/system to override them instead. You should never edit service files in /usr/lib/. Credit @ryeager and @0xC0000022L

** Update from systemd source docs ** When specified "infinity" as a value to any of these timeout params, the timeout logic is disabled.

JobTimeoutSec=, JobRunningTimeoutSec=,TimeoutStartSec=,  TimeoutAbortSec=

The default is "infinity" (job timeouts disabled), except for device units where JobRunningTimeoutSec= defaults to DefaultTimeoutStartSec=.

Reference: enter link description here

Similarly this logic applies to service level and laid out clearly in URL below. Reference: enter link description here

TimeoutStartSec should be used. As @muru mentioned in the comment, the correct way to run a script that should exit is to use Type=oneshot instead of Type=simple.

For example, if your script is normally executed in under 10 seconds, your config would be:

[Service]
Type=oneshot
TimeoutStartSec=10
SyslogIdentifier=myservice
Environment='MYVAR=myvar'
User=deanresin
ExecStart=/usr/bin/python3 /home/deanresin/myscript

If the script takes more than 10 seconds to execute, it will be killed, and the service would show as failed.

To allow a maximum of 5 retries separated by 30 seconds use the following options in the relevant systemd service file.

[Unit]
StartLimitInterval=200
StartLimitBurst=5

[Service]
Restart=always
RestartSec=30

This worked for a service that runs a script using Type=idle. Note that StartLimitInterval must be greater than RestartSec * StartLimitBurst otherwise the service will be restarted indefinitely. The service is considered failed when restarted StartLimitBurst times within StartLimitInterval.

See https://www.freedesktop.org/software/systemd/man/systemd.unit.html#StartLimitIntervalSec=interval and https://www.freedesktop.org/software/systemd/man/systemd.service.html#RestartSec=

To have a service restart 3 times at 90 second intervals include the following lines in your systemd service file:

[Unit]
StartLimitIntervalSec=400
StartLimitBurst=3
[Service]
Restart=always
RestartSec=90

Before systemd-230 it was called just StartLimitInterval:

[Unit]
StartLimitInterval=400
StartLimitBurst=3
[Service]
Restart=always
RestartSec=90

This worked worked for me for a service that runs a script using Type=idle. Note that StartLimitIntervalSec must be greater than RestartSec * StartLimitBurst otherwise the service will be restarted indefinitely.

It took me some time with a lot of trial and error to work out how systemd uses these options, which suggests that systemd isn't as well documented as one would hope. These options effectively provide the retry cycle time and maximum retries that I was looking for.

References: https://manpages.debian.org/testing/systemd/systemd.unit.5.en.html for Unit section https://manpages.debian.org/testing/systemd/systemd.exec.5.en.html for Service section

Partial solution. I basically set these values

StartLimitIntervalSec=1d
StartLimitBurst=5

Which limits 5 restarts per day.

It is possible to terminate systemd services after a certain amount of time.

Use the RuntimeMaxSec option for Type=simple services:

[Unit]
Description=Terminate simple Service Test

[Service]
Type=simple
RuntimeMaxSec=5
ExecStart=/bin/sleep 10

Use the TimeoutStartSec for Type=oneshot services:

[Unit]
Description=Terminate oneshot Service Test

[Service]
Type=oneshot
TimeoutStartSec=5
ExecStart=/bin/sleep 10

Both services will be terminated after 5 seconds before the sleep timer reaches 10 seconds.

Details on RuntimeMaxSec and TimeoutStartSec can be found here: https://www.freedesktop.org/software/systemd/man/systemd.service.html

systemctl daemon-reexec should solve your problem.

The mappings of systemd limits to ulimit

Directive        ulimit equivalent     Unit
LimitCPU=        ulimit -t             Seconds      
LimitFSIZE=      ulimit -f             Bytes
LimitDATA=       ulimit -d             Bytes
LimitSTACK=      ulimit -s             Bytes
LimitCORE=       ulimit -c             Bytes
LimitRSS=        ulimit -m             Bytes
LimitNOFILE=     ulimit -n             Number of File Descriptors 
LimitAS=         ulimit -v             Bytes
LimitNPROC=      ulimit -u             Number of Processes 
LimitMEMLOCK=    ulimit -l             Bytes
LimitLOCKS=      ulimit -x             Number of Locks 
LimitSIGPENDING= ulimit -i             Number of Queued Signals 
LimitMSGQUEUE=   ulimit -q             Bytes
LimitNICE=       ulimit -e             Nice Level 
LimitRTPRIO=     ulimit -r             Realtime Priority  
LimitRTTIME=     ulimit -R             Microseconds

If a ulimit is set to 'unlimited' set it to 'infinity' in the systemd config

ulimit -c unlimited is the same as LimitCORE=infinity
ulimit -v unlimited is the same as LimitAS=infinity
ulimit -m unlimited is the same as LimitRSS=infinity

So a final config would look like

[Unit]
Description=Apache Solr
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
WorkingDirectory=/opt/solr/server
User=solr
Group=solr
LimitAS=infinity
LimitRSS=infinity
LimitCORE=infinity
LimitNOFILE=65536
ExecStart=/opt/solr/bin/solr-foo
Restart=on-failure
SuccessExitStatus=143 0
SyslogIdentifier=solr


[Install]
WantedBy=multi-user.target

In this particular case, I don't know the full java path (since it changes based on server type), and systemd isn't happy about relative paths, I wrap the java command in a simple bash script located at /opt/solr/bin/solr-foo

#!/bin/bash

. /opt/solr/bin/solr.in.sh

# Load $JAVA_HOME from 1 of 2 places where it could be defined
# Last one wins
if [[ -f "/etc/profile.d/jdk.sh" ]]; then
  . /etc/profile.d/jdk.sh
fi

if [[ -f "/etc/profile.d/zing.sh" ]]; then
  . /etc/profile.d/zing.sh
fi

exec ${JAVA_HOME}/bin/java -server \
  -Djetty.port=${SOLR_PORT} \
  ${SOLR_JAVA_MEM} \
  ${GC_TUNE} \
  ${GC_LOG_OPTS} \
  -DzkClientTimeout=${ZK_CLIENT_TIMEOUT} \
  -DzkHost=${ZK_HOST} \
  -DSTOP.PORT=7900 \
  -DSTOP.KEY=foobar \
  -Dhost=${SOLR_HOST} \
  -Duser.timezone=${SOLR_TIMEZONE} \
  -Djetty.home=/opt/solr/server \
  -Dsolr.solr.home=${SOLR_HOME} \
  -Dsolr.install.dir=/opt/solr \
  -Dlog4j.configuration=file:/var/solr/log4j.properties \
  -Xss256k \
  -Dbootstrap_conf=true \
  -Dbootstrap_confdir=/opt/solr/server/solr/configsets/foobar/conf \
  -Dcollection.configName=foobar \
  -jar start.jar --module=http