systemctl start fails during installation but works when run manually

[engageub]

Hi,
Please find the details below with logging enabled in systemctl. The env variables have also been logged to assist with debugging.

The issue observed is that systemctl start fails during the installation process, but the service starts successfully when the same command is executed manually in the terminal after the installation completes.

For reference, please see the related discussion here:
madereddy/EarnApp-Docker#52

I hope this information helps in identifying the necessary updates required in the systemctl configuration to resolve the issue.

Additionally, I have attached the complete systemctl service file currently being used, which includes the debugging configuration. You may compare the env variables when running systemctl start earnapp manually and determine what adjustments might be needed.

Command used for installation:

wget -qO- https://brightdata.com/static/earnapp/install.sh > /tmp/earnapp.sh && sudo bash /tmp/earnapp.sh

File: systemctl.py

Updated code with subprocess.Popen:

            for cmd in cmdlist:
                pid = self.read_mainpid_from(conf)
                env["MAINPID"] = strE(pid)
                exe, newcmd = self.unitfiles.expand_cmd(cmd, env, conf)
                logg.info("%s start %s", runs, shell_cmd(newcmd))
                import subprocess as _subprocess
                import threading as _threading
                logg.info("launching %s", newcmd[0])
                logg.info(env)
                _proc = _subprocess.Popen(newcmd, env=env, close_fds=True,
                    stdout=_subprocess.PIPE, stderr=_subprocess.PIPE,
                    start_new_session=True)
                forkpid = _proc.pid
                self.write_status_from(conf, MainPID=forkpid)
                logg.info("%s started PID %s", runs, forkpid)
                env["MAINPID"] = strE(forkpid)
                time.sleep(3)
                run = subprocess_testpid(forkpid)
                if run.returncode is not None:
                    _out, _err = _proc.communicate()
                    logg.info("%s stopped PID %s (%s) <-%s>", runs, run.pid,
                              run.returncode or "OK", run.signal or "")
                    logg.info("stdout: %s", _out.decode(errors='replace').strip())
                    logg.info("stderr: %s", _err.decode(errors='replace').strip())
                    if doRemainAfterExit:
                        self.set_status_from(conf, "ExecMainCode", strE(run.returncode))
                        active = run.returncode and "failed" or "active"
                        self.write_status_from(conf, AS=active)
                    if run.returncode and exe.check:
                        service_result = "failed"
                        break

Logs:

root@vmi2148846:~/earnapp-test# docker container logs earnapp1

✔ Checking prerequisites...
Installing EarnApp CLI
Welcome to EarnApp for Linux and Raspberry Pi.
EarnApp makes you money by sharing your spare bandwidth.
You will need your EarnApp account username/password.
Visit earnapp.com to sign up if you don't have an account yet

To use EarnApp, allow BrightData to occasionally access websites through your device. BrightData will only access public Internet web pages, not slow down your device or Internet and never access personal information, except IP address - see privacy policy and full terms of service on earnapp.com.
✔ Installing...
✔ Creating directory /etc/earnapp
✔ Fetching earnapp-ssl3-x64-1.607.304
1cf02f86a4017d0a634c1f2b7b13dd12  /tmp/earnapp_1.607.304
✔ Running /tmp/earnapp_1.607.304 install
✔ Moving /tmp/earnapp_1.607.304 to /usr/bin
2026-03-15 00:36:08.333 NOTICE: generated uuid: sdk-node-c8c14fb5cbd54a5a8481bef5544b6b72
INFO:systemctl:EXEC BEGIN /usr/bin/systemctl daemon-reload --system(0)
DEBUG:systemctl:======= systemctl.py daemon-reload
DEBUG:systemctl:found 2 unit files
DEBUG:systemctl:found 2 sysv files
INFO:systemctl:EXEC BEGIN /usr/bin/systemctl enable earnapp --system(0)
DEBUG:systemctl:======= systemctl.py enable earnapp
DEBUG:systemctl:found 2 unit files
DEBUG:systemctl:found 2 sysv files
DEBUG:systemctl:[enable] matched earnapp.service
INFO:systemctl:[sysinit.target] ... system is offline -- ready
DEBUG:systemctl:'/etc/systemd/system/earnapp.service' no --user mode >> accept
DEBUG:systemctl:WantedBy multi-user.target -> /etc/systemd/system/multi-user.target.wants
INFO:systemctl:ln -s  '/etc/systemd/system/earnapp.service' '/etc/systemd/system/multi-user.target.wants/earnapp.service'
INFO:systemctl:EXEC BEGIN /usr/bin/systemctl start earnapp --system(0)
DEBUG:systemctl:======= systemctl.py start earnapp
DEBUG:systemctl:found 2 unit files
DEBUG:systemctl:found 2 sysv files
INFO:systemctl:[sysinit.target] ... system is offline -- ready
DEBUG:systemctl:'/etc/systemd/system/earnapp.service' no --user mode >> accept
DEBUG:systemctl: start unit earnapp.service => '/etc/systemd/system/earnapp.service'
DEBUG:systemctl:ExecStart[0]: /usr/bin/earnapp run
INFO:systemctl:simple start '/usr/bin/earnapp' 'run'
INFO:systemctl:launching /usr/bin/earnapp
INFO:systemctl:{'HOSTNAME': '6638e1040a2f', 'SHLVL': '1', 'HOME': '/root', 'WS_NO_ZCOUNTER': '1', 'PKG_EXECPATH': '/usr/bin/earnapp', 'container': 'docker', '_': '/tmp/earnapp_1.607.304', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'DEBIAN_FRONTEND': 'noninteractive', 'PWD': '/', 'LC_CTYPE': 'C.UTF-8', 'MAINPID': ''}
DEBUG:systemctl:[status] writing to /run/earnapp.service.status
        MainPID=109
INFO:systemctl:simple started PID 109
INFO:systemctl:simple stopped PID 109 (1) <->
INFO:systemctl:stdout:
INFO:systemctl:stderr: node:internal/modules/cjs/loader:949
  throw err;
  ^

Error: Cannot find module '/run'
    at Module._resolveFilename (node:internal/modules/cjs/loader:946:15)
    at Function._resolveFilename (pkg/prelude/bootstrap.js:1951:46)
    at Module._load (node:internal/modules/cjs/loader:787:27)
    at Function.runMain (pkg/prelude/bootstrap.js:1979:12)
    at node:internal/main/run_main_module:17:47 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

Node.js v18.5.0
WARNING:systemctl:simple start not active
✖ Command failed: systemctl start earnapp
INFO:systemctl:EXEC BEGIN /usr/bin/systemctl start earnapp --system(0)
DEBUG:systemctl:======= systemctl.py start earnapp
DEBUG:systemctl:found 2 unit files
DEBUG:systemctl:found 2 sysv files
INFO:systemctl:[sysinit.target] ... system is offline -- ready
DEBUG:systemctl:'/etc/systemd/system/earnapp.service' no --user mode >> accept
DEBUG:systemctl: start unit earnapp.service => '/etc/systemd/system/earnapp.service'
DEBUG:systemctl:ExecStart[0]: /usr/bin/earnapp run
INFO:systemctl:simple start '/usr/bin/earnapp' 'run'
INFO:systemctl:launching /usr/bin/earnapp
INFO:systemctl:{'HOSTNAME': '6638e1040a2f', 'SHLVL': '1', 'HOME': '/root', 'WS_NO_ZCOUNTER': '1', 'PKG_EXECPATH': '/usr/bin/earnapp', 'container': 'docker', '_': '/tmp/earnapp_1.607.304', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'DEBIAN_FRONTEND': 'noninteractive', 'PWD': '/', 'LC_CTYPE': 'C.UTF-8', 'MAINPID': ''}
DEBUG:systemctl:[status] writing to /run/earnapp.service.status
        MainPID=109
INFO:systemctl:simple started PID 109
INFO:systemctl:simple stopped PID 109 (1) <->
INFO:systemctl:stdout:
INFO:systemctl:stderr: node:internal/modules/cjs/loader:949
  throw err;
  ^

Error: Cannot find module '/run'
    at Module._resolveFilename (node:internal/modules/cjs/loader:946:15)
    at Function._resolveFilename (pkg/prelude/bootstrap.js:1951:46)
    at Module._load (node:internal/modules/cjs/loader:787:27)
    at Function.runMain (pkg/prelude/bootstrap.js:1979:12)
    at node:internal/main/run_main_module:17:47 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

Node.js v18.5.0
WARNING:systemctl:simple start not active

INFO:systemctl:EXEC BEGIN /usr/bin/systemctl daemon-reload --system(0)
DEBUG:systemctl:======= systemctl.py daemon-reload
DEBUG:systemctl:found 3 unit files
DEBUG:systemctl:found 2 sysv files
INFO:systemctl:EXEC BEGIN /usr/bin/systemctl enable earnapp_upgrader --system(0)
DEBUG:systemctl:======= systemctl.py enable earnapp_upgrader
DEBUG:systemctl:found 3 unit files
DEBUG:systemctl:found 2 sysv files
DEBUG:systemctl:[enable] matched earnapp_upgrader.service
INFO:systemctl:[sysinit.target] ... system is offline -- ready
DEBUG:systemctl:'/etc/systemd/system/earnapp_upgrader.service' no --user mode >> accept
DEBUG:systemctl:WantedBy multi-user.target -> /etc/systemd/system/multi-user.target.wants
INFO:systemctl:ln -s  '/etc/systemd/system/earnapp_upgrader.service' '/etc/systemd/system/multi-user.target.wants/earnapp_upgrader.service'
INFO:systemctl:EXEC BEGIN /usr/bin/systemctl start earnapp_upgrader --system(0)
DEBUG:systemctl:======= systemctl.py start earnapp_upgrader
DEBUG:systemctl:found 3 unit files
DEBUG:systemctl:found 2 sysv files
INFO:systemctl:[sysinit.target] ... system is offline -- ready
DEBUG:systemctl:'/etc/systemd/system/earnapp_upgrader.service' no --user mode >> accept
DEBUG:systemctl: start unit earnapp_upgrader.service => '/etc/systemd/system/earnapp_upgrader.service'
DEBUG:systemctl:ExecStart[0]: /usr/bin/earnapp autoupgrade
INFO:systemctl:simple start '/usr/bin/earnapp' 'autoupgrade'
INFO:systemctl:launching /usr/bin/earnapp
INFO:systemctl:{'HOSTNAME': '6638e1040a2f', 'SHLVL': '1', 'HOME': '/root', 'WS_NO_ZCOUNTER': '1', 'PKG_EXECPATH': '/usr/bin/earnapp', 'container': 'docker', '_': '/tmp/earnapp_1.607.304', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'DEBIAN_FRONTEND': 'noninteractive', 'PWD': '/', 'LC_CTYPE': 'C.UTF-8', 'MAINPID': ''}
DEBUG:systemctl:[status] writing to /run/earnapp_upgrader.service.status
        MainPID=121
INFO:systemctl:simple started PID 121
INFO:systemctl:simple stopped PID 121 (1) <->
INFO:systemctl:stdout:
INFO:systemctl:stderr: node:internal/modules/cjs/loader:949
  throw err;
  ^

Error: Cannot find module '/autoupgrade'
    at Module._resolveFilename (node:internal/modules/cjs/loader:946:15)
    at Function._resolveFilename (pkg/prelude/bootstrap.js:1951:46)
    at Module._load (node:internal/modules/cjs/loader:787:27)
    at Function.runMain (pkg/prelude/bootstrap.js:1979:12)
    at node:internal/main/run_main_module:17:47 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

Node.js v18.5.0
WARNING:systemctl:simple start not active
✖ Command failed: systemctl start earnapp_upgrader
INFO:systemctl:EXEC BEGIN /usr/bin/systemctl start earnapp_upgrader --system(0)
DEBUG:systemctl:======= systemctl.py start earnapp_upgrader
DEBUG:systemctl:found 3 unit files
DEBUG:systemctl:found 2 sysv files
INFO:systemctl:[sysinit.target] ... system is offline -- ready
DEBUG:systemctl:'/etc/systemd/system/earnapp_upgrader.service' no --user mode >> accept
DEBUG:systemctl: start unit earnapp_upgrader.service => '/etc/systemd/system/earnapp_upgrader.service'
DEBUG:systemctl:ExecStart[0]: /usr/bin/earnapp autoupgrade
INFO:systemctl:simple start '/usr/bin/earnapp' 'autoupgrade'
INFO:systemctl:launching /usr/bin/earnapp
INFO:systemctl:{'HOSTNAME': '6638e1040a2f', 'SHLVL': '1', 'HOME': '/root', 'WS_NO_ZCOUNTER': '1', 'PKG_EXECPATH': '/usr/bin/earnapp', 'container': 'docker', '_': '/tmp/earnapp_1.607.304', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'DEBIAN_FRONTEND': 'noninteractive', 'PWD': '/', 'LC_CTYPE': 'C.UTF-8', 'MAINPID': ''}
DEBUG:systemctl:[status] writing to /run/earnapp_upgrader.service.status
        MainPID=121
INFO:systemctl:simple started PID 121
INFO:systemctl:simple stopped PID 121 (1) <->
INFO:systemctl:stdout:
INFO:systemctl:stderr: node:internal/modules/cjs/loader:949
  throw err;
  ^

Error: Cannot find module '/autoupgrade'
    at Module._resolveFilename (node:internal/modules/cjs/loader:946:15)
    at Function._resolveFilename (pkg/prelude/bootstrap.js:1951:46)
    at Module._load (node:internal/modules/cjs/loader:787:27)
    at Function.runMain (pkg/prelude/bootstrap.js:1979:12)
    at node:internal/main/run_main_module:17:47 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

Node.js v18.5.0
WARNING:systemctl:simple start not active

✔ EarnApp is installed and running.
- Registering Device...

See usage options by running earnapp

✔ Registered
✔ EarnApp is active (making money in the background)
⚠ You must register it for earnings to be added to your account.
⚠ Open the following URL in the browser:
  https://earnapp.com/r/sdk-node-c8c14fb5cbd54a5a8481bef5544b6b72
✔ Installation complete

Logs with systemctl start earnapp and systemctl start earnapp_upgrader

Starting earnapp...
INFO:systemctl:EXEC BEGIN /usr/bin/systemctl start earnapp --system(0)
DEBUG:systemctl:======= systemctl.py start earnapp
DEBUG:systemctl:found 3 unit files
DEBUG:systemctl:found 2 sysv files
INFO:systemctl:[sysinit.target] ... system is offline -- ready
DEBUG:systemctl:'/etc/systemd/system/earnapp.service' no --user mode >> accept
DEBUG:systemctl: start unit earnapp.service => '/etc/systemd/system/earnapp.service'
DEBUG:systemctl:ExecStart[0]: /usr/bin/earnapp run
INFO:systemctl:simple start '/usr/bin/earnapp' 'run'
INFO:systemctl:launching /usr/bin/earnapp
INFO:systemctl:{'HOSTNAME': '92dafa278a1b', 'PWD': '/', 'container': 'docker', 'HOME': '/root', 'SHLVL': '1', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'DEBIAN_FRONTEND': 'noninteractive', '_': '/usr/bin/systemctl', 'LC_CTYPE': 'C.UTF-8', 'MAINPID': '109'}
DEBUG:systemctl:[status] writing to /run/earnapp.service.status
        MainPID=152
INFO:systemctl:simple started PID 152
Starting earnapp_upgrader...
INFO:systemctl:EXEC BEGIN /usr/bin/systemctl start earnapp_upgrader --system(0)
DEBUG:systemctl:======= systemctl.py start earnapp_upgrader
DEBUG:systemctl:found 3 unit files
DEBUG:systemctl:found 2 sysv files
INFO:systemctl:[sysinit.target] ... system is offline -- ready
DEBUG:systemctl:'/etc/systemd/system/earnapp_upgrader.service' no --user mode >> accept
DEBUG:systemctl: start unit earnapp_upgrader.service => '/etc/systemd/system/earnapp_upgrader.service'
DEBUG:systemctl:ExecStart[0]: /usr/bin/earnapp autoupgrade
INFO:systemctl:simple start '/usr/bin/earnapp' 'autoupgrade'
INFO:systemctl:launching /usr/bin/earnapp
INFO:systemctl:{'HOSTNAME': '92dafa278a1b', 'PWD': '/', 'container': 'docker', 'HOME': '/root', 'SHLVL': '1', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'DEBIAN_FRONTEND': 'noninteractive', '_': '/usr/bin/systemctl', 'LC_CTYPE': 'C.UTF-8', 'MAINPID': '121'}
DEBUG:systemctl:[status] writing to /run/earnapp_upgrader.service.status
        MainPID=168
INFO:systemctl:simple started PID 168

[gdraheim]

There is an option EXEC_SPAWN that ensures the execve_from is actually running with a child process. It's used during testing. It is not the default because systemctl is not supposed to wait for an ExecStart to finish. If the pid-file is there then the started unit is considered active.

I know that some services are not actually ready when the port is open but that's the task of the second service to check for. The provided source code shows a sleep(3) and I have made some success adding a sleep in an ExecStartPre on the second service.

[engageub]

Hi,
Thank you for the response.

If you review the error messages, both services are throwing MODULE_NOT_FOUND errors related to Node.js when executed through the installation script.

When the env variable is removed or set to {}, the second service (upgrader) starts successfully, but the first service still fails to start. However, both services are expected to start correctly.

To verify this behavior, you can install the application directly on the host system or temporarily prevent execution during the first run by adding the following code before the _proc = _subprocess.Popen command:

if "PKG_EXECPATH" in env:
    sys.exit()

After doing this, start the services manually using:

systemctl start earnapp
systemctl start earnapp_upgrader

You should observe that during the first run both services start and run successfully.

This suggests that the issue may be related to the env environment being passed to the subprocess, which appears to behave differently from the standard systemd environment.

Thank you

[gdraheim]

The log shows some PKG_EXECPATH in the first line - where is it supposed to be set? Can you point me to the earnapp.service file?

[engageub]

Hi,
The PKG_EXECPATH is being set when executing the installation script. This variable is not present when systemctl start earnapp is run manually.
In order to terminate the systemctl process during installation without executing child process, it is supposed to be set in this file before the following line.
_proc = _subprocess.Popen(newcmd, env=env, close_fds=True,

Thank you

[madereddy]

@gdraheim

I was able to get it working by making a few changes in systemctl3.py along with some changes to the Dockerfile to ignore fstrim warning thrown.

I also changed the minimum_yield to slow it down to 10 from .5, but I dont believe that is required for most uses cases and will just slow down normal statup processes causing further issues for the majority of containers using this.

Fully updated link

In do_start_service_from()
Setting up a child session to start the service
Setup simple services to start

Changed error to debug for env variables - Not needed, just to clean up my debugging output for troubleshooting

Major changes below

for cmd in cmdlist:
                pid = self.read_mainpid_from(conf)
                env["MAINPID"] = strE(pid)
                exe, newcmd = self.unitfiles.expand_cmd(cmd, env, conf)
                logg.info("%s start %s", runs, shell_cmd(newcmd))
                forkpid = os.fork()
                if forkpid == 0:  # first child
                    os.setsid()   # detach session
                
                    second = os.fork()
                    if second == 0:  # grandchild (actual service)
                        self.execve_from(conf, newcmd, env)
                
                    os._exit(0)  # first child exits             

if not self.is_active_from(conf):
        
            # systemd semantics: Type=simple services are considered started
            # once ExecStart is invoked. Some services daemonize or re-exec.
            if runs == "simple":
                logg.debug("simple service not yet active (ignoring)")
                return True
        
            logg.warning("%s start not active", runs)     

---

In execve_from()

Made sure to inherit the full process environment like systemd
Made sure to prevent state _ from breaking the Node.js binary

Fully update below

def execve_from(self, conf: SystemctlConf, cmd: List[str], env: Dict[str, str]) -> NoReturn:
        """ this code is commonly run in a child process // returns exit-code"""
        runs = conf.get(Service, "Type", "simple").lower()
        # logg.debug("%s process for %s => %s", runs, strE(conf.name()), strQ(conf.filename()))
        self.dup2_journal_log(conf)
        cmd_args: List[Union[str, bytes]] = []
        #
        runuser = self.unitfiles.get_User(conf)
        rungroup = self.unitfiles.get_Group(conf)
        xgroups = self.unitfiles.get_SupplementaryGroups(conf)
        envs = shutil_setuid(runuser, rungroup, xgroups)
        badpath = self.chdir_workingdir(conf) # some dirs need setuid before
        if badpath:
            logg.error("(%s): bad workingdir: '%s'", shell_cmd(cmd), badpath)
            sys.exit(1)
        # inherit full process environment like real systemd
        env = {**os.environ, **env}
        
        env = self.extend_exec_env(env)
        env.update(envs) # set $HOME to ~$USER
        env.setdefault("PATH", os.environ.get("PATH", "/usr/bin:/bin")) ## fix for Node.js packages failing by the path changing
        env.pop('_', None) # prevent stale _ from breaking pkg-bundled Node.js binaries
        try:
            if EXEC_SPAWN:
                cmd_args = [arg for arg in cmd] # satisfy mypy
                exitcode = os.spawnvpe(os.P_WAIT, cmd[0], cmd_args, env)
                sys.exit(exitcode)
            else: # pragma: no cover
                env.pop('_', None)
                logg.debug("execve launching: %s", shell_cmd(cmd))
                os.execve(cmd[0], cmd, env)
                sys.exit(11) # pragma: no cover (can not be reached / bug like mypy#8401)

[gdraheim]

The real systemd does not inherit the environment of the systemctl client tool - that one uses a socket to talk to the daemon which may have started on system boot. Well, it requires the services to define what kind of environment variables they like to have. As far as I know, the systemd daemon only loads /etc/environment and for each service /etc/locale.conf

In any case, it seems like some environment part is missing.