Mastering Python Script Execution on Linux: Practical Methods for Automation
There’s more to running Python scripts on Linux than python3 script.py. Achieving predictable, maintainable automation—especially on production hosts—requires a disciplined approach: managed environments, precise permissions, robust scheduling, and service-level reliability.
Environment Isolation: Avoiding “Works on My Machine” Traps
Deploying Python 3.10 code on a fleet with mixed distributions (Ubuntu 22.04, CentOS 8) commonly surfaces dependency chaos. Direct use of system Python (/usr/bin/python3) means exposure to distro package updates and global pip installs.
Solution: Always encapsulate dependencies using venv:
python3.10 -m venv /opt/scripts/venv-myscript
source /opt/scripts/venv-myscript/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
Non-obvious tip: In some corporate environments, $HOME directories may be mounted with noexec. Use /opt or another exec-enabled mount for venvs.
Shebang best practice: Set your shebang to pick up the venv interpreter.
#!/usr/bin/env python3
But if you use compiled C extensions (e.g. via numpy/pandas), specify absolute path to the venv python to avoid environment bleed:
#!/opt/scripts/venv-myscript/bin/python
Permissions: File Ownership and Execution
Scripts must be executable, but avoid chmod 777. Assign correct group and limit write access.
chown root:automation my_script.py # Ownership
chmod 750 my_script.py # Execute for owner/group only
ls -l my_script.py
Known issue: Scripts run by cron as nobody or restricted users can silently error if they lack read/execute rights on the venv or script. Double-check with:
sudo -u nobody /opt/scripts/venv-myscript/bin/python my_script.py
Background Execution: Keeping Jobs Alive
Critical for data pipelines and ETL jobs—if your SSH session dies, don’t lose your script’s state:
| Method | Pros | Cons | Notes |
|---|---|---|---|
nohup | Lightweight, simple | Hard to reattach | Drops SIGHUP. Output to log. |
screen/tmux | Hot attach/detach | Extra install | Can recover terminal at will. |
systemd | Managed restarts | More setup | Best for real daemons. |
Basic nohup example:
nohup /opt/scripts/venv-myscript/bin/python my_script.py >> /var/log/myscript.log 2>&1 &
Screen for debugging:
screen -S myscript
cd /opt/scripts
source venv-myscript/bin/activate
python my_script.py
Detach: Ctrl-A D.
Reattach: screen -r myscript
Scheduling: Cron Reminders and Gotchas
Crontab is not your shell:
Don’t assume PATH, PYTHONPATH, or environment variables are available. Absolute everything.
Typical robust cron entry:
0,30 * * * * /opt/scripts/venv-myscript/bin/python /opt/scripts/my_script.py >> /var/log/myscript.log 2>&1
- If your script expects environment variables (e.g., API_KEYS), source them in a wrapper:
#!/bin/bash
source /opt/scripts/venv-myscript/bin/activate
export API_KEY="xxx"
exec python /opt/scripts/my_script.py "$@"
Gotcha: Crontab logs errors to local mail by default, which often nobody reads. Redirect output to log files you’ll actually check.
Service Approach: systemd Unit for Resilience
Mission-critical tasks—scraping, continuous ingest, multi-hour runs—shouldn’t depend on users or cron. Use a systemd unit:
/etc/systemd/system/my_script.service
[Unit]
Description=Reliable Python Worker
After=network-online.target
[Service]
Type=simple
User=svc-python
WorkingDirectory=/opt/scripts
ExecStart=/opt/scripts/venv-myscript/bin/python my_script.py
Restart=on-failure
RestartSec=3
StandardOutput=append:/var/log/myscript.log
StandardError=append:/var/log/myscript.err
[Install]
WantedBy=multi-user.target
Reload and enable:
sudo systemctl daemon-reload
sudo systemctl enable --now my_script.service
Check logs:
journalctl -u my_script.service -f
Side note: When scripts write to files, confirm the User= specified matches file system permissions to avoid silent write failures.
Summary Table: Production-Ready Script Execution
| Aspect | Recommendation | Real-world Caveat |
|---|---|---|
| Dependencies | Use venv, never system python | Don’t trust pip freeze alone |
| Entrypoint | Shebang with env or direct to venv python | Absolute path for C-extension |
| Permissions | Restrict with user/group, no “world write” | Cron/external users may fail |
| Background run | Prefer systemd, then screen/tmux | nohup drops output on script err |
| Scheduling | cron with wrapper script, log all output | Missing environment is common |
| Daemon/service | systemd with restart/restartsec, log files | Rootless services are preferred |
Advanced workflow:
For clustered jobs, combine systemd with a process supervisor (e.g., supervisord) when you need multiple, interdependent Python services.
Not perfect: systemd’s basic logging isn’t structured—consider stderr JSON output piped to journald for log aggregation.
Practical mastery is less about memorizing commands and more about assembling predictable, debuggable workflows. If your automation fails at 2am, you want every log, PID, and environment unfuzzed and reviewable.
For scenarios involving complex environment set-up, containerization with Docker or scheduling with Celery offers added reliability and scaling. This article intentionally omits those for direct host automation—another potential topic.