TL;DR: Curiously, the unattended-upgrades Python script in my Ubuntu Server installation encountered an ImportError towards the end of its run, and ultimately failed to send the email notification it is supposed to send. Notably, several of the system Python packages received updates during that run, including the email package in the standard library. However, I don’t understand yet the exact mechanism of the script’s failure.
Last weekend, I was surprised to see a “reboot required” message on my Ubuntu Server box after logging in through SSH. I should have received an email for this, but there was none.
I checked the Unattended Upgrades logs at /var/log/unattended-upgrades/unattended-upgrades.log (added text wrapping and newlines between log entries):
2024-09-18 06:06:55,671 INFO Starting unattended upgrades script
2024-09-18 06:06:55,672 INFO Allowed origins are: o=Ubuntu,a=jammy,
o=Ubuntu,a=jammy-security, o=UbuntuESMApps,a=jammy-apps-security,
o=UbuntuESM,a=jammy-infra-security, o=Ubuntu,a=jammy, o=Ubuntu,a=jammy-security,
o=UbuntuESMApps,a=jammy-apps-security, o=UbuntuESM,a=jammy-infra-security
2024-09-18 06:06:55,672 INFO Initial blacklist:
2024-09-18 06:06:55,672 INFO Initial whitelist (not strict):
2024-09-18 06:07:19,237 INFO Packages that will be upgraded: curl
libcurl3-gnutls libcurl4 libexpat1 libexpat1-dev libnginx-mod-http-geoip2
libnginx-mod-http-image-filter libnginx-mod-http-xslt-filter libnginx-mod-mail
libnginx-mod-stream libnginx-mod-stream-geoip2 libpython3.10 libpython3.10-dev
libpython3.10-minimal libpython3.10-stdlib linux-generic linux-headers-generic
linux-image-generic linux-libc-dev nginx nginx-common nginx-core python3.10
python3.10-dev python3.10-minimal python3.10-venv
2024-09-18 06:07:19,238 INFO Writing dpkg log to
/var/log/unattended-upgrades/unattended-upgrades-dpkg.log
2024-09-18 06:08:32,571 INFO All upgrades installed
2024-09-18 06:08:48,716 INFO Packages that were successfully auto-removed:
2024-09-18 06:08:48,717 INFO Packages that are kept back:
2024-09-18 06:08:49,439 ERROR An error occurred: cannot import name
'HeaderWriteError' from 'email.errors' (/usr/lib/python3.10/email/errors.py)
Traceback (most recent call last):
File "/usr/bin/unattended-upgrade", line 2005, in main
send_summary_mail(res.pkgs, res.success, res.result_str,
File "/usr/bin/unattended-upgrade", line 1509, in send_summary_mail
ret = _send_mail_using_sendmail(from_email, to_email, subject, body)
File "/usr/bin/unattended-upgrade", line 1408, in _send_mail_using_sendmail
sendmail.stdin.write(msg.as_string())
File "/usr/lib/python3.10/email/message.py", line 151, in as_string
from email.generator import Generator
File "/usr/lib/python3.10/email/generator.py", line 17, in <module>
from email.errors import HeaderWriteError
ImportError: cannot import name 'HeaderWriteError' from 'email.errors'
(/usr/lib/python3.10/email/errors.py)
It turns out that the package updates that necessitated the reboot were done three days before. However, the script failed to send the email notification because it encountered a Python ImportError while trying to load a class from the standard library’s email package.
Looking at the list of packages upgraded, I spy several related to the system Python installation. I hunted down the changelog for the update to python3.10, and confirmed that the email.errors module is among the files touched by the upgrades.
I looked into the unattended-upgrade Python script itself, and found that the call to send_summary_mail() (for the email notification) is wrapped in a try/except Exception as e catch-all error handling block, so that whatever happens to the first send_summary_mail() call, the script will still get to log the issue—and then call send_summary_mail() again to make sure that the sys admin knows.
Of course, that also failed. I confirmed with the journalctl logs, sudo journalctl --since="2024-09-18 06:08:30":
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: Traceback (most recent call last):
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: File "/usr/bin/unattended-upgrade", line 2005, in main
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: send_summary_mail(res.pkgs, res.success, res.result_str,
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: File "/usr/bin/unattended-upgrade", line 1509, in send_summary_mail
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: ret = _send_mail_using_sendmail(from_email, to_email, subject, body)
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: File "/usr/bin/unattended-upgrade", line 1408, in _send_mail_using_sendmail
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: sendmail.stdin.write(msg.as_string())
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: File "/usr/lib/python3.10/email/message.py", line 151, in as_string
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: from email.generator import Generator
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: File "/usr/lib/python3.10/email/generator.py", line 17, in <module>
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: from email.errors import HeaderWriteError
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: ImportError: cannot import name 'HeaderWriteError' from 'email.errors' (/usr/lib/python3.10/email/errors.py)
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: During handling of the above exception, another exception occurred:
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: Traceback (most recent call last):
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: File "/usr/bin/unattended-upgrade", line 2522, in <module>
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: sys.exit(main(options))
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: File "/usr/bin/unattended-upgrade", line 2027, in main
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: send_summary_mail(["<unknown>"], False, _("An error occurred"),
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: File "/usr/bin/unattended-upgrade", line 1509, in send_summary_mail
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: ret = _send_mail_using_sendmail(from_email, to_email, subject, body)
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: File "/usr/bin/unattended-upgrade", line 1408, in _send_mail_using_sendmail
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: sendmail.stdin.write(msg.as_string())
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: File "/usr/lib/python3.10/email/message.py", line 151, in as_string
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: from email.generator import Generator
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: File "/usr/lib/python3.10/email/generator.py", line 17, in <module>
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: from email.errors import HeaderWriteError
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: ImportError: cannot import name 'HeaderWriteError' from 'email.errors' (/usr/lib/python3.10/email/errors.py)
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: Error in sys.excepthook:
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: Traceback (most recent call last):
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 118, in apport_excepthook
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: pr.add_proc_info(extraenv=['PYTHONPATH', 'PYTHONHOME'])
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: File "/usr/lib/python3/dist-packages/apport/report.py", line 634, in add_proc_info
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: raise ValueError('%s does not exist' % self['ExecutablePath'])
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: ValueError: /usr/bin/python3.10 (deleted) does not exist
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: Original exception was:
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: Traceback (most recent call last):
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: File "/usr/bin/unattended-upgrade", line 2005, in main
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: send_summary_mail(res.pkgs, res.success, res.result_str,
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: File "/usr/bin/unattended-upgrade", line 1509, in send_summary_mail
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: ret = _send_mail_using_sendmail(from_email, to_email, subject, body)
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: File "/usr/bin/unattended-upgrade", line 1408, in _send_mail_using_sendmail
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: sendmail.stdin.write(msg.as_string())
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: File "/usr/lib/python3.10/email/message.py", line 151, in as_string
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: from email.generator import Generator
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: File "/usr/lib/python3.10/email/generator.py", line 17, in <module>
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: from email.errors import HeaderWriteError
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: ImportError: cannot import name 'HeaderWriteError' from 'email.errors' (/usr/lib/python3.10/email/errors.py)
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: During handling of the above exception, another exception occurred:
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: Traceback (most recent call last):
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: File "/usr/bin/unattended-upgrade", line 2522, in <module>
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: sys.exit(main(options))
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: File "/usr/bin/unattended-upgrade", line 2027, in main
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: send_summary_mail(["<unknown>"], False, _("An error occurred"),
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: File "/usr/bin/unattended-upgrade", line 1509, in send_summary_mail
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: ret = _send_mail_using_sendmail(from_email, to_email, subject, body)
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: File "/usr/bin/unattended-upgrade", line 1408, in _send_mail_using_sendmail
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: sendmail.stdin.write(msg.as_string())
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: File "/usr/lib/python3.10/email/message.py", line 151, in as_string
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: from email.generator import Generator
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: File "/usr/lib/python3.10/email/generator.py", line 17, in <module>
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: from email.errors import HeaderWriteError
Sep 18 06:08:49 gemini apt.systemd.daily[43706]: ImportError: cannot import name 'HeaderWriteError' from 'email.errors' (/usr/lib/python3.10/email/errors.py)
Sep 18 06:08:49 gemini apt.systemd.daily[56626]: sendmail: fatal: root(0): No recipient addresses found in message header
Sep 18 06:08:49 gemini postfix/sendmail[56626]: fatal: root(0): No recipient addresses found in message header
Sep 18 06:08:49 gemini postfix/sendmail[56627]: fatal: root(0): No recipient addresses found in message header
Sep 18 06:08:49 gemini apt.systemd.daily[56627]: sendmail: fatal: root(0): No recipient addresses found in message header
Sep 18 06:08:49 gemini systemd[1]: apt-daily-upgrade.service: Deactivated successfully.
Sep 18 06:08:49 gemini systemd[1]: Finished Daily apt upgrade and clean activities.
Sep 18 06:08:49 gemini systemd[1]: apt-daily-upgrade.service: Consumed 1min 51.875s CPU time.
A minor note here: I run Postfix as an SMTP relay host to Amazon SES, so that anything that relies on sendmail working in the system environment gets to, well, send mail. Postfix has nice built-in retry mechanisms that make email sending more reliable (e.g., can overcome temporary network disconnections), but the logs show that the upgrade script did not even get a chance to generate a well-formed email object.
Of much more interest is the log entry that says this:
/usr/bin/python3.10 (deleted) does not exist
So, it appears that in the course of running the script and triggering the system package updates, the script, or the process running the script, at one point was somehow deprived of access to Python itself, or the Python libraries that were updated. At this point, I have also reached the edges of my extremely limited knowledge on Unix/Linux filesystems, Ubuntu/Debian packaging, Python/CPython module loading, or whatever mechanism or combination of mechanisms led to the error.
I know it’s not a severe error, because even if it occurred in a much more important server, there should be minimal impact because the upgrades themselves were installed successfully; it was just the post-install reboot-required notification that failed, and I think some delay with rebooting is not critical. But it would be fascinating to learn what caused it: anyone who has an idea, please drop me an email or perhaps a holler at Mastodon.