What is procmail and why is it using up all my memory?
I recently came across a problem that took longer to resolve than I would have liked. I had a box that was responsible for running a
Java process and nothing else. When the Java process mysteriously died one day, I went to restart
it and it failed immediately due to lack of memory. That was odd... the box had far more memory than I was requesting. When I started looking at
top , I saw that there were dozens of procmail processes chewing up all available memory. Dropping out of
top and looking at the process list, I saw that it was more than dozens - over 10,000 procmail processes were
taking over the entire system. So... where did these come from and what were they doing? (And how should I get rid of them?)
The first order of business was figuring out just what the heck "procmail" was in the first place. I figured from the name that it was
related to Linux's mail subsystem, but it wasn't a process I'd ever encountered in the past. After a bit of research, I discovered
that procmail was apparently Unix's equivalent to Outlook's "rules" folder: it automatically categorizes messages into appropriate folders
based on mail headers. It was pretty obviously something that I had just gotten for "free" with the CentOS distribution; it was being
invoked on each mail forward and what do you know, there was a corresponding sendmail process for each procmail ,
with the counts of each going up.
Googling "too many procmail processes" didn't help, but I did come across a hit that suggested checking /var/log/maillog .
I did, and found:
timeout waiting for input from local during Draining Input
Now, this was something I could Google, and it led me here.
If e-mail is delivered to a program which generates too much
output, then sendmail may issue an error:
timeout waiting for input from local during Draining Input
Make sure that the program does not generate output beyond a
status message (corresponding to the exit status). This may
require a wrapper around the actual program to redirect output
to /dev/null.
So what would be generating mail messages?
Well, by default, when a cron job runs, it mails the results to the owning user. Sure enough, looking at the user's crontab ,
there was a cron process that ran every minute. If cron was mailing the user every minute, there should be...
$ mail
"/var/spool/mail/ec2-user": 110263 messages 387 new 110261 unread
Ah-ha... 110,263 unread mail messages. The mail agent was taking ages to start up, too. After a delete * in the mail console, the problematic
procmail processes started disappearing and enough memory was freed up to run my daemon process. Still, the mail spool
continued to fill up — now that I had the system in a stable state, I could check the mail to see what, exactly, it was so upset
ahbout. It turned out that there was a problem with one of the cron scripts. A quick tweak of the misbehaving script cleared up
the output problem and I was back up and running.
Add a comment:Completely off-topic or spam comments will be removed at the discretion of the moderator. You may preserve formatting (e.g. a code sample) by indenting with four spaces preceding the formatted line(s) |
I'm the author of the book
"Implementing SSL/TLS Using Cryptography and PKI".
Like the title says, this is a from-the-ground-up examination
of the SSL protocol that provides security, integrity and
privacy to most application-level internet protocols, most notably HTTP.
I include the
source
code to a complete working SSL implementation,
including the most popular cryptographic algorithms
(DES, 3DES, RC4, AES, RSA, DSA, Diffie-Hellman, HMAC, MD5, SHA-1,
SHA-256, and ECC), and show how they all fit together
to provide transport-layer security.
Joshua Davies
Past Posts
- April 30, 2021: A Date Picker Control in Vanilla Javascript
- March 31, 2021: A Web Admin Console for Redis, Part Three
- January 27, 2021: A Web Admin Console for Redis, Part Two
- December 21, 2020: A Web Admin Console for Redis, Part One
- November 30, 2020: What is Procmail and why is it using up all my memory?
- September 30, 2020: Minimal Drag and Drop Support in Javascript
- August 31, 2020: Covariance and Contravariance in Generic Types
- July 31, 2020: How Spread Out Are the Floating Point Numbers?
- June 25, 2020: ERD Diagramming Tool, Part Three
- April 30, 2020: ERD Diagramming Tool, Part Two
- March 31, 2020: ERD Diagramming Tool, Part One
- February 28, 2020: MathJax and "t.setAttribute is not a function"
- December 30, 2019: Solving Systems of Equations with Python
- October 30, 2019: Linear Regression with and without numpy
- September 30, 2019: Reading a Parquet file outside of Spark
- August 30, 2019: UML Diagrams with MetaUML
- July 30, 2019: Clustering in Python
- June 25, 2019: A Walkthrough of a TLS 1.3 Handhsake
- May 31, 2019: A DataType Printer in Java
- April 30, 2019: A Simple HTTP Server in Java, Part 3 - Cookies and Keep Alives
- March 28, 2019: A Simple HTTP Server in Java, Part 2 - POST and SSL
- February 28, 2019: A Simple HTTP Server in Java
- January 29, 2019: Angular CLI Behind the Scenes, Part Two
- September 30, 2018: Angular CLI Behind the Scenes, Part One
- August 31, 2018: Into the MMIX MOR Instruction
- July 24, 2018: Undoing Percentage Changes in your Head
- June 30, 2018: Generating Langford Pairs in Scala
- May 25, 2018: Reflections on Three Years of Reading Knuth
- April 30, 2018: java.lang.NoSuchMethodError: org.junit.vintage. engine.descriptor.RunnerTestDescriptor. getAllDescendants
- March 30, 2018: An Excel Spreadsheet for the Academy Awards
- February 28, 2018: Git for Subversion Users
- January 31, 2018: The Evolution of AngularJS
- December 31, 2017: Numerical Integration in Python
- October 31, 2017: Gradle for Java Developers
- September 29, 2017: Reflections on another year of reading Knuth
- August 30, 2017: SSL OCSP Exchange
- July 27, 2017: A walk-through of an SSL certificate exchange
- June 30, 2017: A walk-through of an SSL key exchange
- May 31, 2017: A walk-through of the SSL handshake
- March 31, 2017: A walk-through of the TCP handshake
- February 28, 2017: The TLS Handshake at a High Level
- January 31, 2017: A Walk-through of a JWT Verification
- August 31, 2016: Reflections on a year of reading Knuth
- July 29, 2016: Matching a private key to a public key
- June 30, 2016: A Completely Dissected GZIP File
- May 31, 2016: Automatic Guitar Tablature Generator, Part 2
- April 28, 2016: Automatic Guitar Tablature Generator, Part 1
- March 31, 2016: Import an encrypted private key into a Java Key Store
- February 26, 2016: Import a private key into a Java Key Store
- January 31, 2016: Debian Linux on MacBook Pro
- December 29, 2015: Is Computer Science necessary or useful for programmers?
- November 30, 2015: Client certificate authentication vs. password authentication
- October 28, 2015: A Utility for Viewing Java Keystore Contents
- September 29, 2015: Debugging jQuery with Chrome's Developer Tools
- August 26, 2015: Getting Perl, MySQL and Apache to all work together on Mac OS/X
- July 30, 2015: Extract certificates from Java Key Stores for use by CURL
- June 29, 2015: Using the Chrome web developer tools, Part 9: The Console Tab
- May 28, 2015: Using the Chrome web developer tools, Part 8: The Audits Tab
- April 30, 2015: Using the Chrome web developer tools, Part 7: The Resources Tab
- March 30, 2015: Using the Chrome web developer tools, Part 6: The Memory Profiler Tab
- February 27, 2015: Using the Chrome web developer tools, Part 5: The CPU Profiler Tab
- January 31, 2015: Using the Chrome web developer tools, Part 4: The Timeline Tab
- December 31, 2014: Using the Chrome web developer tools, Part 3: The Sources Tab
- October 31, 2014: Using the Chrome web developer tools, Part 2: The Network Tab
- September 30, 2014: Using the Chrome web developer tools, Part 1: The Elements Tab
- August 11, 2014: Unable to find valid certification path to requested target
- June 30, 2014: Sort by a Hierarchy
- May 29, 2014: OpenSSL Tips and Tricks
- April 25, 2014: Heartbleed: What the Heck Happened
- February 28, 2014: Replace Microsoft Money with a Spreadsheet
- January 29, 2014: An Illustrated Guide to the BEAST Attack
- December 21, 2013: Where does GCC look to find its header files?
- October 24, 2013: Planning a Subversion import
- August 28, 2013: Compile and test an iOS app from the command line
- July 31, 2013: The Hidden Costs of Software Reuse
- June 26, 2013: Beware of mvn war:inplace
- May 29, 2013: Block Font Design Using Javascript
- April 4, 2013: Parsing a POM file using only SED
- February 22, 2013: Inside the PDF File Format
- December 31, 2012:How and why rotation matrices work
- November 27, 2012:Date Management in Java
- October 21, 2012:
Installing Debian Without a Network
- August 14, 2012:
My Review of Matt Neuburg's "Programming iOS 5"
- July 16, 2012:
An example OAuth 1.0 Handshake and mini-library
- May 23, 2012:
A Javascript one-liner to display cookie values
- April 27, 2012:
How SSL Certificates Use Digital Signatures
- March 29, 2012:
A breakdown of a GIF decoder
- February 15, 2012:
The design and implementation of LZW (the GIF compression algorithm)
- January 16, 2012:
Calculate the day of week of any date... in your head
- December 4, 2011:
Understanding CRC32
- October 29, 2011:
Efficient Huffman Decoding
- October 4, 2011:
Extract a private key from a Gnu Keyring file
- September 5, 2011:
From Make to Ant to Maven
- July 18, 2011:
A bottom-up look at the Apache configuration file
- July 6, 2011:
Fun with the HTML 5 Canvas Tag
- Jun 16, 2011:
Pain and disfiguration upon all comment spammers
- May 31, 2011:
Use of RSSI and Time-of-Flight Wireless Signal Characteristics for Location Tracking
- May 7, 2011: Implementing SSL
- Apr 24, 2011: Dissecting the GZIP format
|