HPR4677: UNIX Curio #10 - Checksums and Hashes

This show has been flagged as Clean by the host.

This series is dedicated to exploring little-known—and occasionally useful—trinkets lurking in the dusty corners of UNIX-like operating systems.

In UNIX Curio #8 ( HPR episode 4657 ), I talked about using standard utilities to compare files. Left unmentioned, however, was a method commonly used today—the hash function.

As I've stated in previous entries, while I am an engineer, I don't have a background in computer science, so my understanding of the mathematics is limited. But I can give a practical description of what a hash function does. It takes an input, performs a set of calculations on it, and produces an output. As hash functions are practically used, the input is a set of bytes, such as a file or another piece of data like a password. The output is a numerical value in a fixed range—most often, expressed as hexadecimal characters. Because this "hash value" can always be represented in a certain number of bytes, its length as printed is usually a constant number of characters, padded with leading zeros if necessary. This episode will not cover the use of hashes in programming, focusing instead on using them to validate data.

A hash function, or more specifically, a cryptographic hash function, has an additional property. It should be very difficult to predict what changes to the input would be required to produce a specific change in the output.

An older, related concept is called a "checksum". While these are designed to vary when the input data is damaged or digits are transposed, they do not necessarily have that last property mentioned for cryptographic hashes. You have probably already encountered a checksum, even if you didn't recognize it. On a 16-digit number assigned to a Mastercard or Visa 1 credit or debit card, the first six digits identify the card issuer (such as a bank), the next nine digits are assigned to you by the issuer, and the last digit is a check digit. The check digit is calculated using the values of the previous 15 digits, and it is a simple way to avoid typos in entering a card number.

In another example, every Ethernet frame that your devices send or receive includes a checksum 2 to help ensure that the contents weren't scrambled in transit. This is 32 bits long and is called a cyclical redundancy check, commonly referred to as a CRC. A CRC is also used in many other places—for example, the .zip file format includes one for each archive member, and this allows a program extracting files from the archive to identify if any were damaged.

Our UNIX Curio for today is another example, the cksum utility 3 . It generates a 32-bit CRC based on the Ethernet algorithm. It operates on either a named file or standard input and outputs the CRC value, the length of the input, and the pathname if a file was given as an argument. Unlike most modern hashing programs, the checksum is printed as a decimal integer and is not padded, so it can be anywhere from one to ten digits long. The length value is the number of bytes in the input (actually specified as the number of octets , as systems could potentially use a byte that isn't eight bits long), also expressed as a decimal integer.

There are two major ways that one could use cksum