Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Unix system administration

by Jiří Navrátil

This textbook is a supplementary material for Unix system administration lectures by Jiří Navrátil. It is designed to complement classroom instruction and provide students with a comprehensive reference for understanding Unix fundamentals, system administration practices, and the philosophical principles underlying Unix-like operating systems.

Throughout the course, students will have access to OpenBSD servers for hands-on practice and practical application of the concepts covered in lectures.

Foreword

In these days, when there is a prevailing feeling that everything is living in the cloud, it is crucial that students understand there is still hardware with operating systems using resources which are not unlimited and cost real money. That the hardware is connected via networks that require proper security considerations. While cloud computing abstracts away many infrastructure concerns, system administrators must still comprehend the underlying fundamentals of how systems actually operate, consume resources, and interact with each other.

Students will use OpenBSD with its available source code, good license, secure reputation, and excellent documentation to learn system administration on Unix-like system. OpenBSD provides an ideal learning environment where students can examine every aspect of the system, understand resource constraints, and develop a deep appreciation for security-conscious system design. This hands-on approach with real hardware and actual resource limitations will prepare students for the realities of system administration, whether managing cloud infrastructure, on-premises servers, or hybrid environments.

— Jiří Navrátil

History

Unix represents one of the most influential operating systems in computing history, serving as the foundation for countless modern systems from servers to smartphones. This introduction provides the essential background needed to understand Unix's evolution, its various implementations, and practical system administration concepts.

Multics

Multics (Multiplexed Information and Computer Services) was an experimental time-sharing operating system developed in the mid-1960s as a joint research project by Massachusetts Institute of Technology (MIT), General Electric (GE), and Bell Labs.

It was designed as a time-sharing system that allowed multiple users to access a mainframe simultaneously.

Multics introduced many innovations, such as the use of a single-level store (virtual memory), hierarchical file system with arbitrarily nested subdirectories, concept of the command interpreter being a regular program, with additional commands provided as separate programs

Despite its ambitious goals and innovations, the Multics project was too big and too complex. The Bell Labs team concluded that the system was unlikely to deliver a functional or performant operating system in the near future.

AT&T Bell Labs withdrew from the Multics project in 1969. This left key researchers, including Ken Thompson and Dennis Ritchie, looking for a new endeavour.

Unix

Multics served as a significant source of inspiration for Unix. Ken Thompson, who enjoyed Multics flexibility, sought to create a simpler operating system that could run on less powerful machines. He decided to implement the best ideas from Multics on a smaller scale.

  • Starting in 1969, Thompson ported a simple file-and-process system (initially for his “Space Game”) to a little-used PDP-7.
  • By 1970 he and Ritchie had built a toolkit of utilities (editor, assembler, shell) and adopted the name UNIX1
  • In 1971 they moved Unix onto the PDP-11 and rewrote almost all of it in the new C language, making Unix unusually portable
  • Early versions (such as Version 6 and Version 7 UNIX) spread to academic sites (e.g. UC Berkeley) and later became commercialized by AT&T in the mid-1970s

Key founders and collaborators included Ken Thompson, Dennis Ritchie, Brian Kernighan, Doug McIlroy and Joe Ossanna.

  1. The name "UNICS" (UNiplexed Information and Computing Service), a pun on "Multics" (Multiplexed Information and Computer Services), was suggested by Brian Kernighan in 1970 and eventually shortened to Unix. ↩

Unix Philosophy

The Unix philosophy is a set of cultural norms for developing software that emerged over time from the leading developers of Unix. It characterises the modular design of Unix systems.

At its core, the Unix philosophy dictates that the operating system should provide a set of simple tools, with each tool performing a limited, well-defined function.

Key ideas central to this philosophy include:

  • "Make each program do one thing well"
  • "Write programs that work together"
  • "Write programs that handle text streams", meaning that everything is treated as text. This allows for simpler programs that can interact with a wider variety of inputs and outputs.

Brian Kernighan and Rob Pike summarised this philosophy as "the idea that the power of a system comes more from the relationships among programs than from the programs themselves". This power is largely facilitated by a unified and inode-based filesystem and the use of "pipes" as the main means of inter-process communication. A shell scripting and command language, known as the Unix shell, then allows these simple tools to be combined to perform complex tasks.

This approach was partly a result of the modest hardware available when Unix was first developed. The limited processor speed and memory enforced a "minimality of mechanisms" and a "search for generalizations". For example, the file system interface was designed to be straightforward and could be applied to various tasks, including reading and writing to different devices as if they were ordinary disk files.

Even today, many systems adhere to this philosophy of using small, simple tools that can be combined to achieve complex operations. Programs referred to as "modern Unix" tools are typically those that align with the Unix philosophy, run within the Unix architecture, and are executed from a Unix shell.

C language

Dennis Ritchie wrote the first C compiler.

In 1973, Version 4 Unix was famously rewritten in C. This was considered a "key pioneering approach" and an "unusual step that was visionary", as it went "contrary to the general notion at the time that an operating system's complexity and sophistication required it to be written in assembly language".

Rewriting Unix in C made it highly portable, enabling it to run on diverse platforms and "outlive its original hardware". This significantly reduced the amount of machine-dependent code needed for porting.

Dennis Ritchie attributed Unix's early success to the "readability, modifiability, and portability of its software that in turn follows from its expression in high-level languages" like C.

Unix system calls are regarded as C functions, and many Unix programs follow C's syntax

The C programming language soon spread beyond Unix and became ubiquitous in systems and applications programming

Both Unix and the C programming language were initially distributed to government and academic institutions, which led to both being ported to a wider variety of machine families than any other operating system.

A 2001 study of Red Hat Linux 7.1 revealed that 71% of its source code was written in the C programming language.

Even today, C remains highly relevant

  • The Linux kernel is primarily written in C (along with assembly languages, Rust, and others).
  • Linux distributions offer robust support for programming languages including C and C++.
  • The GNU Compiler Collection (GCC) provides compilers for C, C++, and other languages.
  • C++ (a descendant of C) continues to be one of the most widely used programming languages.

Innovations

Building upon the core tenets of the Unix philosophy – which advocates for simple tools, each doing one thing well, that work together, and handle text streams – several significant innovations emerged, further shaping the landscape of computing.

  • The C Programming Language: The C language was intrinsically linked with Unix's evolution. Dennis Ritchie, a key figure in Unix's creation, wrote the first C compiler. In a "key pioneering approach," Version 4 Unix was famously rewritten in C in 1973, an "unusual step that was visionary" at the time. This decision significantly enhanced Unix's portability, allowing it to "outlive its original hardware" by reducing machine-dependent code. C provided an "unprecedented combination of efficiency and expressiveness to programming". Unix system calls are regarded as C functions, and many Unix programs follow C's syntax.

  • Pipes as a Foundational Inter-Process Communication (IPC) Mechanism: While the concept of pipes is part of the Unix architecture and philosophy, their development was a crucial innovation. Pipes enabled the creation of a more modular code base and quicker development cycles. They made a powerful programming paradigm, known as "coroutines," widely available by allowing modular chains of producer-consumer processes. This reinforced the idea of connecting simple tools to perform complex tasks.

  • Hierarchical File System and Device Abstraction: Unix popularised the hierarchical file system with arbitrarily nested subdirectories, a concept originally introduced by Multics. Unix drastically simplified its file model compared to contemporaries, treating all files as simple byte arrays. A key concept added by Ritchie was the device file, which abstracted away different hardware devices by treating them as files within the file system, allowing programs to manipulate them using standard file operations.

  • Modular Command Interpreters (Shells) and Separate Commands: Unix popularized the idea of the command interpreter (shell) being an ordinary user-level program, with commands themselves being separate programs. This design, also influenced by Multics, meant that new commands could be added without altering the shell, and users could choose or even write their own shells. This flexibility is evident in the variety of shells available, such as the Bourne Shell (sh) and the later, more feature-rich Bourne Again SHell (Bash).

  • Software Tools Movement and Emphasis on Modularity/Reusability: Early Unix developers were instrumental in bringing the concepts of modularity and reusability into software engineering practice, which spurred a "software tools" movement. This directly reflects the philosophy of "make each program do one thing well".

  • Rapid Adoption of TCP/IP Networking Protocols: Unix's design facilitated the quick implementation of TCP/IP networking protocols. This capability, often built into the operating system itself, significantly contributed to the "Internet explosion" of worldwide connectivity and formed the basis for networking implementations on many other platforms. The Berkeley Software Distribution (BSD) branch of Unix, in particular, was vital in this, as its network code became the ancestor of much of the TCP/IP network code used today, including in early versions of Microsoft Windows.

  • Extensive Online Documentation (Man Pages): Unix was one of the first operating systems to include all of its documentation online in machine-readable form. The "man page" format, first published in the UNIX Programmer's Manual in 1971, documented commands with terse reference information and is still in use today.

  • Catalyst for the Free Software and Open Source Movements: Unix's policy of widely distributing source code for educational use, coupled with its extensive online documentation, "raised programmer expectations" and contributed to the launch of the free software movement in 1983. This movement, led by Richard Stallman's GNU Project, aimed to create a free Unix-like system. The later development of the Linux kernel by Linus Torvalds in 1991, which was a "reimplementation of Unix from scratch", combined with GNU tools, created the GNU/Linux operating system, a powerful and widely adopted "Unix-like" alternative that adhered to the Unix philosophy. This also led to the development of other free Unix derivatives like FreeBSD, NetBSD, OpenBSD, and DragonFly BSD, which trace their lineage directly from original Unix source code.

Unix Wars and Fragmentation

Availability of source code and portability led to significant fragmentation and rivalry within the Unix ecosystem, primarily occurring from the late 1980s through the early 1990s. This era was characterised by multiple, often incompatible, versions of Unix stemming from different vendors and development branches.

Here's a breakdown of the Unix wars:

  • The Problem of Fragmentation and Incompatibility

    • By the late 1980s, the Unix landscape was plagued by a multitude of implementations. These were typically based on either AT&T's System V, the Berkeley Software Distribution (BSD), or a combination of the two, often with proprietary extensions.
    • This situation stifled the free exchange of source code and led to fragmentation and incompatibility. Although the trademark "UNIX" was ubiquitous, it was applied to a multitude of different, incompatible products.
    • Customers and vendors, such as Informix Corporation, which had over 1,000 product SKUs to serve these variants, began demanding standardisation.

  • Key Factions and Initiatives

    • The principal players in the Unix wars were broadly divided into two main camps:
      • AT&T's System V: AT&T, which initially owned Unix, sought to commercialise the operating system after being relieved of a 1956 antitrust decree in 1983. They introduced UNIX System V into the market.
      • Berkeley Software Distribution (BSD): Developed at the University of California, Berkeley, BSD continued to evolve as an alternative, partly in response to AT&T's more restrictive licensing of newer Unix versions.
    • In an attempt to unify the market, AT&T announced a pact with Sun Microsystems in 1987. This led to the formation of UNIX International (UI), which aimed to merge System V, BSD, SunOS, and Microsoft's Xenix into a single unified Unix, resulting in System V Release 4 (SVR4).
    • However, other vendors viewed this development with concern, believing their markets were threatened. In response, they formed the Open Software Foundation (OSF) to work on their own unified Unix, OSF/1. This division between the UI and OSF camps defined the "Unix wars".

  • Movement Towards Standardisation and Resolution

    • Several standardisation efforts emerged during this period:
      • System V Interface Definition (SVID): AT&T's initial standard, issued in 1985.
      • X/Open Consortium: Established by European computer vendors in 1984, this group aimed to create an open system specification based on Unix. X/Open continued to standardise APIs throughout the wars.
      • POSIX (Portable Operating System Interface): Published by the IEEE in 1988, POSIX was designed as a compromise API that could be implemented on both BSD and System V platforms. It was mandated by the US government for many systems in 1993.
    • The "Unix wars" continued into the 1990s but proved to be less destructive than initially feared.
    • A significant step towards resolution was the Common Open Software Environment (COSE) initiative in 1993, formed by the major Unix players, which marked the end of the most notorious phase of the wars.
    • This was followed by the merger of UI and OSF in 1994, creating a new combined entity that retained the OSF name.
    • In 1993, Novell, which had acquired AT&T's Unix System Laboratories, transferred the "UNIX" trademark and certification rights to the X/Open Consortium.
    • In 1996, X/Open merged with OSF to form The Open Group. The Open Group now owns the UNIX trademark and defines what constitutes a "UNIX" operating system through the Single UNIX Specification (SUS), ensuring compliance and compatibility for certified systems.

  • Impact and Legacy

    • Despite the intense competition, an industry analyst noted in 1989 that "Two Unixes are a lot better than 225".
    • The wars and the subsequent standardisation efforts solidified the Unix philosophy and architecture, paving the way for consistent interfaces.
    • The Unix wars also contributed to the rise of the free software movement. Richard Stallman's GNU Project, initiated in 1983, aimed to create a free Unix-like system, partly in response to the proprietary nature and fragmentation of Unix.
    • The Linux kernel, developed by Linus Torvalds from 1991, was a "reimplementation of Unix from scratch" that avoided the legal issues of original Unix source code, becoming a powerful "Unix-like" alternative.

The GNU Project

The GNU Project is a significant initiative in the history of computing, particularly for its role in the free software movement and its close relationship with the Linux operating system.

Here are the key aspects of the GNU Project:

  • Foundation and Purpose

    • The GNU Project was initiated by Richard Stallman in 1983. He formally announced his plans for a complete Unix-like operating system composed entirely of free software in September 1983, with development work beginning in January 1984.
    • The project's primary goal was to create a free (as in freedom) version of the Unix operating system. This "free" refers to the liberty to use, study, modify, and redistribute software, rather than merely its cost.
    • Stallman's motivation stemmed partly from his dissatisfaction with AT&T's shift towards a more restrictive, proprietary licensing model for Unix, which had previously been more open for academic use. He was "pissed" at AT&T for changing gears from an "open" to a restrictive model.

  • Key Contributions and Components

    • The GNU Project successfully developed a vast number of essential software components required for an operating system. By 1991, its "mid-level portions" were nearly complete, though it lacked a functional kernel.
    • Notable contributions include:
      • The GNU General Public License (GPL): Authored by Stallman in 1989, this license governs much of the free software created by the project and ensures its continued freeness.
      • GNU C Compiler (GCC): A crucial component for compiling code, including the Linux kernel itself.
      • Emacs text editor.
      • GNU Core Utilities (coreutils): A collection of fundamental Unix tools like ls, rm, cp, grep, and find.
      • GNU C Library (glibc): An implementation of the C standard library that acts as a wrapper for the Linux kernel's system calls.
      • Bash (Bourne-Again Shell): A popular command-line shell.
      • GNU Hurd: The kernel officially developed by GNU, which aimed for an ambitious microkernel design but proved difficult to implement and was largely incomplete when Linux emerged.
      • Other significant tools and libraries like GRUB bootloader, GNU Binutils, gzip, tar, gettext, grep, awk, sed, Findutils, gnupg, libgcrypt, gnutls, readline, ncurses, GNOME, Ghostscript, and GNU Chess.

  • Relationship with Linux

    • In 1991, Linus Torvalds independently released the Linux kernel. This kernel "fit into the last major gap in the GNU system," providing the missing kernel component that GNU Hurd had struggled to deliver.
    • Early Linux developers ported GNU code to run with the Linux kernel. This combination resulted in a "complete free system," which Richard Stallman and the Free Software Foundation advocate calling "GNU/Linux".
    • The "GNU/Linux naming controversy" highlights Stallman's argument that the name acknowledges the GNU Project's substantial contributions, particularly as the system as a whole is "basically GNU with Linux added". He believes that calling the entire operating system simply "Linux" conveys a mistaken idea of the system's origin, history, and purpose, and makes people "think it's all Linux, that it was all started by Mr. Torvalds in 1991".
    • While Android also uses the Linux kernel, the FSF acknowledges that "GNU/Linux" is not an appropriate name for such systems because they do not primarily use GNU components.

  • Impact and Philosophy

    • The GNU Project's idealism has been "extremely practical," leading to the creation of the free GNU/Linux operating system. It aims to inspire people to fight for their freedom and community in software development.
    • The project provides "software customization," allowing users to choose from different command-line shells and adapt the OS to their liking. It promotes transparency by enabling users to study, modify, and share the source code.
    • The GNU Project continues to be a driving force for free software, and many popular Linux distributions, such as Debian, Fedora, and Ubuntu, utilize significant GNU components.

Linux Kernel

The Linux kernel is a crucial component of the Linux operating system, often described as its heart or core. It is the fundamental program responsible for managing a computer's hardware resources and acting as an interface between the hardware and other software programs.

Here's a detailed overview of the Linux kernel:

  • Origin and Creation

    • The Linux kernel was developed by Linus Torvalds, who independently released its first version in 1991.
    • Torvalds was a student at the University of Helsinki when he began this project, partly out of frustration with the licensing limitations of the MINIX operating system, which at the time restricted it to educational use.
    • MINIX, a Unix-like system created by Professor Andrew S. Tanenbaum, served as a catalyst for Linux development, with Torvalds learning some system calls from Tanenbaum's text and initially developing the Linux kernel on MINIX.
    • Torvalds stated that if the GNU kernel (Hurd) or 386BSD had been available in 1991, he likely would not have created Linux.
    • Unlike the original Unix, the Linux kernel was written from scratch and is not derived from the original Unix source code, thus avoiding the legal issues prevalent at the time.

  • Relationship with GNU

    • The Linux kernel "fit into the last major gap" in the GNU system, which had been under development since 1983 by Richard Stallman to create a complete free Unix-like operating system but lacked a functional kernel.
    • Early Linux developers ported GNU code, including the GNU C Compiler, to work with the Linux kernel.
    • This combination of the Linux kernel with GNU components (such as the GNU C Library, GNU Core Utilities, GNU Compiler Collection, and Bash shell) resulted in a "complete free system".
    • Richard Stallman and the Free Software Foundation (FSF) advocate calling this combined system "GNU/Linux" to acknowledge the GNU Project's substantial contributions.
    • The FSF argues that the GNU Project set out to create an integrated system, and the Linux kernel filled a crucial missing piece. They believe that simply calling the system "Linux" attributes its origin and purpose solely to Torvalds, overlooking GNU's foundational role.
    • However, Linus Torvalds and many others prefer to call the whole operating system "Linux," arguing that the name is more widely used and that "GNU/Linux" is too cumbersome.
    • It's important to note that not all Linux-based systems use GNU components; for example, Google's Android uses the Linux kernel but does not primarily use GNU components. Distributions like Alpine Linux also stand out as non-GNU Linux-based operating systems, often using BusyBox and musl instead of traditional GNU libraries.

  • Relationship with Unix

    • The Linux kernel is a Unix-like operating system kernel. This means it behaves like Unix and adheres to Unix design principles and standards, such as POSIX.
    • However, unlike BSDs (FreeBSD, NetBSD, OpenBSD, DragonFly BSD), which trace their software revisions directly back to the original AT&T Unix source code, Linux was developed independently and does not share that direct source code lineage.
    • Despite this, Linux (and its distributions) maintain similar commands and file structures to Unix due to this inspiration and adherence to standards. Many common Unix commands like ls, rm, cp, grep, and find are also found in Linux.

  • Design and Features

    • The Linux kernel uses a monolithic kernel design, where essential functions like memory management, process scheduling, and hardware drivers are tightly integrated.
    • However, it incorporates modularity through loadable kernel modules (LKMs), allowing dynamic expansion of kernel functionality without requiring a full system reboot. Device drivers are often integrated directly or added as modules.
    • The kernel handles process control, networking, access to peripherals, and file systems.
    • The Linux kernel is licensed under the GNU General Public License (GPL) version 2, which means its source code can be freely used, modified, and distributed. Linus Torvalds does not plan to move the kernel to GPLv3.
    • It supports a wide variety of hardware architectures, from personal computers and smartphones to mainframes and supercomputers, due to its portability and the community's efforts.

  • Usage and Popularity

    • The Linux kernel powers a vast majority of devices today, including servers, mainframes, and smartphones (via Android).
    • It is the predominant operating system for servers, with estimates suggesting it runs on over 96.4% of the top one million web servers and at least 80% of public cloud workloads.
    • All of the world's 500 fastest supercomputers use Linux.
    • Linux distributions, which combine the Linux kernel with other software components, have made Unix technologies accessible to home users and are used in a wide variety of applications.

Key Contributors

The development of Unix involved several key figures, primarily from Bell Labs, whose work laid the foundation for modern operating systems, including Unix-like systems such as Linux. Their contributions were instrumental in shaping the software landscape.

Here are the key contributors to Unix and their specific contributions:

  • Ken Thompson

    • Started the development of Unix (initially called UNICS, a pun on Multics) in 1969. His desire to make his "Space Game" program run economically drove the initial implementation.
    • He was a researcher at Bell Labs experimenting with operating system designs.
    • He wrote a custom interface for a disk drive on a PDP-7 that evolved into the prototype Unix system.
    • Along with Dennis Ritchie, he rewrote the Unix kernel in C in 1973.
    • He also developed the first version of his B programming language.
    • He received the Turing Award in 1983 for his work on Unix, and was jointly awarded the Japan Prize in 2011. He was also inducted into the National Inventors Hall of Fame in 2019 for his invention of Unix.

  • Dennis Ritchie

    • A key figure in Unix's creation alongside Ken Thompson.
    • Wrote the first C compiler [C Language, 151], and was the primary developer of the C programming language.
    • Rewrote the Version 4 Unix kernel in the higher-level C language in 1973 with Ken Thompson [C Language, 151, 169, 230, 320, 325, 391, 401, 434]. This was a "key pioneering approach" that went "contrary to the general notion at the time that an operating system's complexity and sophistication required it to be written in assembly language" [C Language, 169, 230].
    • He introduced the crucial concept of a device file, abstracting hardware devices as files within the file system.
    • He attributed Unix's early success to the "readability, modifiability, and portability of its software that in turn follows from its expression in high-level languages" like C [C Language, 327].
    • He viewed both Linux and BSD Unix operating systems as a "continuation of the basis of the Unix design" and "derivatives of Unix".
    • He received the Turing Award in 1983 for his work on Unix, and was jointly awarded the Japan Prize in 2011. He was also inducted into the National Inventors Hall of Fame in 2019 for his invention of Unix.

  • Brian Kernighan

    • He is credited with suggesting the name "Unics" (a pun on Multics), which was later shortened to Unix.
    • Along with Rob Pike, he summarised the Unix philosophy: "the idea that the power of a system comes more from the relationships among programs than from the programs themselves" [Unix Philosophy, 387].

  • Douglas McIlroy (Doug McIlroy)

    • One of the last researchers to leave the Multics project, who decided to redo the work on a smaller scale with Thompson and Ritchie.
    • He ported the TMG compiler-compiler to PDP-7 assembly, creating the first high-level language running on Unix.
    • He is also credited by Dennis Ritchie with the name Unics.

  • Joe Ossanna

    • He was part of the Bell Labs team that developed Unix, having also been involved in the Multics project.

  • Bill Joy

    • While Ken Thompson spent a sabbatical at the University of California at Berkeley, Bill Joy and Chuck Haley wrote the first Berkeley version of Unix, known as BSD (Berkeley Software Distribution).
    • He later co-founded Sun Microsystems in 1982 and created SunOS.

  • Richard Stallman

    • Founded the Free Software Foundation (FSF) in 1984.
    • Initiated the GNU Project in 1984 with the goal of creating a free (as in freedom) version of the Unix operating system. This was partly in response to AT&T's move towards a more restrictive, proprietary licensing model for Unix.
    • Authored the GNU General Public License (GPL).
    • The GNU Project successfully built a vast number of essential components, including the GNU C Compiler (GCC), the Emacs text editor, and other fundamental tools, which became crucial for the later development of Linux-based systems.
    • He is a prominent advocate for the use of the term "GNU/Linux" to acknowledge the GNU Project's substantial contributions to the full operating system often referred to simply as "Linux".

  • Linus Torvalds

    • Developed the Linux kernel, releasing the first version in 1991.
    • His project was a "reimplementation of Unix from scratch", avoiding the legal issues faced by Unix derivatives that used AT&T's original source code.
    • He initially developed the Linux kernel on the MINIX operating system.
    • He decided to release the Linux kernel under the GNU General Public License (GPL).
    • He serves as the lead maintainer for the Linux kernel.
    • He stated that if the GNU kernel (Hurd) or 386BSD had been available in 1991, he likely would not have created Linux, highlighting the opportune timing for his project.

Historical Milestones

The Birth of Unix (1969-1970)

Unix originated at Bell Labs when Ken Thompson and Dennis Ritchie developed it as a reaction to the complexity of the MULTICS project. Written initially in assembly language for the PDP-7, Unix embodied the philosophy of simplicity and modularity that would define its future development.

Key Evolutionary Milestones

1971: First Unix manual published, documenting the system's core concepts and commands.

1973: Unix rewritten in C programming language, making it highly portable across different hardware platforms. This decision proved revolutionary, as it enabled Unix to transcend specific hardware limitations.

1975: Unix Version 6 released to universities, spreading Unix culture throughout academic institutions and creating a generation of Unix-literate developers.

1977-1979: Berkeley Software Distribution (BSD) emerges from University of California, Berkeley, introducing networking capabilities and the vi editor.

1982: Sun Microsystems founded, commercializing Unix workstations and advancing network computing.

1984: AT&T releases System V, establishing one of the two major Unix lineages alongside BSD.

1988: POSIX standard published, attempting to standardize Unix interfaces across different implementations.

1991: Linus Torvalds begins developing Linux, creating a Unix-like system available under free software licenses.

1993: FreeBSD released, continuing the BSD tradition in the open-source era.

1995: OpenBSD forks from NetBSD, emphasizing security and code correctness.

The BSD World: A Unix Legacy

Historical Context

The Berkeley Software Distribution emerged from the University of California's Computer Systems Research Group in the late 1970s. BSD introduced networking capabilities, the Berkeley Fast File System, and numerous utilities that became standard across Unix implementations.

Modern BSD Variants

FreeBSD: Focuses on performance and ease of use, popular for servers and embedded systems. Provides excellent documentation and a cohesive system design.

NetBSD: Emphasizes portability across hardware platforms, supporting more architectures than any other Unix-like system.

OpenBSD: Prioritizes security and code correctness, conducting regular security audits and implementing innovative security features.

BSD Philosophy

BSD systems maintain the traditional Unix philosophy of providing a complete, integrated system rather than a collection of separate components. This approach results in consistent behavior and comprehensive documentation, but may limit flexibility compared to Linux distributions.

Unix-like

The term "Unix-like" describes a broad category of operating systems that share a common structure and behave similarly to the original Unix operating system, even if they are not directly derived from its source code or officially certified as "UNIX". This distinction is crucial as "UNIX" (all caps) is a registered trademark owned by The Open Group, and only operating systems that pay for and pass their certifications can officially bear this trademark.

Here's more about Unix-like systems:

  • Core Principles and Design

    • Unix-like systems adhere to Unix design principles and standards, such as POSIX (Portable Operating System Interface). POSIX was created to standardise Unix-derived operating systems so they would "look and feel the same".
    • They typically follow the "Unix philosophy", which includes ideas like:
      • "Make each program do one thing well".
      • "Write programs that work together".
      • "Write programs that handle text streams" (everything is text).
    • They share a Unix architecture, featuring a unified file system that often uses inter-process communication through pipes, and are executed from a Unix shell. The common denominator in the I/O system is the byte, unlike "record-based" file systems, which made Unix pipes especially useful and encouraged simple, general tools.
    • Both Unix and Unix-like systems follow a multi-user, multitasking model.
    • Many of the command-line tools and their syntax are very similar to those found in original Unix, such as ls, rm, cp, grep, and find. This familiarity means that knowledge of Unix commands translates well to Linux systems.

  • Relationship with Linux

    • Linux is the most popular descendant and a prominent example of a Unix-like operating system.
    • Linus Torvalds developed the Linux kernel from scratch in 1991, not from the original Unix source code, thus avoiding the legal issues of the time.
    • The Linux kernel was created as a free alternative to the MINIX operating system and was inspired by Unix. Torvalds stated that if the GNU kernel or 386BSD had been available in 1991, he likely would not have created Linux.
    • Linux distributions, which combine the Linux kernel with other software components, effectively made Unix technologies accessible to home users on personal computers, whereas previously they had been confined to sophisticated workstations.

  • Relationship with BSD

    • BSDs (FreeBSD, NetBSD, OpenBSD, DragonFly BSD) are also Unix-like operating systems.
    • Technically, BSDs are considered more directly "Unix" than Linux due to their direct source code lineage tracing back to the original AT&T Unix. While Linux was a reimplementation, BSD evolved from code licensed from AT&T Unix at the University of California, Berkeley. Legal disputes in the early 1990s eventually clarified Berkeley's right to distribute BSD Unix freely.
    • Dennis Ritchie, one of Unix's creators, viewed both Linux and BSD as a "continuation of the basis of the Unix design and are derivatives of Unix".

  • Examples of Unix-like Systems

    • Many popular Linux distributions, such as Ubuntu, Fedora, Debian, Mint, Arch Linux, and Red Hat Enterprise Linux, are Unix-like.
    • macOS is a Unix-like operating system, based on BSD and the Mach kernel, and is even a certified UNIX operating system.
    • Other examples include Android (which uses the Linux kernel but not primarily GNU components), ChromeOS, and embedded systems.

  • Impact and Usage

    • Unix-like systems are highly portable and can run on a wide variety of hardware architectures, from personal computers and smartphones to mainframes and supercomputers.
    • They are the predominant operating system for servers (over 96.4% of the top one million web servers run Linux).
    • All of the world's 500 fastest supercomputers use Linux.
    • Their open-source nature, particularly for Linux, allows for extensive customisation, flexibility, and a large, active community support system, distinguishing them from traditional proprietary Unix systems.

UNIX™

The term "UNIX" (in all caps) is a registered trademark owned by The Open Group1. Only operating systems that pay for and pass their stringent certifications can officially bear this trademark, signifying compliance with standards like the Single UNIX Specification (SUS).

Here is a detailed overview of UNIX:

  • Origin and History

    • UNIX was conceived and implemented starting in 1969 at AT&T Bell Labs.
    • Key figures in its creation were Ken Thompson and Dennis Ritchie.
    • It originated from their work on the experimental time-sharing operating system called Multics, a joint research effort with General Electric and MIT. After AT&T withdrew from Multics due to its complexity, Thompson and Ritchie aimed to implement the best ideas from Multics on a smaller scale, specifically on a little-used PDP-7 minicomputer.
    • The name "UNICS" (UNiplexed Information and Computing Service) was initially suggested as a pun on Multics in 1970, with Brian Kernighan often credited for the name, which later became "UNIX".
    • Initially written in assembly language, a groundbreaking move occurred in 1973 when Version 4 Unix was rewritten almost entirely in the C programming language by Dennis Ritchie. This decision made UNIX highly portable, allowing it to "outlive its original hardware". The C language itself offered an "unprecedented combination of efficiency and expressiveness to programming".

  • Early Distribution and Commercialization

    • Due to a 1956 antitrust consent decree, AT&T was initially forbidden from entering the computer business, which led them to license Unix's source code freely or at nominal fees to educational institutions and, later, to companies.
    • This "pragmatically open" distribution to universities, particularly the University of California, Berkeley, fostered widespread adoption and development within the academic community.
    • However, after the 1983 breakup of the Bell System (due to a second antitrust case), AT&T was relieved of this obligation and promptly began selling UNIX as a proprietary product, restricting users' rights to modify it. This shift led to fragmentation and incompatibility issues among different UNIX versions.

  • UNIX Variants and the "Unix Wars"

    • The proprietary licensing led to many different versions (variants) of UNIX, such as UC Berkeley's Berkeley Software Distribution (BSD), AT&T's System V (SVR4), Sun's Solaris, IBM's AIX, HP's HP-UX, and Microsoft's Xenix.
    • The "Unix wars" in the late 1980s and early 1990s were a period of intense rivalry and efforts to standardize these diverging implementations.
    • Standardization efforts included AT&T's System V Interface Definition (SVID) and the IEEE's POSIX specification (Portable Operating System Interface). POSIX aimed to create a common API that could be implemented on both BSD and System V platforms.
    • In 1992, AT&T sold its UNIX System Laboratories to Novell, which later transferred the "UNIX" trademark and certification rights to The Open Group.

  • Design and Features

    • UNIX systems are characterized by a modular design often called the "Unix philosophy". This philosophy advocates for:
      • Making "each program do one thing well".
      • Writing programs that "work together".
      • Writing "programs that handle text streams" (everything is text).
    • They typically feature a unified file system that uses inter-process communication through "pipes," executed from a Unix shell. The focus on byte-based I/O and text streams made pipes exceptionally useful and encouraged simple, general tools that could be combined for complex tasks.
    • UNIX supports a multi-user, multitasking model.
    • The kernel is the core component, handling essential tasks like memory management, process scheduling, and device control. UNIX systems typically employ a monolithic kernel design, where these functions are tightly integrated.
    • Initially, UNIX primarily used a Command-Line Interface (CLI). While powerful, this could pose a learning curve for new users. Later, graphical user interfaces (GUIs) like the Common Desktop Environment (CDE) were developed for some UNIX systems.
    • Common commands like ls, rm, cp, grep, and find are fundamental to UNIX and are also found in Unix-like systems.

  • Relationship to Linux and Other Unix-like Systems

    • The Linux kernel, developed by Linus Torvalds in 1991, was written from scratch and is not derived from the original UNIX source code. Instead, it was inspired by UNIX and its design principles, and was created partly as a free alternative to MINIX (another Unix-like system).
    • Due to their shared philosophy and adherence to standards like POSIX, Linux is considered a "Unix-like" operating system.
    • BSD systems (FreeBSD, NetBSD, OpenBSD, DragonFly BSD) are also Unix-like. However, they are considered more directly "Unix" than Linux because their source code lineage traces back to the original AT&T Unix.
    • macOS (formerly Mac OS X), developed by Apple, is a prominent example of a certified UNIX operating system, based on BSD and the Mach kernel.
    • Dennis Ritchie, one of UNIX's creators, viewed both Linux and BSD as a "continuation of the basis of the Unix design and are derivatives of Unix".

  • Usage and Market Share

    • UNIX was once the dominant operating system for servers, minicomputers, and mainframes.
    • Today, UNIX continues to be used in specific fields, particularly in legacy systems, finance, telecommunications, and high-end server operations by large corporations. Examples include IBM AIX, Oracle Solaris, and HP-UX.
    • However, Linux has largely surpassed UNIX in overall market share, especially in servers and supercomputers. All of the world's 500 fastest supercomputers use Linux.

  • Key Differences and Similarities with Linux

    • Licensing: UNIX is proprietary and typically requires licensing fees, with access to source code often restricted. Linux is open-source and free, governed by licenses like the GNU General Public License (GPL), allowing free use, modification, and redistribution of its source code.
    • Development Model: UNIX historically followed a more centralised, vendor-controlled development model, leading to slower updates and adaptation. Linux thrives on a distributed, community-driven model, resulting in frequent updates, quick bug fixes, and rapid evolution.
    • Source Code: UNIX's source code is generally not publicly accessible. Linux's source code is open to the public.
    • Hardware Compatibility: UNIX tends to be limited to specific hardware platforms (often proprietary and RISC systems). Linux is highly portable and compatible with a vast range of hardware, from PCs and smartphones to supercomputers.
    • User Interface: While both have a Command-Line Interface (CLI), UNIX was initially command-based with GUIs like CDE developing later. Linux offers a wider variety of GUI desktop environments (e.g., GNOME, KDE, Xfce) and is more easily coupled with GUIs.
    • Commands: Both share many similar command-line utilities due to adherence to POSIX standards, such as ls, cd, mkdir, rm, and grep. However, UNIX commands may have fewer options and can vary more between variants.
    • Security: Both prioritise security. UNIX is often seen as secure due to its controlled, proprietary environment. Linux, through its open-source community, benefits from continuous scrutiny and rapid security updates.
    • Cost: UNIX typically involves licensing fees, making it more expensive. Linux is generally free (though paid enterprise versions and support are available).
    • Core Similarities: Both are multi-user and multitasking systems, feature a hierarchical file system, provide robust support for programming languages (like C, C++, Python, Shell scripting), and have strong networking capabilities. Both kernels are largely monolithic in design.

  • Impact

    • UNIX made large-scale networking of diverse computing systems, and the Internet, practical. Its environment and the client-server program model were essential to the development of the Internet.
    • Its design principles, such as modularity and text-based tools, heavily influenced subsequent operating systems and software engineering practices.
    • The UNIX programming interface became the basis for the widely implemented POSIX standard. The C programming language, developed for and with UNIX, also became ubiquitous.
    • The open-source movement, including the GNU Project, was partly inspired by the extensive on-line documentation and ready access to UNIX source code in its academic days.

  1. Historically, AT&T owned the trademark until the 1990s, when they sold it to Novell. Novell then transferred the UNIX trademark and certification rights to the X/Open Consortium in 1993, which later merged with the Open Software Foundation (OSF) to form The Open Group in 1996. ↩

Licensing

Licensing models in the software domain, particularly in the context of UNIX and Linux, have distinct philosophies and implications for use, modification, and distribution. These licences largely define the "open source" or "free software" nature of a system, contrasting with proprietary models.

Here's an overview of key licensing types:

  • Free Software and Open Source Software At its core, the concept of "free software" (often used interchangeably with "open source software") means that the source code is freely available for use, viewing, modification, and redistribution. Richard Stallman, founder of the Free Software Foundation (FSF), began the GNU project in 1984 with the goal of creating a free version of the Unix operating system. For Stallman, "free" refers to freedom, similar to "free speech, not free beer," meaning users have the liberty to run, copy, distribute, study, change, and improve the software as needed. The FSF actively promotes and defends this user freedom.

  • GNU General Public License (GPL)

    • The GPL is a "copyleft" license. This means that any software derived from GPL-licensed code must also be released under the same or a compatible GPL license. This ensures that the software, and its future modifications, remains free and open for others to use and modify.
    • The Linux kernel itself, developed by Linus Torvalds, is licensed under the GPL version 2 (GPLv2). This licence was chosen to make the fledgling operating system freely available.
    • Many of the key components of a typical Linux distribution, particularly those from the GNU Project (like the GNU C Library, GNU Core Utilities, and GNU Compiler Collection), are also primarily licensed under the GPL.
    • The GPL has played a significant role in Linux's success by promoting openness and distributed, community-driven development, leading to frequent updates, quick bug fixes, and rapid evolution of features.

  • GNU General Public License, Version 3 (GPLv3)

    • While many components of the GNU system use GPL, Linus Torvalds has explicitly stated that the Linux kernel will not move from GPLv2 to GPLv3.
    • His primary reason for this stance is a dislike of certain provisions in GPLv3 that prohibit the use of the software in digital rights management (DRM).
    • Additionally, moving to GPLv3 would be impractical due to the immense effort required to obtain permission from the thousands of copyright holders who have contributed to the Linux kernel.

  • BSD License

    • The BSD license is a permissive free software license, contrasting with the "copyleft" nature of the GPL.
    • It allows users to freely use, modify, and redistribute the source code, including in proprietary software, without requiring that derived works also be open-source or under the same license. This means you can take BSD-licensed code, make changes, and distribute it privately or commercially without revealing your modifications.
    • Berkeley Software Distribution (BSD) systems (such as FreeBSD, NetBSD, OpenBSD, and DragonFly BSD) trace their source code lineage directly back to the original AT&T Unix code that was licensed to the University of California at Berkeley. They are often considered more directly "Unix" than Linux due to this inheritance.
    • While BSD systems themselves are open source, their permissive license means their code can be incorporated into proprietary systems; for example, macOS is based on BSD and the Mach kernel. Google's Android operating system also uses some BSD-derived userland tools.

  • MIT License

    • The MIT License is another permissive free software license, similar in philosophy to the BSD license.
    • It is known for its brevity and simplicity, imposing very few restrictions on reuse.
    • An example of its use cited in the sources is the X.Org implementation of the X Window System, which is licensed under the MIT License.

Key Differences and Impact:

The fundamental distinction lies in their impact on derivative works:

  • GPL (Copyleft): Ensures that software remains free by requiring modifications and derived works to also be open-source under the GPL. This fosters a continuously growing pool of open-source software that cannot be "closed off" by commercial entities.
  • BSD/MIT (Permissive): Offers maximum flexibility for developers and businesses, allowing the integration of open-source code into proprietary systems without imposing reciprocal sharing requirements. This has enabled wider commercial adoption of some Unix-like technologies, as seen with macOS.

Historically, AT&T's initial proprietary licensing of Unix, which restricted users' rights to modify it, led to fragmentation and incompatibility among different Unix versions. The emergence of licenses like GPL and BSD, in contrast, facilitated the growth of the free and open-source software movement, providing alternatives that encouraged sharing and collaboration.

Standards

Standards play a crucial role in the evolution and interoperability of operating systems, especially within the Unix and Linux ecosystems, due to the historical fragmentation of Unix into various incompatible versions. The push for standardisation was driven by a need for consistency and interworking among diverse Unix systems.

Here's an overview of key standardisation efforts:

POSIX (Portable Operating System Interface)

POSIX is a set of open operating system standards developed by the IEEE (Institute of Electrical and Electronics Engineers). Its primary goal was to provide a common baseline or API (Application Programming Interface) for operating systems. This standard was designed as a compromise API readily implementable on both BSD and System V platforms, which were the two major divergent lines of Unix development.

  • Purpose: The standard aims to ensure that programs written for POSIX-compliant systems can run on any other substantially or wholly compliant operating system. This promotes portability of applications across different Unix-like environments.
  • Government Mandate: In 1993, POSIX was mandated by the United States government for many of its own systems.
  • Relationship with Linux: Linux systems adhere to POSIX standards. Linux commands, for instance, are designed to be as similar to Unix commands as possible by adhering to POSIX standards. While many Linux distributions are "Unix-like" and largely implement POSIX, only a few have been officially certified as POSIX.1 compliant, such as Linux-FT.
  • Relationship with Windows: Interestingly, the sources indicate that Windows has also chosen to follow the POSIX standard.

The Single UNIX Specification (SUS) and UNIX® Trademark

The term UNIX (all caps) is a registered trademark. It is now owned and administered by The Open Group, an industry standards consortium.

  • Certification: Only operating systems that pay for and pass stringent certifications can officially bear the UNIX trademark. This certification signifies compliance with the Single UNIX Specification (SUS).
  • Definition: The Open Group dictates that "UNIX" refers more to a class of operating systems than a specific implementation. The SUS defines the structure and behaviour of a compliant system, not its implementation.
  • Certified Systems: Several commercial Unix variants are certified UNIX operating systems, including IBM AIX, HP-UX, Oracle Solaris, and Apple's macOS (which is based on BSD and the Mach kernel). Notably, some Linux distributions, such as Huawei EulerOS and Inspur K-UX, have also achieved UNIX 03 certification.
  • "Unix-like" vs. "UNIX": Many Unix variants (like most BSDs) and Linux distributions are considered "Unix-like" because they behave similarly and adhere to Unix design principles and POSIX standards, but they are not officially certified as UNIX, often due to the cost of certification.
  • Trademark Usage: The Open Group requests that UNIX always be used as an adjective followed by a generic term (e.g., "UNIX system") to prevent it from becoming a genericised trademark. Historically, the all-caps "UNIX" styling came from the use of small caps in early documentation.

The Unix Wars and Standardisation Efforts

The original proprietary licensing of Unix by AT&T led to significant fragmentation and incompatibility among different Unix versions. This era, known as the Unix wars, saw fierce rivalry between vendors who based their systems on either AT&T's System V or the Berkeley Software Distribution (BSD). Customers and vendors alike demanded standardisation due to the difficulty of ensuring compatibility across the multitude of incompatible Unix products.

  • Initial Attempts: AT&T responded by issuing the System V Interface Definition (SVID) in 1985, requiring conformance for "System V" branding.
  • Consortia Formation: In 1984, European vendors formed the X/Open consortium to create an open system specification based on Unix. The Open Software Foundation (OSF) was formed by other vendors in response to AT&T's collaboration with Sun Microsystems to unify Unix variants.
  • Unification: The creation of the Common Open Software Environment (COSE) initiative in 1993, by major Unix players, marked the end of the most notorious phase of the Unix wars. This was followed by the merger of UI (UNIX International, the AT&T/Sun faction) and OSF in 1994, forming The Open Group.
  • Unified Standard: The Open Group continues to develop and evolve the Single UNIX Specification, which, along with the IEEE POSIX specification, became the Open Group Base Specification by 2008. This separation of the UNIX trademark from any specific code stream and the stability of core interfaces encouraged the development of a rich set of software tools, with the open-source movement building on this foundation.

The underlying Unix philosophy, which advocates for small, single-purpose tools that work together by handling text streams and using pipes for inter-process communication, also functions as a de facto set of design standards that both Unix and Unix-like systems, including Linux, continue to follow.

Conclusion

Unix is an operating system with a rich history, originating from AT&T Bell Labs in the late 1960s. It was initially proprietary, though its source code was often freely licensed to universities. A pivotal moment in its development was its rewriting in the C programming language in 1973, which significantly enhanced its portability across diverse computer platforms.

The core of Unix is its "Unix philosophy", which advocates for a modular design. This means using small, single-purpose tools that work together by handling text streams and using pipes for inter-process communication. It also features a hierarchical file system. Historically, the proprietary licensing led to many divergent and incompatible Unix versions, an era famously known as the "Unix wars".

This fragmentation led to a crucial push for standardisation. Key efforts include POSIX (Portable Operating System Interface), developed by IEEE as a compromise API for different Unix branches, and the Single UNIX Specification (SUS), now administered by The Open Group. Today, UNIX® (all caps) is a registered trademark owned by The Open Group. Only operating systems that pay for and pass stringent certifications can officially bear this trademark, signifying compliance with the SUS. This distinguishes certified UNIX systems from "Unix-like" ones.

While Linux is a "Unix-like" operating system, inspired by Unix and largely adhering to POSIX standards, it was written from scratch by Linus Torvalds and his community, and is not derived from the original Unix source code. In contrast, BSD systems (like FreeBSD or macOS, which is a certified UNIX operating system) are considered more directly "Unix" due to their genealogical link to the original AT&T Unix source code.

Although Linux has largely surpassed Unix in popularity across many modern computing domains, including web servers, supercomputers, and mobile devices, proprietary Unix variants such as IBM AIX, HP-UX, and Oracle Solaris continue to be widely used in high-end, mission-critical enterprise systems. These systems are valued for their robustness, stability, and ability to handle high workloads efficiently.