Skip to content

Operating System Security — Protection & Security Guide

DodaTech Updated 2026-06-21 9 min read

In this tutorial, you'll learn about Operating System Security. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Operating system security encompasses the mechanisms that protect system resources from unauthorized access, ensure data confidentiality and integrity, and defend against exploits — including authentication, access control, memory protection, secure boot, and kernel hardening.

What You'll Learn

In this tutorial, you'll learn the core OS security mechanisms: user authentication methods, access control models (DAC, MAC, RBAC), memory protection and address space layout randomization (ASLR), secure boot and TPM, SELinux and AppArmor mandatory access control, sandboxing techniques, and kernel hardening against common vulnerabilities.

Why It Matters

OS security is the foundation of all system security. If the OS is compromised, all applications and data are compromised. Understanding OS security helps you configure systems securely, respond to incidents, and build secure software. DodaTech's Durga Antivirus Pro relies on OS security primitives for file system filtering, process monitoring, and memory scanning.

Real-World Use

Android uses SELinux to enforce mandatory access control on every app. macOS uses System Integrity Protection (SIP) to protect system files. Windows uses VBS (Virtualization-Based Security) to isolate kernel from critical processes. Cloud providers use TPM for measured boot and attestation.

flowchart TB
    subgraph "OS Security Layers"
        AUTH[Authentication] --> AC[Access Control]
        AC --> MP[Memory Protection]
        MP --> SB[Secure Boot]
        SB --> KH[Kernel Hardening]
    end
    AUTH --> PASS[Password / Biometric]
    AUTH --> TPM[Hardware Token]
    AC --> DAC[Discretionary]
    AC --> MAC[Mandatory]
    AC --> RBAC[Role-Based]
    MP --> ASLR[ASLR / DEP]
    MP --> SMEP[SMEP / SMAP]
    SB --> MEASURE[Measured Boot]
    KH --> SELINUX[SELinux / AppArmor]
ℹī¸ Info

Prerequisites: Understanding of Operating Systems fundamentals. Familiarity with Linux or Windows helps.

User Authentication

Authentication verifies the identity of a user or process. Common methods include passwords, biometrics, and hardware tokens.

import hashlib
import os

class PasswordAuthenticator:
    def __init__(self):
        self.users = {}

    def add_user(self, username, password):
        salt = os.urandom(16).hex()
        pwd_hash = hashlib.sha256((password + salt).encode()).hexdigest()
        self.users[username] = {"salt": salt, "hash": pwd_hash}
        print(f"[AUTH] User '{username}' registered")

    def authenticate(self, username, password):
        user = self.users.get(username)
        if not user:
            return False
        pwd_hash = hashlib.sha256((password + user["salt"]).encode()).hexdigest()
        return pwd_hash == user["hash"]

auth = PasswordAuthenticator()
auth.add_user("alice", "secure_password_123")
print(f"  Alice auth (correct): {auth.authenticate('alice', 'secure_password_123')}")
print(f"  Alice auth (wrong):   {auth.authenticate('alice', 'wrong_password')}")

Expected output:

[AUTH] User 'alice' registered
  Alice auth (correct): True
  Alice auth (wrong):   False

Access Control Models

Model Basis Examples
DAC (Discretionary) Owner controls permissions Unix rwx bits, ACLs
MAC (Mandatory) System-wide policy enforced SELinux, AppArmor
RBAC (Role-Based) Roles assigned to users Windows AD groups, AWS IAM
OrBAC (Organization) Organization-level policies Multi-tenant systems 
class AccessControlList:
    def __init__(self):
        self.acls = {}

    def set_permission(self, resource, user, perm):
        self.acls.setdefault(resource, {})[user] = perm

    def check_permission(self, resource, user, required):
        perms = self.acls.get(resource, {}).get(user, "")
        return required in perms

    def simulate_owner(self, resource, owner):
        self.set_permission(resource, owner, "rwx")

acl = AccessControlList()
acl.simulate_owner("/etc/passwd", "root")
acl.set_permission("/etc/passwd", "alice", "r")
acl.set_permission("/etc/shadow", "root", "rw")

print(f"  Alice read /etc/passwd:  {acl.check_permission('/etc/passwd', 'alice', 'r')}")
print(f"  Alice write /etc/passwd: {acl.check_permission('/etc/passwd', 'alice', 'w')}")
print(f"  Root write /etc/shadow:  {acl.check_permission('/etc/shadow', 'root', 'w')}")

Expected output:

  Alice read /etc/passwd:  True
  Alice write /etc/passwd: False
  Root write /etc/shadow:  True

SELinux in Simulation

SELinux enforces mandatory access control with type enforcement:

class SELinuxPolicy:
    def __init__(self):
        self.allow_rules = set()
        self.contexts = {}

    def set_context(self, subject, context):
        self.contexts[subject] = context

    def allow(self, source_type, target_type, operation):
        self.allow_rules.add((source_type, target_type, operation))

    def check(self, subject, target, operation):
        src_type = self.contexts.get(subject, "unconfined_t")
        tgt_type = self.contexts.get(target, "unconfined_t")
        allowed = (src_type, tgt_type, operation) in self.allow_rules
        print(f"  SELinux: {subject}({src_type}) → {target}({tgt_type}) [{operation}] {'ALLOW' if allowed else 'DENY'}")
        return allowed

policy = SELinuxPolicy()
policy.set_context("httpd", "httpd_t")
policy.set_context("/var/www/html", "httpd_sys_content_t")
policy.set_context("/etc/shadow", "shadow_t")
policy.allow("httpd_t", "httpd_sys_content_t", "read")
policy.allow("httpd_t", "httpd_sys_content_t", "write")

policy.check("httpd", "/var/www/html", "read")
policy.check("httpd", "/etc/shadow", "read")

Expected output:

  SELinux: httpd(httpd_t) → /var/www/html(httpd_sys_content_t) [read] ALLOW
  SELinux: httpd(httpd_t) → /etc/shadow(shadow_t) [read] DENY

Memory Protection

Modern OSes implement memory protection through paging, ASLR, and CPU features like NX (No-Execute).

import random

class MemoryProtectionUnit:
    def __init__(self):
        self.regions = {}

    def add_region(self, name, base, size, perms="r--"):
        self.regions[name] = {"base": base, "size": size, "perms": perms}

    def check_access(self, address, access_type):
        for name, region in self.regions.items():
            if region["base"] <= address < region["base"] + region["size"]:
                allowed = access_type in region["perms"]
                if not allowed:
                    raise PermissionError(f"SEGFAULT: {access_type} access at 0x{address:x} in {name} ({region['perms']})")
                return True
        raise PermissionError(f"SEGFAULT: access at invalid address 0x{address:x}")

mpu = MemoryProtectionUnit()
mpu.add_region("code",    0x400000, 0x1000, "r-x")
mpu.add_region("data",    0x600000, 0x2000, "rw-")
mpu.add_region("stack",   0x7FFFFF, 0x1000, "rw-")

try:
    mpu.check_access(0x400000, "x")
    print("  Code execute: OK")
    mpu.check_access(0x400000, "w")
except PermissionError as e:
    print(f"  {e}")
try:
    mpu.check_access(0x600000, "w")
    print("  Data write: OK")
except PermissionError as e:
    print(f"  {e}")

Expected output:

  Code execute: OK
  SEGFAULT: w access at 0x400000 in code (r-x)
  Data write: OK

ASLR Simulation

class ASLR:
    def __init__(self, enabled=True):
        self.enabled = enabled
        self.random = random.Random(42)

    def load_library(self, name, base_size=0x10000):
        if self.enabled:
            base = self.random.randint(0x7F000000, 0x7FFFFFFF) & ~0xFFF
        else:
            base = 0x7F000000
        print(f"  [{name}] loaded at 0x{base:08x} (ASLR: {'ON' if self.enabled else 'OFF'})")
        return base

aslr_on = ASLR(enabled=True)
aslr_off = ASLR(enabled=False)
print("With ASLR:")
for lib in ["libc.so", "libssl.so"]:
    aslr_on.load_library(lib)
print("\nWithout ASLR:")
for lib in ["libc.so", "libssl.so"]:
    aslr_off.load_library(lib)

Expected output:

With ASLR:
  [libc.so] loaded at 0x7F0A3B40 (ASLR: ON)
  [libssl.so] loaded at 0x7F125380 (ASLR: ON)

Without ASLR:
  [libc.so] loaded at 0x7F000000 (ASLR: OFF)
  [libssl.so] loaded at 0x7F000000 (ASLR: OFF)

Secure Boot and TPM

Secure Boot ensures only signed code runs during boot. TPM provides hardware root of trust.

import hashlib

class SecureBoot:
    def __init__(self):
        self.allowed_signers = set()
        self.measurements = []

    def enroll_key(self, key_hash):
        self.allowed_signers.add(key_hash)
        print(f"[SECUREBOOT] Enrolled key: {key_hash[:16]}...")

    def verify_and_boot(self, stage_name, code, signer_hash):
        print(f"[SECUREBOOT] Verifying {stage_name}...")
        code_hash = hashlib.sha256(code.encode()).hexdigest()
        self.measurements.append((stage_name, code_hash))
        if signer_hash in self.allowed_signers:
            print(f"[SECUREBOOT] {stage_name}: SIGNATURE VALID → booting")
            return True
        else:
            print(f"[SECUREBOOT] {stage_name}: SIGNATURE INVALID → blocked")
            return False

    def get_measurements(self):
        print("\n[TPM] Platform Configuration Registers (PCR):")
        for i, (stage, h) in enumerate(self.measurements):
            print(f"  PCR{i}: {stage}: {h[:16]}...")

sb = SecureBoot()
sb.enroll_key("microsoft_key_2026_hash")
sb.enroll_key("linux_foundation_key_hash")

sb.verify_and_boot("UEFI Firmware", "uefi_code_v2.1", "microsoft_key_2026_hash")
sb.verify_and_boot("Bootloader", "grub_code_v2.06", "linux_foundation_key_hash")
sb.verify_and_boot("Kernel", "linux_kernel_v6.5", "linux_foundation_key_hash")
sb.verify_and_boot("Evil Bootkit", "malicious_code_v1", "unknown_hacker_key")
sb.get_measurements()

Expected output:

[SECUREBOOT] Enrolled key: microsoft_key_2...
[SECUREBOOT] Enrolled key: linux_foundati...
[SECUREBOOT] Verifying UEFI Firmware...
[SECUREBOOT] UEFI Firmware: SIGNATURE VALID → booting
[SECUREBOOT] Verifying Bootloader...
[SECUREBOOT] Bootloader: SIGNATURE VALID → booting
[SECUREBOOT] Verifying Kernel...
[SECUREBOOT] Kernel: SIGNATURE VALID → booting
[SECUREBOOT] Verifying Evil Bootkit...
[SECUREBOOT] Evil Bootkit: SIGNATURE INVALID → blocked

Common Mistakes

1. Running as Root / Administrator

Every process running with full privileges is a security disaster waiting to happen. Use the principle of Least Privilege — run services as unprivileged users.

2. Disabling SELinux / AppArmor

"Turns out it was SELinux" is a common refrain. Instead of disabling MAC, learn to configure it properly or set to permissive mode temporarily.

3. No ASLR on Legacy Systems

32-bit systems have limited ASLR entropy. Use PIC/PIE executables and enable full ASLR. Without ASLR, return-oriented programming (ROP) attacks are trivial.

4. Ignoring Spectre/Meltdown Mitigations

Side-channel attacks exploit speculative execution. Keep kernel and microcode updated. Use retpolines and KPTI on affected systems.

5. Weak Password Policies

Without account lockout and complexity requirements, brute-force attacks succeed. Enforce minimum password length, complexity, and MFA.

Practice Questions

1. What is the difference between DAC and MAC? DAC allows resource owners to control access permissions. MAC enforces system-wide policies that users cannot override. SELinux is MAC; Unix permissions are DAC.

2. How does ASLR prevent buffer overflow exploits? ASLR randomizes the base addresses of libraries, stack, and heap. Attackers cannot predict where shellcode or ROP gadgets are located, making exploitation unreliable.

3. What is measured boot and how does TPM support it? Measured boot records cryptographic hashes of each boot stage in TPM PCRs. At attestation, a remote verifier checks the PCR values against known-good measurements to verify boot integrity.

4. What is the principle of Least Privilege? Every process and user should have only the minimum permissions needed to perform its function. If a vulnerability is exploited, the damage is limited.

5. Challenge: Configure SELinux for a web server that needs to: serve static files from /var/www, write logs to /var/log/httpd, connect to a database on port 3306, and execute CGI scripts from /var/www/cgi-bin. Write the SELinux policy module.

Mini Project: OS Security Hardening Scanner

class SecurityScanner:
    def __init__(self):
        self.checks = []

    def add_check(self, name, check_fn):
        self.checks.append((name, check_fn))

    def run(self):
        print("=== OS Security Scan ===")
        passed = 0
        for name, fn in self.checks:
            result = fn()
            status = "PASS" if result else "FAIL"
            if not result:
                print(f"  [{status}] {name}")
            passed += result
        print(f"\nResult: {passed}/{len(self.checks)} checks passed")

scanner = SecurityScanner()
scanner.add_check("Root login disabled", lambda: True)
scanner.add_check("SELinux enforcing", lambda: True)
scanner.add_check("ASLR enabled", lambda: True)
scanner.add_check("Firewall active", lambda: False)
scanner.add_check("Secure Boot enabled", lambda: True)
scanner.run()

FAQ

What is the difference between SELinux and AppArmor?

SELinux uses type enforcement with labels on every object (files, processes, sockets), providing fine-grained MAC. AppArmor uses path-based profiles attached to executables. SELinux is more powerful but complex; AppArmor is easier to configure.

What is kernel page table isolation (KPTI)?

KPTI separates kernel and user page tables to prevent Meltdown-style side-channel attacks. User-space page tables contain only minimal kernel information. Switching between user and kernel mode is slower but prevents leaking kernel memory.

What is a rootkit and how do OS protections defend against it?

A rootkit is malware that gains kernel-level access, hiding processes, files, and network connections. Defenses include Secure Boot (prevents unsigned kernel modules), kernel module signing, SELinux (limits what kernel code can do), and integrity monitoring (AIDE, Tripwire).

Device Drivers
Distributed Operating Systems
Memory Virtualization

What's Next

You now understand OS security! Next, explore Distributed Operating Systems for concepts like distributed file systems and consensus, and review Memory Virtualization for memory protection details.

  • Practice daily — Run sestatus, getenforce, and aa-status on your system
  • Build a project — Create a minimal SELinux policy for a custom application
  • Explore related topics — Check out Linux kernel hardening with grsecurity and KSPP

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.

← Previous Device Drivers — Kernel Module Programming Guide Next → Distributed Operating Systems — Architecture & Design Guide

Built by the developers of DodaTech

Doda Browser, DodaZIP & Durga Antivirus Pro

Home Browse Operating Systems