Malware Analysis Basics Guide -- Static Analysis, Dynamic Analysis & Reverse Engineering

In this tutorial, you'll learn about Malware Analysis Basics Guide. We cover key concepts, practical examples, and best practices.

Malware analysis is the process of examining malicious software to understand its behavior, purpose, and impact -- using static techniques like disassembly and dynamic techniques like sandbox execution to extract indicators of compromise.

What You'll Learn

You will learn to perform static analysis of PE files and scripts, execute malware in a sandbox safely, use disassemblers like Ghidra and IDA Free, extract IOCs, and write analysis reports for incident response.

Why It Matters

New malware samples appear at a rate of 560,000 per day according to AV-TEST. Organizations that perform in-house malware analysis reduce mean response time from 11 days to 4 hours and can detect zero-day threats before signature-based tools.

Real-World Use

A SOC analyst receives a suspicious email attachment. Static analysis reveals an obfuscated PowerShell loader. Sandbox execution shows it beacons to a C2 domain. The domain is blocked within 90 minutes, preventing ransomware deployment across 2000 endpoints.

Malware Analysis Approaches Compared

Static Analysis: Strings and PE Headers

# Extract printable strings from a suspicious binary
strings --all --print-file-name suspicious-sample.exe > strings-output.txt

# Filter for relevant indicators
grep -iE "(http|https|www\.|\.com|\.exe|\.dll|cmd|powershell|encrypt|decrypt)" \
  strings-output.txt | sort -u

# Expected output:
# http://malicious-c2.evil.com/beacon
# C:\Windows\System32\cmd.exe
# /c powershell -EncodedCommand
# AES-256-CBC
# persist.exe

# Analyze PE header structure
pev peinfo suspicious-sample.exe

# PE information
# Entry point: 0x00123456
# Section count: 3
# Sections: .text (executable), .rdata, .data
# Imported libraries: kernel32.dll, advapi32.dll, ws2_32.dll
# Suspicious imports: CreateProcess, VirtualAlloc, InternetOpenUrl

Expected behavior: Strings reveal URLs, file paths, and API calls. PE header analysis shows imported functions that indicate network activity (ws2_32.dll), process injection (VirtualAlloc), and payload execution (CreateProcess).

YARA Rule Creation

YARA allows writing detection signatures based on patterns found during analysis.

rule Suspicious_PowerShell_Loader {
    meta:
        description = "Detects obfuscated PowerShell download cradles"
        author = "DodaTech Malware Analysis Team"
        date = "2026-06-22"
        severity = "high"

    strings:
        $download_cradle1 = "System.Net.WebClient"
        $download_cradle2 = "Invoke-WebRequest"
        $download_cradle3 = "Start-BitsTransfer"
        $obfuscation1 = " -EncodedCommand " nocase
        $obfuscation2 = "FromBase64String" nocase
        $obfuscation3 = "[Convert]::"
        $payload_exec = "Invoke-Expression" nocase
        $payload_exec2 = "IEX" nocase
        $reflection = "Reflection.Assembly" nocase
        $memory_exec = "Load(" nocase

    condition:
        // PowerShell download cradle with obfuscation
        (1 of ($download_cradle*)) and
        (1 of ($obfuscation*)) and
        (1 of ($payload_exec*))
}

rule CobaltStrike_Beacon_Indicators {
    meta:
        description = "Detects Cobalt Strike beacon artifacts"
        author = "DodaTech Malware Analysis Team"

    strings:
        $mutex = "Global\\MSF"{2}"A"{8}"A" wide
        $named_pipe = "\\\\.\\pipe\\msagent_" wide
        $config = "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
        $rsa_key = "-----BEGIN RSA PRIVATE KEY-----" wide

    condition:
        // Cobalt Strike specific patterns
        uint32(0) == 0x00905a4d and  // MZ header
        ($mutex or $named_pipe)
}

Expected behavior: YARA rules detect specific malware families by matching binary patterns. The first rule identifies obfuscated PowerShell download cradles. The second detects Cobalt Strike beacons. Rules can be scanned with yara -s rules.yar suspicious-sample.exe.

Sandbox Execution Setup

# docker-compose.yml for Cuckoo sandbox (modern fork)
version: "3.8"
services:
  cuckoo:
    image: cuckoo-sandbox/cuckoo:latest
    ports:
      - "8090:8090"
    volumes:
      - ./storage:/opt/cuckoo/storage
      - ./analyses:/opt/cuckoo/analyses
    environment:
      - CUCKOO_CUCKOO_MACHINERY=virtualbox
      - CUCKOO_RESULTSERVER_IP=192.168.56.1
      - CUCKOO_RESULTSERVER_PORT=2042
    devices:
      - "/dev/vboxdrv:/dev/vboxdrv"
    privileged: true
    networks:
      - isolated_network

networks:
  isolated_network:
    driver: bridge
    ipam:
      config:
        - subnet: 192.168.56.0/24

Expected behavior: Cuckoo creates an isolated analysis environment. The malware sample is submitted via the web interface or API. Cuckoo executes the sample in a virtual machine and records all system calls, file operations, registry changes, and network connections.

Memory Forensics with Volatility

# Dump process memory from an infected system
volatility -f memory.dump windows.psscan

# Expected output:
# PID  PPID  Name
# 1234  888   explorer.exe
# 5678  1234  suspicious.exe  <-- Unrecognized process
# 9012  888   svchost.exe

# Extract injected code from suspicious process
volatility -f memory.dump windows.malfind.Malfind \
  --pid 5678

# Expected output:
# Process: suspicious.exe Pid: 5678
# InjectedPage: 0x1a00000
# VAD tags: VadS Protection: PAGE_EXECUTE_READWRITE
# Flags: CommitCharge: 1, PrivateMemory: 1
# Hex dump shows shellcode-like patterns

# Extract network artifacts
volatility -f memory.dump windows.netscan.NetScan

# Expected output:
# Offset  Proto  LocalAddr      ForeignAddr      State
# 0xabc   TCP    192.168.1.5:4923  203.0.113.5:443  ESTABLISHED
# 0xdef   TCP    192.168.1.5:4924  203.0.113.5:443  ESTABLISHED

Expected behavior: Memory analysis reveals hidden processes, injected code regions with PAGE_EXECUTE_READWRITE protection, and active network connections to C2 infrastructure. These artifacts are invisible to traditional file-based scans.

Malware Analysis Workflow

graph TD
    A[Suspicious Sample Received] --> B[Isolate in Sandbox]
    B --> C[Static Analysis]
    C --> D[Strings, hashes, metadata]
    C --> E[PE structure, imports]
    C --> F[YARA signature match]
    B --> G[Dynamic Analysis]
    G --> H[File system changes]
    G --> I[Registry modifications]
    G --> J[Network connections]
    G --> K[Process injection]
    F --> L[IOC Extraction]
    H --> L
    I --> L
    J --> L
    K --> L
    L --> M[Blocklist IOCs]
    L --> N[Write Detection Rules]
    L --> O[Incident Response Report]
    style A fill:#e74c3c,stroke:#fff,color:#fff
    style O fill:#27ae60,stroke:#fff,color:#fff
    style L fill:#4a90d9,stroke:#fff,color:#fff

Automated IOC Extraction

#!/usr/bin/env python3
"""Extract IOCs from malware analysis report."""
import json
import re
import hashlib
from datetime import datetime

def extract_iocs(sample_path, analysis_json):
    with open(analysis_json) as f:
        analysis = json.load(f)

    iocs = {
        "file": {},
        "network": [],
        "registry": [],
        "mutex": [],
        "timestamp": datetime.utcnow().isoformat()
    }

    # File hashes
    with open(sample_path, "rb") as f:
        data = f.read()
    iocs["file"]["md5"] = hashlib.md5(data).hexdigest()
    iocs["file"]["sha1"] = hashlib.sha1(data).hexdigest()
    iocs["file"]["sha256"] = hashlib.sha256(data).hexdigest()
    print(f"SHA256: {iocs['file']['sha256']}")
    # Expected: SHA256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

    # Network IOCs
    for conn in analysis.get("network", {}).get("tcp", []):
        ioc = {
            "dst_ip": conn.get("dst"),
            "dst_port": conn.get("dport"),
            "state": conn.get("state")
        }
        if ioc["dst_ip"] and ioc["dst_port"]:
            iocs["network"].append(ioc)
            print(f"C2: {ioc['dst_ip']}:{ioc['dst_port']}")

    # Registry IOCs
    for reg in analysis.get("behavior", {}).get("registry", []):
        if reg.get("status") == "success":
            iocs["registry"].append(reg.get("key"))
            print(f"Registry: {reg.get('key')}")

    return iocs

# Usage
iocs = extract_iocs("suspicious-sample.exe", "analysis-report.json")

# Export to MISP format
with open("iocs-misp.json", "w") as f:
    json.dump({"Event": {"info": "Malware IOCs", "Attribute": [
        {"type": "sha256", "value": iocs["file"]["sha256"]},
        {"type": "ip-dst", "value": ioc["dst_ip"]}
        for ioc in iocs["network"]
    ]}}, f, indent=2)

Expected behavior: The script extracts file hashes, C2 IP addresses and ports, registry persistence points, and mutex names from the sandbox analysis report. IOCs are exported in MISP format for sharing with threat intelligence platforms.

Common Errors

1. Analyzing malware on an internet-connected system -- Malware can spread to other systems, exfiltrate data, or notify attackers. Always use an air-gapped analysis environment with network simulation (INetSim or FakeNet) rather than real internet access.

2. Ignoring anti-analysis techniques -- Malware detects sandboxes, debuggers, and virtual machines using techniques like IsDebuggerPresent, timing checks, and hardware fingerprinting. Use custom kernel drivers or bare-metal analysis for advanced samples.

3. Executing malware without snapshots -- A destroyed analysis environment requires full reinstallation. Take VM snapshots before each execution and revert immediately after analysis completes.

4. Over-relying on automated sandbox reports -- Sandboxes execute malware briefly and may miss time-based triggers or user interaction requirements. Always supplement automated reports with manual static and dynamic analysis.

5. Forgetting to extract IOCs after analysis -- The primary output of malware analysis should be actionable IOCs. Without extracting hashes, domains, IPs, and rules, the analysis helps only the current incident and not future detection.

Practice Questions

1. What is the difference between static and dynamic malware analysis? Static analysis examines the binary without executing it -- strings, PE headers, disassembly, and YARA rules. Dynamic analysis executes the sample in a controlled environment to observe runtime behavior. Static analysis is safer but limited; dynamic analysis reveals actual behavior.

2. Why do malware authors obfuscate PowerShell scripts? Obfuscation bypasses signature-based detection by encoding commands in Base64, splitting strings across variables, and using reflection to load assemblies dynamically. Security tools that match plain-text patterns miss obfuscated payloads.

3. What does PAGE_EXECUTE_READWRITE memory protection indicate? It indicates memory that is both writable and executable -- a common sign of code injection. Normal executable code is PAGE_EXECUTE_READ. Malware uses VirtualAlloc with PAGE_EXECUTE_READWRITE to write shellcode and then execute it in the same memory region.

4. How does Volatility detect hidden processes? Volatility compares process listings from multiple sources: the EPROCESS linked list, PID table, session processes, and CSRSS handles. Hidden processes present in some sources but not others are flagged as suspicious.

5. Challenge: Download a known malware sample (Mirai, Emotet, or similar) from a malware repository like MalShare or VirusTotal. Perform static analysis with pestr, yara, and strings. Execute it in a sandbox and document all system calls, network connections, and persistence mechanisms. Write a YARA rule to detect the family.

Mini Project

Build a malware analysis pipeline. Set up a Python-based sandbox orchestrator using libvirt or VirtualBox APIs. Create scripts that accept file submissions, run static analysis (pefile library, YARA), execute samples in isolated VMs for 5 minutes, capture Wireshark packet captures, extract IOCs using Volatility, and generate MISP-formatted threat intelligence reports.

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.