Malware Forensic Field Guides: Tool Box 

Chapter 1     Malware Incident Response

(as excerpted in Linux Malware Incident Response: A Practitioner's Guide to Forensic Collection and Examination of Volatile Data

In Chapter 1 (excerpted in the Linux Malware Incident Response: A Practitioner's Guide to Forensic Collection and Examination of Volatile Data, hereinafter "Practitioner's Guide") we examined the incident response process step-by-step, using certain tools to acquire different aspects of stateful data from subject system. There are a number of tool suites specifically designed to collect digital evidence in an automated fashion from Windows systems during incident response, and generate supporting documentation of the preservation process.

        Some of these local incident response tool suites execute commands on the compromised computer, and rely on system libraries on the compromised system.

      • Other programs, commonly known as “remote forensics tools,” address some of the limitations of local incident response suites and use a servlet that enables remote evidence gathering while trying to rely on the compromised operating system as little as possible (with varying degrees of  success).

      Using remote forensic tools, digital investigators can access many machines from a central console, making more effective use of your expertise than spending time running around to touch each machine physically.

      Furthermore, using a remote forensics tool is more subtle than running various commands on the system and is less likely to alert the subject of investigation. 

      These tool options, including the strengths and weakness of these tools, are covered in this section.

 Name:  LINReS v1.1 - Linux Incident Response Script
 Page Reference: 7
 Author/Distributor:  Nii Consulting
 Available From:  http://www.niiconsulting.com/innovation/linres.html
 Description:  LINReS is a live response tool that uses four different scripts to invoke over 80 different trusted binaries to collect volatile and non-volatile data from a subject system.  The initiating script, ir.sh, is the main script that calls the three "sub-scripts" in a pre-defined order. The first sub-script,  main.sh, collects emphemeral data such as running processes, open network connections, last logins, bad logins, among other information . The tertiary script,  metadata.sh, collects metadata information from all the files on the system. The final script, hash.sh, gathers MD5 hashes from each file on the system.  The data collected by the scripts is transferred remotely over the network to a forensic workstation using netcat, which is automatically invoked during the execution of the scripts.  LINRes was originally designed for live data collection from older generation Red Hat systems, thus, the digital investigator may need to adjust the scripts to ensure effective and forensically sound collection efforts from target systems.






  Name:  Helix (Linux Incident Response Script [linux-ir.sh] and Static Binaries)
  Page Reference: 7
  Author/Distributor:  E-Fense
  Available From:  https://www.e-fense.com/store/index.php?_a=viewProd&productId=11
 Description:  Older (non-proprietary) versions of the Helix Incident Response CD-ROM include an automated live response script (linux-ir.sh) for gathering volatile data from a compromised system. linux-ir.sh sequentially invokes over 120 statically compiled binaries (that do not reference libraries on the subject system).  The script has several shortcomings, including gathering limited information about running processes and taking full directory listings of the entire system.





 Name:  Linux Live Response Toolkit
 Page Reference: 7
 Author/Distributor: Enno Ewers and Sebastian Krause
 Available From: http://computer-forensik.org/tools/ix/; and http://ewers.net/llr/
 Description:  The Linux Live Response (llr) Toolkit is a robust script that invokes over 80 trusted static binaries to collect volatile and non-volatile data from subject systems ( kernel versions 2.4 and 2.6).  Unlike other live response tool suites, llr collects physical (/dev/mem and dev/kmem) and process memory dumps from the subject system in an automated fashion.  As the llr was developed in Germany, much of the supporting documentation and instructions is in German, which may require the digital investigator to conduct some additional steps (such as translation through an Internet based translation service like Google Translate) and configuration to ensure effective usage.






 Remote Collection Tools


Recall that in some instances, to reduces system interaction, it is preferable and conducive to deploy live response tools from your trusted toolkit locally on a subject system but collect the acquired data remotely. This process requires establishing a network connection, typically with a netcat or cryptcat listener, and transferring the acquired system data over the network to a collection server.  Remember, that although this method reduces system interaction, it relies on the ability of being able to traverse the subject network through the ports established by the netcat listener.

Name:  Netcat
Page Reference: 3
Author/Distributor: Hobbit
Available From: http://netcat.sourceforge.net
Description:  Commonly referred to as the "Swiss Army Knife" of tools, netcat is a versatile networking utility which reads and writes data across network connections, using the TCP/IP protocol.  Netcat is commonly used by digital investigators during live response as a network based transfer solution.
Helpful Switches:

Switch

Function

-l

Listen mode, for inbound connections

-p

local port number

-h

help menu







Name:  Cryptcat
Page Reference: 3
Author/Distributor: L0pht
Available From: http://cryptcat.sourceforge.net/
Description:  Netcat enhanced with twofish encryption
 Helpful Switches:

Switch

Function

-l

Listen mode, for inbound connections

-p

local port number

-h

help menu







 Name:  F-Response TACTICAL
 Page Reference: 58
 Author/Distributor: 
 Available From:  http://www.f-response.com/
 Description: A streamlined solution for onsite live response, F-Response Tactical uses a unique dual-dongle/storage device solution to quickly and seamlessly allow the digital investigator to conduct remote forensic acquisition with limited knowledge of the subject network typology.  The dual-dongles—one for the subject system, one for the examiner system (shown below)—work as a pair to connect the remote subject system to the digital investigator’s examination system; TACTICAL runs directly from the dongles and no installation is required on the subject system.  Like other versions of F-Response, in addition to Linux systems, TACTICAL can acquire both Windows and Macintosh OS X subject systems.
 

 Shown in the story-board figure below, the TACTICAL “subject” dongle, when plugged into the subject system, houses the “TACTICAL Subject” directory which contains the executables for Windows, Linux and Macintosh OS X systems. 
 
 Once invoked from the command line, the Linux tactical subject executable initiates an iSCSI session, as shown in the Figure, below:
 
 root@ubuntu:/media/SUBJECT/TACTICAL Subject# ./f-response-tacsub-lin
F-Response TACTICAL Subject (Linux Edition) Version 4.00.02
F-Response Disk: /dev/sda (41943040 sectors, 512 sector size)
20480 MB write blocked storage on F-Response Disk:sda
F-Response Disk: /dev/sdb (3947520 sectors, 512 sector size)
1927 MB write blocked storage on F-Response Disk:sdb
 

On the examiner system (the system in which the digital investigator conducts his/her collection of data), the companion “Examiner” dongle is connected.   Depicted in the story-board figure below, the TACTICAL “Examiner” dongle houses the “TACTICAL Examiner” directory which contains the Linux executables to use Examiner from the command line (f-response-tacex-lin) or the GUI (f-response-tacex-lin-gui).
 
 
 Once invoked, the digial investigator has the option of connecting to the subject system manually by providing the details of the subject system (in the GUI, as shown below), or using the “auto-connection” feature, which automatically tries to identify and acquire the subject system.
 
Once acquired, TACTICAL Examiner provides the details regarding the acquired subject system.  Similar to other versions of F-Response, once connected to the subject system, the digitial investigator can use tools of his/her choice to collect data from the system. 
 






 Volatile Data Collection and Analysis Tools




 Physical Memory Acquisition

The Practioner's Guide emphasizes the importance of first acquiring a full memory dump from the subject system prior to gathering data using the various tools in your live response toolkit. This is important, particularly due to the fact that running incident response on the subject system will alter the contents of memory.   To get the most digital evidence out of physical memory, it is advisable to perform a full memory capture prior to running any other incident response processes.  There are a variety of tools to accomplish this task, described below.



Name:  LiME
 Page Reference: 19
Author/Distributor: Joe Sylve
Available From:  http://code.google.com/p/lime-forensics/
Description:   The Linux Memory Extractor (LiME) is a loadable kernel module developed to acquire the contents of physical memory from Linux and Android systems.  This utility supports acquisition of memory to a local file system (e.g. removable USB device or SD Card) or over the network.

        Usage:    ./insmod   /sdcard/lime.ko   "path=/sdcard/ram.padded format=padded"
Helpful Switches:

Switch

Function

path= Location to save acquired data
format= Padded, lime or raw
dio= 1 to enable Direct IO attempt (default), 0 to disable








Name:  SecondLook Physical Memory Acquisition Script                               (secondlook-memdump.sh)
Page Reference:  18
Author/Distributor:  Andrew Tappert/Raytheon Pikewerks
Available From:  http://pikewerks.com/sl
Description:  The SecondLook Physical Memory Acquisition Script (secondlook-memdump.sh) enables the digital investigator to collect physical memory from a Red Hat or CentOS Linux system using the crash driver (/dev/crash), or from other systems using a user-specified memory access device (such as /dev/mem) or the proprietary Pikewerks' physical memory access driver (PMAD) (creating an accessible pseudo-device /dev/pmad).  Physical memory collected with secondlook-memdump.sh can then be examined in the SecondLook Memory Forensics tool.

                         Usage: ./secondlook-memdump.sh dumpfile [memdevice]






Name:  fmem
Page Reference:  17
Author/Distributor:  Ivor Kollar
Available From:  http://hysteria.sk/~niekt0/fmem/

Description: fmem is a custom kernel module that comes with the tool Foriana  (FOrensic Ram Image ANAlyzer), enabling the digital investigator to acquire physical memory. In particular the  fmem kernel module (fmem.ko) creates device a pseudo-device,/dev/fmem, similar to /dev/mem but without the  acquisition limitations. This psuedo-device (physical memory) can be copied using dd or other tools.  The tool has a shell script (run.sh) to execute the acquisition process.






Name: memdump
Page Reference:  16
Author/Distributor:  Dan Farmer and Wietse Venema
Available From:  http://sourceforge.net/projects/mdd/
Description:  The memdump command in the Coroner's Toolkit, a suite of tools for forensic acquisition and analysis of Linux/UNIX systems, can be used to save the contents of physical memory into a file.





Name: dc3dd
Page Reference:  8
Author/Distributor:  Defense Cyber Crime Institute
Available From:  http://sourceforge.net/projects/dc3dd

Description: A forensically enhanced add-on to the de facto dd utility on Linux systems used to copy and convert files.  The versatile functionality of the tool provides the digital investigator with a ability to acquire physical memory, hard drives, and other media alike.

Example usage for physical memory acquisition on Linux systems without restrictions on /dev/mem:

                              dc3dd if=/dev/mem/ of=/media/IR/memdump.img


Helpful Switches:

Switch

Function

 ssz=BYTES  Use BYTES bytes for the sector size
 cnt=SECTORS  Copy only SECTORS input sectors

if=FILE

Read from FILE instead of stdin

of=FILE

Write to FILE instead of stdout

hash=md5

Hash algorithm to verify input/output: md5, sha1, sha256, sha384 or sha512

hlog=

Send md5 hash output to FILE instead of stderr

log=

File to log all I/O statistics, diagnostics and total hashes






 Collecting Subject System Details

System details are a fundamental aspect of understanding a malicious code crime scene.  In particular, system details will inevitably be crucial in establishing an investigative timeline, and identifying the subject system in logs and other forensic artifacts.  In addition to the tools mentioned in this book, others tools consider include:




Name: Uname
Page Reference:   23
Author/Distributor: David MacKenzie
Available From: GNU coreutils (native to Linux Systems); http://www.gnu.org/software/coreutils
Description: Displays system information, including operating system, kernel version, kernel details, network hostname, and hardware machine name, among other information.

Helpful Switches:

Switch

Function

 -a  Displays  all  information
 -s  Displays kernel name
 -n Displays network node name 
 -r Displays kernel release 
-m Displays machine name
-o Displays operating system
-i Displays hardware platform
-p Displays processor





Name:   linuxinfo
Page Reference:  23
Author/Distributor:  Alex Buell
Available From:  http://www.munted.org.uk/programming/linuxinfo-1.1.8.tar.gz
Description:  Displays system details; no command switches required:

 malwarelab@ubuntu:~$linuxinfo
Linux ubuntu 2.6.35-22-generic #33-Ubuntu SMP Mon Mar 19 20:34:50 UTC 2012
One Intel Unknown 1596MHz processor, 3192.30 total bogomips, 1015M RAM
System library 2.12.1






 Name:   id
 Page Reference:  21
 Author/Distributor:  Arnold Robbins and David MacKenzie
 Available From:  GNU coreutils (native to Linux Systems); http://www.gnu.org/software/coreutils
 Description:  Displays user and group information for a target user, or for the current user if a target user is not queried.
 Helpful Switches:

Switch

Function

 -n Print a name instead of a number, for -ugG
-u
Print only the effective user ID
-g
Print only the effective group ID
-G
Print all group IDs





 
 Name:   logname
 Page Reference:  21
 Author/Distributor:  FIXME: unknown
 Available From:   GNU coreutils (native to Linux Systems); http://www.gnu.org/software/coreutils
 Description:  Displays name of the current user; no switches needed.





 Name:   printenv
 Page Reference:  23
 Author/Distributor:   David MacKenzie and Richard Mlynarik
 Available From:  GNU coreutils (native to Linux Systems); http://www.gnu.org/software/coreutils
 Description:   Displays environment variables. No switches required, but specific variables can be queried to isolate and granulate output (e.g. printenv PATH).





 Name:   sa
 Page Reference:  24
 Author/Distributor:  Noel Cragg
 Available From: http://www.gnu.org/software/acct/
 Description:  As a part of the GNU Accounting Utilities (developed to provide login and process accounting utilities for GNU/Linux and other systems), the sa utility collects and displays information from the system acct (process accounting file). When process  accounting is enabled on a subject system, the kernel writes a record to the acct file as each process on the system terminates.
 Helpful Switches:

Switch

Function

-u For each command in the accounting file, print the userid and command name.
-m Shows the number of processes and number of CPU minutes on a per-user basis.
-t
For each entry, print the ratio of real time to the sum of
system and user times.





 Name:   sar
 Page Reference:  25
 Author/Distributor:  Sebastien Godard
 Available From:  Included in the Systat Utilities for Linux, http://sebastien.godard.pagesperso-orange.fr/index.html
 Description:  Collects and displays a broad scope of system activity information.





 Name:  ifconfig
 Page Reference: 21
 Author/Distributor:  Fred N. van Kempen, Alan Cox, Phil Blundell, Andi Kleen, and Bernd Eckenfels
 Available From: Native to Linux systems.
 Description: Displays network interface details and configuration options.
 Helpful Switches:

Switch

Function

-a Display  all  interfaces  which are currently available on the subject system,
even if the interface is down
-s Display a short list of network interfaces (like netstat -i)
k interface details and configuration options.





 Name:  ifdata
 Page Reference:  21
 Author/Distributor:
 Available From:  Native to most Linux distributions.
 Description:  Displays network interface details.
  Helpful Switches:

Switch

Function

 -p Displays complete interface configuration 
-pa
Displays the IPv4 address of the interface
-ph
Displays the hardware address of the interface
-pN
Displays the network address of the interface




 Identifying Users Logged into the System

Remember that identifying users logged into the subject system servers a number of investigative purposes: 1) help discover any potential intruders logged into the compromised system; 2) identify additional compromised systems;  3) provide insight into a malicious insider malware incident;  and provides additional investigative context by being correlated with other artifacts.  Some other tools to consider for this task include:




Name:  W
Page Reference:  26
Author/Distributor: Charles Blake, (re-written based on the version by Larry Greenfield and Michael K. Johnson)
Available From: Native to most Linux distributions

Description: Shows logged on users and associated activity.

Helpful Switches:

Switch

Function

-u Ignores the username and identifies the current  process  and cpu  times.
-s “Short” or abbreviated listing that does not include login
time,  JCPU  or  PCPU times.
user Show information about the specified user only





Name:   who
Page Reference:  26
Author/Distributor:  Joseph Arceneaux, David MacKenzie, and Michael Stone
Available From:  GNU coreutils (native to Linux Systems); http://www.gnu.org/software/coreutils

Description: Displays information about users who are currently logged in.

 Helpful Switches:

Switch


Function

 -a    All
 -b    Time of last system boot
 -d    Display dead system processes
 --ips    Displays IP addresses  instead  of  hostnames
 --lookup    Attempts to canonicalize hostnames via DNS
-l Display system login processes
-q Show all login names and number of users logged on
-r Shows current runlevel





Name: Finger
Page Reference:  26
Author/Distributor: David Zimmerman/Les Earnest
Available From: Native to most Linux distributions.
Description: User information lookup program.

Helpful Switches:

Switch

Function

-s Finger displays the user's login name, real name,
terminal name and write status (as a ``*'' after the
terminal name if write permission is denied), idle time,
 login time, office location and office phone number.
Login time is displayed as month, day, hours and
minutes, unless more than six months ago, in which
case the year is displayed rather than the hours and minutes.
Unknown devices as well as nonexistent idle and login
times are displayed as single asterisks.
-l Produces a multi-line format displaying all of the
information described for the -s option as well as the user's
home directory, home phone number, login shell, mail status,
and the contents of the files “.plan”, “.project”, “.pgpkey”
and “.forward” from the user's home directory.





 Name: last
 Page Reference:  64
 Author/Distributor: Miquel van Smoorenburg
 Available From:  Native to most Linux distributions
 Description:  Displays a listing of last logged in users by querying the /var/log/wtmp
since that file was created.

 Helpful Switches:

Switch

Function

 -f  Points the tool to use a specific file instead of /var/log/wtmp
-t Displays  the  state of logins as of the specified time.  
This is useful to identify who was logged in at a particular  time.
-d For remote logins, Linux stores the host name of the remote host
and the associated IP address.  This option  translates the IP address
back into a hostname.
-i This  option is like -d in that it displays the IP address of the
remote host in  standard octet format.





 Name: users
 Page Reference:  26
 Author/Distributor:  Joseph Arceneaux and David MacKenzie
 Available From:  GNU coreutils (native to Linux Systems); http://www.gnu.org/software/coreutils
 Description:  Displays the user names of users currently logged into the subject system.  No command switches required.




 Network Connections and Activity

Malware network connectivity is a critical factor to identify and document; connectivity from a subject system may be to communicate with an attacker’s command and control structure, download additional malicious files, or to exfiltrate data from the system, among other things. In addition to netstat and lsof, other to consider are fuser, route, socklist and ss.


Name:  fuser
Page Reference: 40
Author/Distributor:  Werner Almesberger and Craig Small
Available From:    Native to most Linux distributions
Description:  Displays processes using files or sockets

 Helpful Switches:

Switch

Function

-u “user”; Appends the user name of the process owner to each PID. 

For example a query for the PID associated with the suspicious upd port 52475, use:

fuser –u 52475/udp
-n “Name space” variable. The name spaces file (a target file name, which is the  default),
 udp (local UDP ports), and tcp (local TCP ports) are supported.

For example, to query for the PID and user associated with
suspicious TCP port 3329, use:

fuser -nuv tcp 3329
-v Verbose mode






 Name:  route
 Page Reference:  28
 Author/Distributor:  Originally written by Fred N. van Kempen, and then modified by Johannes  Stille  and Linus  Torvalds. Currently  maintained  by Phil Blundell an Bernd Eckenfels.
 Available From:  Native to most Linux distributions.
 Description: Shows the IP routing table on the subject system.





 Name:  socklist
 Page Reference:  28
 Author/Distributor:  Larry Doolittle
 Available From:  Native to most Linux distributions.
 Description:  Displays a list of open sockets, including types, port, inode, uid, pid and associated program.





 Name:  ss (socket statistics)
 Page Reference:  28
 Author/Distributor:  Alexey Kuznetosv
 Available From:  Native to most Linux distributions
 Description: Versatile utility to examine sockets
 Helpful Switches:

Switch

Function

 -a  Displays  all  sockets
 -l  Displays listening sockets
 -e Displays detailed socket information 
 -m Displays socket memory usage
-p Displays process using socket
-i Displays internal TCP information
-t Displays only TCP sockets
-u Displays only UDP sockets






 Process Analysis

As many malware specimens (such as worms, viruses, bots, key loggers, and Trojans) often manifest on the subject system as a process, collecting information relating to processes running on a subject system is essential in malicious code live response forensics.  Process analysis should be approached holistically—examine all relevant aspects of a suspicious process, as outlined in this Practitioner's Guide.  Below are additional tools to consider for your live response toolkit.



Name: pslist
Page Reference:  31
Author/Distributor: Peter Penchev
Available From: https://launchpad.net/ubuntu/lucid/i386/pslist/1.3-1
Description:  Gathers target process details, including process ID (PID), command name, and the PIDS of all child processes.  Target processes may be specificed by name or PID.





Name:  pstree
 Page Reference:  35
 Author/Distributor:  Werner Almesberger and Craig Small
 Available From:   Native to most Linux distributions.
 Description:  Displays a textual tree hierarchy of running processes (parent/ancestor and child processes).

 Helpful Switches:

Switch

Function

 -a  Show command line arguments
 -A  Use ASCII characters to draw tree
 -h  Highlights the current process and its ancestors
 -H  Highlights the specified process
 -l  Displays long lines
 -n  Sorts processes with the same ancestor by PID instead of by name.
 -p  Displays PIDs
-u Displays uid transitions
 




Name:  vmstat
Page Reference:  31
Author/Distributor:  Henry Ware, Fabian Frédérick
Available From:     Native to most Linux distributions.
Description:  Reports virtual memory statistics (processes, memory, etc.).





Name:  dstat
Page Reference:  31
Author/Distributor:  Dag Wieers
Available From:  http://dag.wieers.com/home-made/dstat/
Description: Reports robust system statistics; Replacement for vmstat.





 Name:  iostat
 Page Reference:  31
 Author/Distributor: Sebastien Godard
 Available From:  Native to most Linux distributions.
 Description: Monitor input/output devices.





 
 Name:  procinfo
 Page Reference:  31
 Author/Distributor:  Adam Schrotenboer/ Sander Van Malssen
 Available From:  ibiblio.org/pub/Linux/system/status/ps/procinfo.lsm;
for Ubuntu http://manpages.ubuntu.com/manpages/jaunty/man8/procinfo.8.html
 Description:   Displays system status details as collected from /proc directory





 Name:  pgrep
 Page Reference:  31
 Author/Distributor:   Kjetil Torgrim Homme and Albert Cahalan
 Available From:  Native to most Linux distributions.
 Description:   Enables the digital investigator to query a target process by process ID (PID), process name, and/or user name.
 Helpful Switches:
 

Switch

Function

-l List the process name and the PID
-U Only match processes whose real user ID is listed





 Name:  pmap
 Page Reference:  36
 Author/Distributor:  Albert  Cahalan
 Available From:  Native to most Linus distributions.
 Description:  Provides a process memory map.
 Helpful Switches: 

Switch

Function

-x Displays extended format
-d Displays device format




 Loaded Moldules





Name:  lsmod
Page Reference:  47
Author/Distributor:  Rusty Russell
Available From:  Native to most Linux distributions.
Description:  Displays status of modules in the subject system’s Kernel (as reported from the contents of /proc/modules).





Name:  modinfo
Page Reference:  47
Author/Distributor:  Rusty Russell
Available From:  Native to most Linux distributions.
Description:  Displays information about a kernel module.
Helpful Switches:

Switch

Function

 -F Displays only the specified field value per line.
Field values include author, description, license, parm,
and file name.  These fields can be designated by respective
shortcut switches as described in this table.
 -a  Author
 -d  Description
 -l  License
 -p  Parm
-n File name
 
 




Name:  modprobe
Page Reference:  47
Author/Distributor:  Rusty Russell
Available From:   Native to most Linux distributions.
Description:  Utility to explore (and alter) module properties, dependencies and configuration.




 Opened Files
Opened files on a subject system may provide clues about the nature and purpose of the malware involved in an incident, as well as correlative artifacts for your investigation.  In the Practitioner's Guide we examined the tool lsof; another tool to consider is fuser.



Name:  fuser
Page Reference:  44
Author/Distributor:  Werner Almesberger; Craig Small
Available From:  Native to most Linux distributions.
Description:  .Displays processes using files or sockets.

Helpful Switches:

Switch

Function

-u “user”; Appends the user name of the process owner to each PID.  

For example a query for the user and PID associated  with the
suspicious file libnss_dns-2.12.1.so, use:

#fuser -u /lib/libnss_dns-2.12.1.so

/lib/libnss_dns-2.12.1.so:  5365m(victim)
-n “Name space” variable.
The name spaces file (a target file name, which is the  default),  
udp (local UDP ports), and tcp (local TCP ports) are supported.
-v -v    Verbose mode




Command History

Name:  lastcomm
Page Reference:  48
Author/Distributor:   Noel  Cragg
Available From:   The  GNU  accounting  utilities, http://www.gnu.org/software/acct/.
Description:  Displays information about previously executed commands on the subject system.
Helpful Switches:
 

Switch

Function

 
--strict-match
Displays only entries that match all of the
arguments on the command line.
 --user Displays records for the user name
--command Displays records for the command name
--tty Displays records for the tty name
--pid Displays records for the PID