Password brute-force on GPU

It's not a secret that GPU is largely used for bitcoin mining and it's also a prefect solution for password cracking.  The purpose of this article is to show how insecure it is to use simple passwords.

Nowadays Ubuntu uses SHA-512 for password hashing by default:

$ mkpasswd -m help
sha512crypt     SHA-512
sha256crypt     SHA-256
md5crypt        MD5
descrypt        standard 56 bit DES-based crypt(3)

If you open /etc/shadow you will see something like this:

user:$6$8f5pVFu2VWchx77Z$YpfJuoJltko.d0Iu/6C9m2phVSLs.ZYuR6PEt6xoZJcxoOPMTlfgfqiYPdhMOdr3j7whvrfYe3wYO3GLYAeFp0:18418:0:99999:7:::

The first field means algorithm used for hashing:

  • $1 = MD5 hashing algorithm.
  • $2 =Blowfish Algorithm is in use.
  • $2a = blowfish Algorithm
  • $5 =SHA-256 Algorithm
  • $6 =SHA-512 Algorithm

In my case it's SHA-512.

Second field is salt which is used together with password to create a hash for it.

Let's now download and install hashcat tool:

$ apt-get install hashcat

Hashcat uses GPU to accelerate hash generation for password cracking. hashcat provides several rules for password generation and allows to use dictionaries.

Execution on GPU is possible thanks to OpenCL supported by many graphic cards. If you have a look at hashcat sources you will find many .cl files developed exactly to be used on systems where parallel processing is possible. In this experiment we used Radeon GPU:

$ /opt/rocm/opencl/bin/clinfo 
Number of platforms:				 2
  Platform Profile:				 FULL_PROFILE
  Platform Version:				 OpenCL 1.2 pocl 1.4, None+Asserts, LLVM 9.0.1, RELOC, SLEEF, DISTRO, POCL_DEBUG
  Platform Name:				 Portable Computing Language
  Platform Version:				 OpenCL 2.0 AMD-APP (3137.0)
  Platform Name:				 AMD Accelerated Parallel Processing
  Platform Vendor:				 Advanced Micro Devices, Inc.

  Platform Name:				 AMD Accelerated Parallel Processing
Number of devices:				 1
  Device Type:					 CL_DEVICE_TYPE_GPU
  Board name:					 Ellesmere [Radeon RX 470/480/570/570X/580/580X/590]
  Device Topology:				 PCI[ B#1, D#0, F#0 ]
  Max compute units:				 36
  Max work items dimensions:			 3
  Max work group size:				 256
  Max clock frequency:				 1370Mhz
  Address bits:					 64
  Max memory allocation:			 7301444403
  Image support:				 Yes
  Max number of images read arguments:		 128
  Max number of images write arguments:		 8
  Max image 2D width:				 16384
  Max image 2D height:				 16384
  Max image 3D width:				 2048
  Max image 3D height:				 2048
  Max image 3D depth:				 2048
  Max samplers within kernel:			 26591
  Cache type:					 Read/Write
  Cache line size:				 64
  Cache size:					 16384
  Global memory size:				 8589934592
  Constant buffer size:				 7301444403
  Platform ID:					 0x7f61c7a3dcf0
  Name:						 gfx803
  Vendor:					 Advanced Micro Devices, Inc.
  Device OpenCL C version:			 OpenCL C 2.0 
  Driver version:				 3137.0 (HSA1.1,LC)
  Profile:					 FULL_PROFILE
  Version:					 OpenCL 1.2 

Make sure GPU is detected by hashcat:

$ hashcat -I
hashcat (v5.1.0) starting...

OpenCL Info:

Platform ID #1
  Vendor  : The pocl project
  Name    : Portable Computing Language
  Version : OpenCL 1.2 pocl 1.4, None+Asserts, LLVM 9.0.1, RELOC, SLEEF, DISTRO, POCL_DEBUG

  Device ID #1
    Type           : CPU
    Vendor ID      : 1
    Vendor         : AuthenticAMD
    Name           : pthread-AMD Ryzen 3 3200G with Radeon Vega Graphics
    Version        : OpenCL 1.2 pocl HSTR: pthread-x86_64-pc-linux-gnu-znver1
    Processor(s)   : 4
    Clock          : 4000
    Memory         : 4096/13951 MB allocatable
    OpenCL Version : OpenCL C 1.2 pocl
    Driver Version : 1.4

Platform ID #2
  Vendor  : Advanced Micro Devices, Inc.
  Name    : AMD Accelerated Parallel Processing
  Version : OpenCL 2.0 AMD-APP (3137.0)

  Device ID #2
    Type           : GPU
    Vendor ID      : 1
    Vendor         : Advanced Micro Devices, Inc.
    Name           : gfx803
    Version        : OpenCL 1.2 
    Processor(s)   : 36
    Clock          : 1370
    Memory         : 6963/8192 MB allocatable
    OpenCL Version : OpenCL C 2.0 
    Driver Version : 3137.0 (HSA1.1,LC)

Run our test with options:

-m 1800 which means:

   1800 | sha512crypt $6$, SHA512 (Unix)                   | Operating Systems

-a 3 in attack-mode

-O ?l?l?l?l?l?l?d?d?d it's a guess mask which assumes the password has 9 characters where first 6 are low case symbols and last 3 are digits

-d 2 to use our GPU device for hash generation.

$ hashcat -a 3  -m 1800 '$6$8f5pVFu2VWchx77Z$YpfJuoJltko.d0Iu/6C9m2phVSLs.ZYuR6PEt6xoZJcxoOPMTlfgfqiYPdhMOdr3j7whvrfYe3wYO3GLYAeFp0' -O ?l?l?l?l?l?l?d?d?d -d 2
hashcat (v5.1.0) starting...

OpenCL Platform #1: The pocl project
====================================
* Device #1: pthread-AMD Ryzen 3 3200G with Radeon Vega Graphics, skipped.

OpenCL Platform #2: Advanced Micro Devices, Inc.
================================================
* Device #2: gfx803, 6963/8192 MB allocatable, 36MCU

Hashes: 1 digests; 1 unique digests, 1 unique salts
Bitmaps: 16 bits, 65536 entries, 0x0000ffff mask, 262144 bytes, 5/13 rotates

Applicable optimizers:
* Optimized-Kernel
* Zero-Byte
* Single-Hash
* Single-Salt
* Brute-Force
* Uses-64-Bit

Minimum password length supported by kernel: 0
Maximum password length supported by kernel: 16

Watchdog: Temperature abort trigger set to 90c
[s]tatus [p]ause [b]ypass [c]heckpoint [q]uit => 

Note that hashcat sets watchdog to stop the test when GPU temperature rises 90C.

Power consumption/temperature/rpm values increase during the test:

amdgpu-pci-0100
Adapter: PCI adapter
vddgfx:        1.16 V  
fan1:        3577 RPM  (min =  900 RPM, max = 4500 RPM)
edge:         +72.0°C  (crit = +94.0°C, hyst = -273.1°C)
power1:      154.21 W  (cap = 155.00 W)

Comparing to before:

Adapter: PCI adapter
vddgfx:      750.00 mV 
fan1:        1641 RPM  (min =  900 RPM, max = 4500 RPM)
edge:         +28.0°C  (crit = +94.0°C, hyst = -273.1°C)
power1:       32.16 W  (cap = 155.00 W)

Check also libraries and card nodes used by process:

0101b000-0101c000 rw-s dd7f800000000000 00:06 526                        /dev/kfd
0101c000-0101e000 ---p 00000000 00:00 0 
0101e000-0101f000 rw-p 00000000 00:00 0 

55d200000-581200000 ---p 15d98a000 00:06 538                             /dev/dri/renderD128
581200000-581600000 ---p 00000000 00:00 0 
5557fd9b1000-5557fda7d000 r-xp 00005000 08:04 4074901                    /home/evg/projects/hashcat-6.0.0/hashcat

7fcc52208000-7fcc57a16000 r-xp 00000000 08:03 4718606                    /opt/rocm-3.5.0/lib/libamd_comgr.so.1.6.30500
7fcc57a16000-7fcc57c15000 ---p 0580e000 08:03 4718606                    /opt/rocm-3.5.0/lib/libamd_comgr.so.1.6.30500
7fcc57c15000-7fcc57f94000 r--p 0580d000 08:03 4718606                    /opt/rocm-3.5.0/lib/libamd_comgr.so.1.6.30500
7fcc57f94000-7fcc57fa2000 rw-p 05b8c000 08:03 4718606                    /opt/rocm-3.5.0/lib/libamd_comgr.so.1.6.30500
7fcc57fa2000-7fcc58000000 rw-p 00000000 00:00 0 


7fcc5d583000-7fcc5d59d000 r-xp 00000000 08:03 4719401                    /opt/rocm-3.5.0/lib/libhsakmt.so.1.0.30500
7fcc5d59d000-7fcc5d79c000 ---p 0001a000 08:03 4719401                    /opt/rocm-3.5.0/lib/libhsakmt.so.1.0.30500
7fcc5d79c000-7fcc5d79f000 r--p 00019000 08:03 4719401                    /opt/rocm-3.5.0/lib/libhsakmt.so.1.0.30500
7fcc5d79f000-7fcc5d7ab000 rw-p 0001c000 08:03 4719401                    /opt/rocm-3.5.0/lib/libhsakmt.so.1.0.30500
7fcc5d7ab000-7fcc5d7ac000 rw-p 00000000 00:00 0 
7fcc5d7ac000-7fcc5d867000 r-xp 00000000 08:03 4719424                    /opt/rocm-3.5.0/hsa/lib/libhsa-runtime64.so.1.1.30500
7fcc5d867000-7fcc5da67000 ---p 000bb000 08:03 4719424                    /opt/rocm-3.5.0/hsa/lib/libhsa-runtime64.so.1.1.30500
7fcc5da67000-7fcc5da6c000 r--p 000bb000 08:03 4719424                    /opt/rocm-3.5.0/hsa/lib/libhsa-runtime64.so.1.1.30500
7fcc5da6c000-7fcc5da6d000 rw-p 000c0000 08:03 4719424                    /opt/rocm-3.5.0/hsa/lib/libhsa-runtime64.so.1.1.30500
7fcc5da6d000-7fcc5da6f000 rw-p 00000000 00:00 0 
7fcc5da6f000-7fcc5db8e000 r-xp 00000000 08:03 4719441                    /opt/rocm-3.5.0/opencl/lib/libamdocl64.so
7fcc5db8e000-7fcc5dd8e000 ---p 0011f000 08:03 4719441                    /opt/rocm-3.5.0/opencl/lib/libamdocl64.so
7fcc5dd8e000-7fcc5dd93000 r--p 0011f000 08:03 4719441                    /opt/rocm-3.5.0/opencl/lib/libamdocl64.so
7fcc5dd93000-7fcc5dd98000 rw-p 00124000 08:03 4719441                    /opt/rocm-3.5.0/opencl/lib/libamdocl64.so

OK now let's check the status:

Session..........: hashcat
Status...........: Running
Hash.Type........: sha512crypt $6$, SHA512 (Unix)
Hash.Target......: $6$8f5pVFu2VWchx77Z$YpfJuoJltko.d0Iu/6C9m2phVSLs.ZY...YAeFp0
Time.Started.....: Wed Jun 24 18:43:57 2020 (32 secs)
Time.Estimated...: Thu Sep 17 18:28:41 2020 (84 days, 23 hours)
Guess.Mask.......: ?l?l?l?l?l?l?d?d?d [9]
Guess.Queue......: 1/1 (100.00%)
Speed.#2.........:    42069 H/s (11.01ms) @ Accel:64 Loops:16 Thr:64 Vec:1
Recovered........: 0/1 (0.00%) Digests, 0/1 (0.00%) Salts
Progress.........: 1327104/308915776000 (0.00%)
Rejected.........: 0/1327104 (0.00%)
Restore.Point....: 0/11881376000 (0.00%)
Restore.Sub.#2...: Salt:0 Amplifier:9-10 Iteration:256-272
Candidates.#2....: tarier123 -> tmedda123
Hardware.Mon.#2..: Temp: 72c Fan: 61% Core:1370MHz Mem:2000MHz Bus:0

[s]tatus [p]ause [b]ypass [c]heckpoint [q]uit => 

From the status we can understand that our GPU is able to generate 42k hashes per second which is quite impressive comparing to CPU (almost 10 time faster):

$6$8f5pVFu2VWchx77Z$YpfJuoJltko.d0Iu/6C9m2phVSLs.ZY...YAeFp0
Time.Started.....: Wed Jun 24 20:07:43 2020 (9 secs)
Time.Estimated...: Fri Sep 10 06:14:45 2038 (18 years, 77 days)
Guess.Mask.......: ?l?l?l?l?l?l?d?d?d [9]
Guess.Queue......: 1/1 (100.00%)
Speed.#1.........:      538 H/s (11.34ms) @ Accel:256 Loops:32 Thr:1 Vec:4

Interesting to note that hashcat estimates finishing time (using GPU) to 84 days, 23 hours.

The test was successfully finished after 10 mins, 30 secs with the following results:

$6$8f5pVFu2VWchx77Z$YpfJuoJltko.d0Iu/6C9m2phVSLs.ZYuR6PEt6xoZJcxoOPMTlfgfqiYPdhMOdr3j7whvrfYe3wYO3GLYAeFp0:qwerty123
                                                 
Session..........: hashcat
Status...........: Cracked
Hash.Type........: sha512crypt $6$, SHA512 (Unix)
Hash.Target......: $6$8f5pVFu2VWchx77Z$YpfJuoJltko.d0Iu/6C9m2phVSLs.ZY...YAeFp0
Time.Started.....: Wed Jun 24 18:47:14 2020 (10 mins, 30 secs)
Time.Estimated...: Wed Jun 24 18:57:44 2020 (0 secs)
Guess.Mask.......: ?l?l?l?l?l?l?d?d?d [9]
Guess.Queue......: 1/1 (100.00%)
Speed.#2.........:    42173 H/s (10.82ms) @ Accel:64 Loops:16 Thr:64 Vec:1
Recovered........: 1/1 (100.00%) Digests, 1/1 (100.00%) Salts
Progress.........: 26542080/308915776000 (0.01%)
Rejected.........: 0/26542080 (0.00%)
Restore.Point....: 884736/11881376000 (0.01%)
Restore.Sub.#2...: Salt:0 Amplifier:23-24 Iteration:4992-5000
Candidates.#2....: qnhmxy123 -> qbvgga123
Hardware.Mon.#2..: Temp: 87c Fan: 61% Core:1300MHz Mem:2000MHz Bus:0

Started: Wed Jun 24 18:47:13 2020
Stopped: Wed Jun 24 18:57:45 2020

Now let's generate simple password by repeating the symbol and digit:

perl -e 'print crypt("n1n1n1n1n1","\$6\$8f5pVFu2VWchx77Z\$") . "\n"'
$6$8f5pVFu2VWchx77Z$mR4QZ1d9qybVJjZqw8oVSLXTHZbb/nBMMGS9cntceXYpO.w3TXiOei/XqaLdrPYlK/1HUhsrWyuJXekm9xbVJ/

And run hashcat again as follows:

$ hashcat -a 3  -m 1800 '$6$8f5pVFu2VWchx77Z$mR4QZ1d9qybVJjZqw8oVSLXTHZbb/nBMMGS9cntceXYpO.w3TXiOei/XqaLdrPYlK/1HUhsrWyuJXekm9xbVJ/' -O ?l?d?l?d?l?d?l?d?l?d 

First status:

Session..........: hashcat
Status...........: Running
Hash.Type........: sha512crypt $6$, SHA512 (Unix)
Hash.Target......: $6$8f5pVFu2VWchx77Z$mR4QZ1d9qybVJjZqw8oVSLXTHZbb/nB...9xbVJ/
Time.Started.....: Wed Jun 24 20:21:30 2020 (12 secs)
Time.Estimated...: Sun May 16 07:42:01 2021 (325 days, 11 hours)
Guess.Mask.......: ?l?d?l?d?l?d?l?d?l?d [10]
Guess.Queue......: 1/1 (100.00%)
Speed.#2.........:    42251 H/s (10.82ms) @ Accel:64 Loops:16 Thr:64 Vec:1
Recovered........: 0/1 (0.00%) Digests, 0/1 (0.00%) Salts
Progress.........: 442368/1188137600000 (0.00%)
Rejected.........: 0/442368 (0.00%)
Restore.Point....: 0/45697600000 (0.00%)
Restore.Sub.#2...: Salt:0 Amplifier:3-4 Iteration:2400-2416
Candidates.#2....: b2s1a1a1a1 -> b5s9c2k1a1
Hardware.Mon.#2..: Temp: 57c Fan: 20% Core:1370MHz Mem:2000MHz Bus:0

Now it's 10 symbols password and estimation of 325 days in total.

However in Candidates field hashcat for some reasons predicts password not with the same digit and not the same character.

OK. We stopped test after 20 minutes and decided to play around with word-lists. We downloaded a dictionary from here and re-run test test:

Host memory required for this attack: 696 MB

Dictionary cache built:
* Filename..: ~/Downloads/rockyou.txt
* Passwords.: 14344391
* Bytes.....: 139921497
* Keyspace..: 14344384
* Runtime...: 1 sec

[s]tatus [p]ause [b]ypass [c]heckpoint [q]uit => s

Session..........: hashcat
Status...........: Running
Hash.Name........: sha512crypt $6$, SHA512 (Unix)
Hash.Target......: $6$8f5pVFu2VWchx77Z$mR4QZ1d9qybVJjZqw8oVSLXTHZbb/nB...9xbVJ/
Time.Started.....: Wed Jun 24 20:55:05 2020 (5 secs)
Time.Estimated...: Wed Jun 24 21:26:48 2020 (31 mins, 38 secs)
Guess.Base.......: File (~/Downloads/rockyou.txt)
Guess.Queue......: 1/1 (100.00%)
Speed.#2.........:     7544 H/s (7.43ms) @ Accel:8 Loops:16 Thr:64 Vec:1
Recovered........: 0/1 (0.00%) Digests
Progress.........: 18432/14344384 (0.13%)
Rejected.........: 0/18432 (0.00%)
Restore.Point....: 18432/14344384 (0.13%)
Restore.Sub.#2...: Salt:0 Amplifier:0-1 Iteration:4992-5000
Candidates.#2....: sunshine13 -> holabebe
Hardware.Mon.#2..: N/A

As you can see the speed was significantly reduced to 7.5k comparing to 42k in previous test. Estimation for the test 31 minutes (which is not so bad).

OK, now let's create a rule to add 'n1' to each password in the list and re-run the test again:

$ echo "$n$1" > ./rule.rule
$ hashcat -a 0 -d 2 -r rule.rule -m 1800 '$6$8f5pVFu2VWchx77Z$mR4QZ1d9qybVJjZqw8oVSLXTHZbb/nBMMGS9cntceXYpO.w3TXiOei/XqaLdrPYlK/1HUhsrWyuJXekm9xbVJ/' ~/Downloads/rockyou.txt
Session..........: hashcat
Status...........: Running
Hash.Name........: sha512crypt $6$, SHA512 (Unix)
Hash.Target......: $6$8f5pVFu2VWchx77Z$mR4QZ1d9qybVJjZqw8oVSLXTHZbb/nB...9xbVJ/
Time.Started.....: Wed Jun 24 21:09:06 2020 (36 secs)
Time.Estimated...: Wed Jun 24 21:40:42 2020 (31 mins, 0 secs)
Guess.Base.......: File (~/Downloads/rockyou.txt)
Guess.Mod........: Rules (rule.rule)
Guess.Queue......: 1/1 (100.00%)
Speed.#2.........:     7570 H/s (7.23ms) @ Accel:8 Loops:16 Thr:64 Vec:1
Recovered........: 0/1 (0.00%) Digests
Progress.........: 258048/14344384 (1.80%)
Rejected.........: 0/258048 (0.00%)
Restore.Point....: 258048/14344384 (1.80%)
Restore.Sub.#2...: Salt:0 Amplifier:0-1 Iteration:3008-3024
Candidates.#2....: tubby101n1 -> chevy91n1
Hardware.Mon.#2..: N/A

And finally password was cracked:

$6$8f5pVFu2VWchx77Z$mR4QZ1d9qybVJjZqw8oVSLXTHZbb/nBMMGS9cntceXYpO.w3TXiOei/XqaLdrPYlK/1HUhsrWyuJXekm9xbVJ/:n1n1n1n1n1
                                                 
Session..........: hashcat
Status...........: Cracked
Hash.Name........: sha512crypt $6$, SHA512 (Unix)
Hash.Target......: $6$8f5pVFu2VWchx77Z$mR4QZ1d9qybVJjZqw8oVSLXTHZbb/nB...9xbVJ/
Time.Started.....: Wed Jun 24 21:09:06 2020 (12 mins, 56 secs)
Time.Estimated...: Wed Jun 24 21:22:02 2020 (0 secs)
Guess.Base.......: File (/home/evg/Downloads/rockyou.txt)
Guess.Mod........: Rules (rule.rule)
Guess.Queue......: 1/1 (100.00%)
Speed.#2.........:     6775 H/s (8.37ms) @ Accel:8 Loops:16 Thr:64 Vec:1
Recovered........: 1/1 (100.00%) Digests
Progress.........: 5253120/14344384 (36.62%)
Rejected.........: 0/5253120 (0.00%)
Restore.Point....: 5234688/14344384 (36.49%)
Restore.Sub.#2...: Salt:0 Amplifier:0-1 Iteration:4992-5000
Candidates.#2....: nabetamon1 -> mytwinshinen1
Hardware.Mon.#2..: N/A

Started: Wed Jun 24 21:09:04 2020
Stopped: Wed Jun 24 21:22:03 2020

This test shows how simple passwords can be easily cracked taking into account powerful mining farms available for everyone and low cost on such equipment.