name: Painless-Password-Rotation class: center,middle,title-slide count: false  .titletext[Painless Password Rotation] For Linux and Windows Local System Accounts --- name: Retro class: center,middle .center[  <br><sub>Sean at the Seattle Computer Museum</sub> ] ??? My name is Sean Carolan and I'm a Solutions Engineer with HashiCorp. I like building things. In my past life I was a Linux system administrator, now I travel and help users get started with Infrastructure as Code. My first computer was a Commodore 64. This photo was taken a few years ago at the Living Computer Museum. If you are ever in Seattle, you owe it to yourself to make a pilgrimage to this place. There's nothing that makes you feel old like seeing your childhood toy in a museum. Elliot and I are playing Oregon Trail on an Apple IIe. Password Rotation! Woooo! Who's with me? No...nobody? Why isn't anyone excited? Are you sure you don't want to go hear about Terraform 0.12? Ok, you're stuck with me! Let's be honest, nobody came to this talk because they like changing passwords. I'll give you an example...(Advance to Next slide) --- name: Change-Your-Password-14 class: center,middle  ??? Does anyone else do this? Oh I've got two whole weeks. I don't want to change my password, I can still get two weeks more use out of it. After all, it took me three weeks just to memorize the darn thing. --- name: Change-Your-Password-7 class: center,middle  ??? You know what I'm busy right now. Come back later. --- name: Change-Your-Password-3 class: center,middle  ??? A day may come when I change this password. But today is not that day. --- name: Change-Your-Password-2 class: center,middle  ??? Why can't I keep the password I have? I really like my password. --- name: Change-Your-Password-1 class: center,middle  ??? Ok, I'll do it this afternoon. --- name: Change-Your-Password-Expired class: center,middle  ??? Nobody likes this! And this is just one password on one machine. Why do we participate in this crazy ritual every few months? How did we get here? And most importantly, who can we blame? I did some research and found the inventor of the modern computer password... --- name: Inventor-of-Password class: center Inventor of the Computer Password ------------------------- .center[  <br> <sub>Fernando Corbató with MIT's IBM 7090</sub> ] ??? This dashing gentleman is Fernando Corbató. He is one of the pioneers of timesharing computer systems. He lead the team that developed the first Compatible Time Sharing System for mainframe computers. He also invented the modern computer password. You see, back in those days computers were giant hulking machines that filled entire rooms. These powerful machines were big and expensive to run. Users would write their programs on punch cards, and then submit the punch cards to operators to run. (Think about that next time you go to deploy your code...) Programs were put in a queue and run in serial, and you often had to wait days or hours for your program to run. It wasn't practical to have developers connected to the mainframe all the time, because most of the time they were writing code, not actually running it. Fernando and his team invented time sharing to solve this problem. Multiple users could now connect to the same mainframe computer, and the idle time of one user could be used by another user connected to the system. Finally multiple users could interact with the mainframe system in real time. "The key problem was that we were setting up multiple terminals which were to be used by multiple persons but with each person having his own private set of files," he told Wired. "Putting a password on for each individual user as a lock seemed like a very straightforward solution." Thus the password was born... And if Hollywood has taught us anything, it's that the handsome scientist's experiment is going to go horribly wrong by the end of the movie... Source: https://www.wired.com/2012/01/computer-password/ --- name: Science-Gone-Wrong class: center Science Gone Wrong ------------------------- .center[  <br> <sub>Return of the Fly (1959)</sub> ] ??? Ok, maybe things didn't turn out quite *that* badly. Dr. Corbato is still alive and healthy, and he didn't turn into a fly. But there problems with the password began almost immediately. Less than a year after the password was invented, the first instance of password theft occurred in 1962. You see, these machines were very expensive to run and you were sharing them with everyone else on the team. Each user was allocated only four hours per week of compute time for their projects. One of the other researchers on the project, Dr. Allan Scherr, was frustrated with the 4 hour time limit. He knew that there was an 'offline print' function whereby you could submit a punchcard, and print out files overnight while no one was in the lab. So he submitted a job to print out the entire list of system passwords on Friday night, and picked it up first thing on Saturday morning. In order to cover his tracks he shared the password list with several other users, one of whom immediately began to troll the administrator by leaving messages on the system under his user account. And just like a sci-fi movie plot, the brilliant scientist's creation eventually came back to haunt him. "It's become kind of a nightmare," says the 87-year-old retired researcher. "I don't think anybody can possibly remember all the passwords." Let's turn the clock forward a bit to the 1970s... --- name: Miss-Piggy Meet Miss Piggy ------------------------- .center[  <br> <sub>Living Computer Museum PDP-11</sub> ] ??? This morning, just for fun I wanted to log onto the oldest legacy machine I could find. The Living Computer Museum that I mentioned on the first slide has a bunch of legacy machines that you can get accounts on. This one is a PDP-11/70, first built in 1975. This particular machine is running Unix v7. Everybody's passwords on this machine are still stored in that same file, but we've made some improvements. --- name: Evolution-of-the-Computer-Password Evolution of Computer Passwords ------------------------- <br><br><br><br> .center[  <sub>Linux /etc/shadow file</sub> ] ??? The password was here to stay, and for a long time passwords were stored in plaintext, usually in a file called /etc/passwd. In the 1970s, Robert Morris developed a way to store passwords in a hashed form. This way the computer system didn't have to store your password in plain text, making it much harder for someone to guess your password. Later the concept of the 'salted' hash was created, whereby we add some random data to the password before hashing it. These measures made things a little bit better but there are still many ways password authentication can go wrong. But as time went by, computers got more powerful. Breaking a simple hashed password can now be done in seconds with a password cracking tool. Security experts have long recommended regular rotation of passwords. Security experts also recommended longer passwords with numbers and symbols in them. Let's go ahead and try to set our password! --- name: Cats-Are-Evil class: center .center[  <br> <sub>No animals were harmed in the making of this slide.</sub> ] ??? I had hoped to score some cheap internet points by putting a cat in my presentation, but she wanted nothing to do with it. This is my demon cat Mochiko. Mochiko means 'white rice flour'. We'll use our pet's name for our password, because it's easy to remember right? --- name: Choosing-a-Secure-Password-Too-Short class: center,middle # Mochi ??? This is easy to remember, but unfortunately also easy to crack. I ran a password cracking tool against a hashed version of this and it took less than two minutes to break, using an instance that costs around 7 cents an hour to run. --- name: Choosing-a-Secure-Password-Not-Complex class: center,middle # Mochiko ??? A little better but not much. Security folks say it needs letters *and* numbers. --- name: Choosing-a-Secure-Password-Needs-Symbol class: center,middle # Mochiko1 ??? Do we really think we're fooling anyone by adding a 1 to the end of this? And yet its still not secure enough. We have to add at least one non alphanumeric symbol to it. --- name: Choosing-a-Secure-Password-Okay-Finally class: center,middle # M0ch!k01$ ??? Ok, we've substituted a ! for the i, and tacked a dollar sign on the end. Nice and secure. Let's make sure we never forget this password... --- name: Save-Your-Password class: center,middle .center[  ] ??? Why does this happen? Because we like to torture our users with really weird complexity requirements for passwords. Humans can on average, reliably remember a group of 7 digits, six letters, and five words. https://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two Believe it or not there was a time when many of us could recite a dozen phone numbers by heart. Can anyone remember five phone numbers? Three? Do you know any phone numbers? Some people don't even remember own phone number anymore. Yet we've told our poor users they have to remember this string of 8 or 10 digits/symbols/letters. What if instead we gave them a string of four or five words to learn? --- name: A-Better-Password class: center .center[  <br> <sub>Source: XKCD https://xkcd.com/936/</sub> ] ??? Now you only have to remember four things. You already know how to spell these words so you no longer have to memorize each and every letter. Sadly super long passwords made of words are not supported on many systems, because they are not 'complex' enough. --- name: The-Password-Safe The Password Safe ------------------------- .contents[ * A safe or 'wallet' for all your passwords * Stored on smartphone, laptop, or cloud * Uses a master password to unlock * Okay for personal use * Challenging for teams * Nearly impossible for automated apps ] ??? Nowadays we have to have a 'wallet' or 'password safe' just to keep track of the many passwords we use for various accounts and services. Many people store their passwords on their browser or in an app on their smartphone. So the situation on personal devices is more or less tolerable...but how can we deal with password rotation on servers and cloud instances? What if you have 300 machines to manage, each with its own local accounts? We've looked at the history of password security, let's take a short peek into the present and future: --- name: Dynamic-Credentials class: center Self-Destructing Credentials -------------------------  ??? Here's one possible solution to the problem of password management in a modern infrastructure: What if you could hand out usernames and passwords that automatically expire. Users and applications can simply check out new credentials when they need them, use them for a specific amount of time, and then the credentials automatically vanish when they expire? Sounds great doesn't it? --- name: Dynamic-Credentials-for-Cloud-and-DB class: center Dynamic Database or Cloud Credentials with HashiCorp Vault ------------------------- <br><br><br><br>  ??? Imagine a ski resort. You walk up to the main counter and purchase a lift pass. The lift pass is like dynamic credentials. You might have a beginner's pass that is only good for the bunny slopes. Or you might have a pass that gives you elevated privileges, and allows you to ride to the top of the mountain. All of the passes have an expiration date on them. You simply present your pass at the lift, an employee scans it (or it happens automatically via RFID), and you get access to the lift. Vault works the same way. You can request database credentials, or public cloud API keys, and get temporary access to the systems and APIs that you need to work with? Sounds great right? Yes, I can hear you thinking..."But Sean, I'm stuck with all these legacy machines that I have to manage, and we can't do all this fancy dynamic stuff." --- name: Legacy-Systems Think of the Legacy Systems ------------------------- <br><br><br><br><br><br> ```bash $ uptime 18:17:07 up 2555 days, 3:57, 1 user, load average: 0.16, 0.07, 0.06 ``` leg·a·cy (adjective): _denoting software or hardware that has been superseded but is difficult to replace because of its wide use._ ??? Let's talk about legacy systems. Anyone have a machine like Miss Piggy on your network? Who here manages legacy systems for your organization? Please stand up. Let's give these people a hand. These folks are heroes for taking care of those ancient systems that nobody else wants to touch. Be nice to your sysadmins and ops people. What are we going to do about these legacy systems? Some of my customers have started giving them new names. A couple weeks ago an IT director introduced me to his "Classic" environment. But I think my favorite is the bank, with their "Heritage" collection of machines. Heritage systems. You can put that on your resume. Curator of Heritage Systems. If there's one thing we know about these legacy systems, it's that they are not going away any time soon. These dinosaurs have survived this long, and somebody's gotta care for them. The security team says the root and admin passwords on these boxes has to be rotated every 90 days. How are we going to tackle this challenge? We set out to design a system that would automate system password rotation and storage in a secure way. Here are the design requirements: --- name: Design-Requirements-1 Design Requirements ------------------------- .biglist[ - No external software. Works with Bash or Powershell. ] ??? You don't need any special management tools other than the Bash or Powershell that already ships with your system. --- name: Design-Requirements-2 Design Requirements ------------------------- .biglist[ - No external software. Works with Bash or Powershell. - Trusted source of entropy for password generation. ] ??? The first revision of this script used openssl or powershell to generate random strings. I didn't like this because 1. We are not sure we can trust every machine to get this right and 2. We might not have openssl or other random generator tools installed on our clients. Instead, we're going to use Vault itself to generate new passwords. --- name: Design-Requirements-3 Design Requirements ------------------------- .biglist[ - No external software. Works with Bash or Powershell. - Trusted source of entropy for password generation. - Each machine responsible for its own password rotation. ] ??? We do not want to be in the business of micro-managing all these passwords on various machines. Each system should take care of itself. --- name: Design-Requirements-4 Design Requirements ------------------------- .biglist[ - No external software. Works with Bash or Powershell. - Trusted source of entropy for password generation. - Each machine responsible for its own password rotation. - Supports random secure passwords or passphrases. ] ??? We want to support either long strings of characters, or the more human friendly passphrase style passwords. --- name: Design-Requirements-5 Design Requirements ------------------------- .biglist[ - No external software. Works with Bash or Powershell. - Trusted source of entropy for password generation. - Each machine responsible for its own password rotation. - Supports random secure passwords or passphrases. - All passwords are encrypted in transit and at rest. ] ??? Everything must be encrypted both in transit and at rest. --- name: Design-Requirements-6 Design Requirements ------------------------- .biglist[ - No external software. Works with Bash or Powershell. - Trusted source of entropy for password generation. - Each machine responsible for its own password rotation. - Supports random secure passwords or passphrases. - All passwords are encrypted in transit and at rest. - Servers have write-only access to HashiCorp Vault. ] ??? Servers should be able to write, but not read their own passwords. This essentially creates a one-way door, so that even if a system is compromised the attacker still can't read the passwords. --- name: Design-Requirements-7 Design Requirements ------------------------- .biglist[ - No external software. Works with Bash or Powershell. - Trusted source of entropy for password generation. - Each machine responsible for its own password rotation. - Supports random secure passwords or passphrases. - All passwords are encrypted in transit and at rest. - Servers have write-only access to HashiCorp Vault. - N number of previous password versions stored in Vault. ] ??? We want to have previous versions of the password just in case we need them. --- name: Design-Requirements-8 Design Requirements ------------------------- .biglist[ - No external software. Works with Bash or Powershell. - Trusted source of entropy for password generation. - Each machine responsible for its own password rotation. - Supports random secure passwords or passphrases. - All passwords are encrypted in transit and at rest. - Servers have write-only access to HashiCorp Vault. - N number of previous password versions stored in Vault. - Humans are granted read access to passwords by policy. ] ??? Humans should be able to get to these passwords by logging onto vault or using the API. Humans may not *write* to the password store, access is read only. --- name: Design-Requirements-9 Design Requirements ------------------------- .biglist[ - No external software. Works with Bash or Powershell. - Trusted source of entropy for password generation. - Each machine responsible for its own password rotation. - Supports random secure passwords or passphrases. - All passwords are encrypted in transit and at rest. - Servers have write-only access to HashiCorp Vault. - N number of previous password versions stored in Vault. - Humans are granted read access to passwords by policy. - Passwords are automatically rotated every X days. ] ??? And finally, the whole point of this presentation, is that passwords get rotated every 90 days. Let's go ahead and show you how to set it all up in your own environment: --- name: Setup-Vault Set Up Vault ------------------------- <br><br><br><br><br><br> .center[Instructions for setting up an HA-enabled Vault cluster are here:] .center[ ###https://learn.hashicorp.com/vault/ ] --- name: Setup-Vault Install the Password Generator Plugin ------------------------- <br><br><br><br><br><br> Password generator plugin, creates new `gen/password` and `gen/passphrase` API endpoints: .center[ ### https://github.com/sethvargo/vault-secrets-gen ] ??? Next we'll configure policies for both our servers and our sysadmins. The servers will be allowed to write new passwords to Vault, while the sysadmins have read only access to the passwords. --- name: Enable-KV-Secrets-Engine Enable a Key/Value v2 Secrets Engine ------------------------- <br><br><br><br><br> Version 2 of the kv secrets engine supports secret versioning. ```bash vault secrets enable -version=2 -path=systemcreds/ kv ``` --- name: Configure-Policies-Linux Configure ACL Policies for Linux ------------------------- rotate-linux.hcl ```bash # Allows hosts to write new passwords path "systemcreds/data/linux/*" { capabilities = ["create", "update"] } # Allow hosts to generate new passphrases path "gen/passphrase" { capabilities = ["update"] } # Allow hosts to generate new passwords path "gen/password" { capabilities = ["update"] } ``` linuxadmin.hcl ```bash # Allows admins to read passwords. path "systemcreds/*" { capabilities = ["list"] } path "systemcreds/data/linux/*" { capabilities = ["list", "read"] } ``` --- name: Configure-Policies-Windows Configure ACL Policies for Windows ------------------------- rotate-windows.hcl ```bash # Allows hosts to write new passwords path "systemcreds/data/windows/*" { capabilities = ["create", "update"] } # Allow hosts to generate new passphrases path "gen/passphrase" { capabilities = ["update"] } # Allow hosts to generate new passwords path "gen/password" { capabilities = ["update"] } ``` windowsadmin.hcl ```bash # Allows admins to read passwords. path "systemcreds/*" { capabilities = ["list"] } path "systemcreds/data/windows/*" { capabilities = ["list", "read"] } ``` --- name: Generate-a-Token Generate a Token ------------------------- Create a new token for each of your Windows and Linux servers. You'll probably want to script this part if you have a large number of machines. ```bash $ vault token create -policy=rotate-linux -period=24h Key Value --- ----- token 6vtW8IOVRTBkvPfUV3RNx7vB token_accessor 6CNHqSIG79t7mzROq6YRYPTl token_duration 24h token_renewable true token_policies [default rotate-linux] ``` ??? This is what's known as a 'periodic' token, which has no TTL or expiration date, however, the user or application must check in at least every 24 hours or Vault will invalidate the token. --- name: Configure-the-Scripts Configure the Scripts ------------------------- Copy the script to your server. Set up your environment variables. Windows .center[  ] Linux ```bash export VAULT_ADDR=https://your.vaultserver.com:8200 export VAULT_TOKEN=762XkidjlyxyzBwYqPGKWAAE ``` --- name: Run-the-Script Run the Password Rotation Script ------------------------- Bash: ```bash ./rotate-linux-password.sh root ``` Powershell: ```powershell ./rotate-windows-password.ps1 Administrator ``` --- name: Renew-Our-Token Renew Our Token ------------------------- Bash: ```bash curl -sS --fail -X POST -H "X-Vault-Token: $VAULT_TOKEN" ${VAULT_ADDR}/v1/auth/token/renew-self | grep -q 'lease_duration' retval=$? if [[ $retval -ne 0 ]]; then echo "Error renewing Vault token lease." fi ``` Powershell: ```powershell Invoke-RestMethod -Headers @{"X-Vault-Token" = ${VAULT_TOKEN}} -Method POST -Uri ${VAULT_ADDR}/v1/auth/token/renew-self if(-Not $?) { Write-Output "Error renewing Vault token lease." } ``` --- name: Generate-New-Password Generate a New Password ------------------------- Bash: ```bash NEWPASS=$(curl -sS --fail -X POST -H "X-Vault-Token: $VAULT_TOKEN" -H "Content-Type: application/json" --data '{"words":"4","separator":"-"}' ${VAULT_ADDR}/v1/gen/passphrase | jq -r '.data|.value') JSON="{ \"options\": { \"max_versions\": 12 }, \"data\": { \"${USERNAME}\": \"$NEWPASS\" } }" ``` Powershell: ```powershell $NEWPASS = (Invoke-RestMethod -Headers @{"X-Vault-Token" = ${VAULT_TOKEN}} -Method POST -Body "{`"length`":`"36`",`"symbols`":`"0`"}" -Uri ${VAULT_ADDR}/v1/gen/password).data.value $SECUREPASS = ConvertTo-SecureString $NEWPASS -AsPlainText -Force $JSON="{ `"options`": { `"max_versions`": 12 }, `"data`": { `"$USERNAME`": `"$NEWPASS`" } }" ``` --- name: Save-New-Password-Linux Save the New Password (Linux) ------------------------- ```bash # First commit the new password to vault, then capture the exit status curl -sS --fail -X POST -H "X-Vault-Token: $VAULT_TOKEN" --data "$JSON" ${VAULT_ADDR}/v1/systemcreds/data/linux/$(hostname)/${USERNAME}_creds | grep -q 'request_id' retval=$? if [[ $retval -eq 0 ]]; then # After we save the password to vault, update it on the instance echo "$USERNAME:$NEWPASS" | sudo chpasswd retval=$? if [[ $retval -eq 0 ]]; then echo -e "${USERNAME}'s password was stored in Vault and updated locally." else echo "Error: ${USERNAME}'s password was stored in Vault but *not* updated locally." fi else echo "Error saving new password to Vault. Local password will remain unchanged." exit 1 fi ``` --- name: Save-New-Password-Windows Save the New Password (Windows) ------------------------- ```powershell Invoke-RestMethod -Headers @{"X-Vault-Token" = ${VAULT_TOKEN}} -Method POST -Body $JSON -Uri ${VAULT_ADDR}/v1/systemcreds/data/windows/${HOSTNAME}/${USERNAME}_creds if($?) { Write-Output "Vault updated with new password." $UserAccount = Get-LocalUser -name $USERNAME $UserAccount | Set-LocalUser -Password $SECUREPASS if($?) { Write-Output "${USERNAME}'s password was stored in Vault and updated locally." } else { Write-Output "Error: ${USERNAME}'s password was stored in Vault but *not* updated locally." } } else { Write-Output "Error saving new password to Vault. Local password will remain unchanged." } ``` --- name: Schedule-It Schedule the Script ------------------------- Linux ``` # m h dom mon dow command */5 * * * * /usr/local/bin/rotate-linux-password.sh root ``` Windows .center[  ] --- name: Live-Demo class: center,middle Live Demo ========================= ??? Let's do a live demo! --- name: Security-Considerations Security Considerations ------------------------- *What if an attacker gets their hands on one of these system tokens?* * Can write arbitrary data to systemcreds/linux or systemcreds/windows * Cannot overwrite existing password, could roll off all previous versions * Possibly write so much data that the k/v storage gets filled up * DDOS or failure of Vault cluster could lock you out of systems *Possible ways to mitigate risk* * Configure Vault token so it can only be used during a short time window (5 minutes). Example: 03:57:30 to 04:02:30. (Enterprise) * Limit length of data that can be written into the k/v store. (Enterprise) * Use Vault in HA mode and use DR replication for emergencies. (Enterprise) * Monitor Vault audit logs for any unusual activity on systemcreds/linux and systemcreds/windows endpoints --- name: Active-Directory-Rotation Active Directory Password Rotation ------------------------- If you have system accounts in Active Directory, you can also rotate those passwords using Vault. * Uses "Lazy" rotation * Passwords must be past their TTL *and* be requested * Designed for high-load environments where many instances are accessing and sharing the password .center[https://www.vaultproject.io/docs/secrets/ad/index.html] --- name: Download-the-Code Download the Code ------------------------- <br><br><br><br><br> .center[ #https://git.io/fxKtU #https://scarolan.github.io/painless-password-rotation ] --- name: Thank-You class: middle,center Thank You for Attending! ------------------------- .center[  ]