Skip to content

shell-notes.org

Latest commit

 

History

History
452 lines (431 loc) · 14.9 KB

shell-notes.org

File metadata and controls

452 lines (431 loc) · 14.9 KB

Notes on the Shell and associated commands

-*- mode: org; comment-column: 0; -*-

This file contains a list of shell commands that I’ve found handy in situations I’ve faced over time. These are commands that I have used (successfully). I’m not claiming that this is the only way or even the right way to handle a particular situation that you may face. YMMV.

The commands are ordered by type rather than difficulty, so you are going to find intermediate and slightly advanced commands intermingled with the easy ones. I mean for this document to be used as a cheat-sheet, not a tutorial, but if you are confused about something shoot me a mail

Redirection

  • some_cmd > some_file 2>&1 &
  • some_cmd > >(tee -a stdout.log) 2> >(tee -a stderr.log >&2)

Shell shortcuts

Copy the last command on the shell

  • !!

Copy the nth command on the shell

  • !N

Copy the last argument of the previous command

  • !$ or Esc .

Copy the nth argument from the previous command

  • !:3 gets you the 3rd argument of the previous command
  • !* gets all the arguments of the previous command
  • !:1-2 gets all but the last three arguments of the previous command

Just print any command in history starting with something. eg: rsync:p

  • :p

Expand the variable

  • C-M-e or ESC C-e

Search through previous command history

  • C-r

Copy all args of the last command

  • !*

Delete backwards

  • C-w

Replace old with new in the previous command

  • ^old^new

Exansions across multiple folders. eg: cp handlers/{user,site,profile}/ .

  • {}

Get specific arguments from previous commands (with pos/neg count)

  • M-.

Show matching expansions

  • M-=

Expand hostname

  • M-@

Comment out the command

  • M-#

Redirect both error and output of a command to a file using tee

lein deps :tree |& tee deps.log

This command (above) is a shortcut for:

lein deps :tree 2>&1 | tee deps.log

Redirect error to one file and output to another file

If you want to collect the output into two different files, use process substitution. 
lein deps :tree > >(tee depsout.log) 2> >(tee depserr.log >&2)

Info

  • install-info –info-file=sicp.info –dir-file=dir

Finding things

How do I run a command in all directories under the current directory?

find . -maxdepth 1 -type d \( ! -name . \) -print0 | xargs -0 -L1 sh -c 'echo "Working in $0" && cd "$0" && git fetch origin'
  
find . -maxdepth 1 -type d \( ! -name . \) -exec sh -c 'echo "Working in $0" && cd "$0" && git fetch origin'

How do I list all files of *.h and *.cc?

find . \( -name "*.h" -o -name "*.cc" \) -print

How to I list all the directories in which a certain string occurs?

Useful for finding, for example, API keys across repos 
rg -l -uu string-to-search . | cut -d'/' -f2 | uniq

Shell + Emacs

  • command | (f=$(mktemp); cat > $f; emacsclient $f; rm -v $f)
  • command | (f=$(mktemp); cat > $f; emacsclient $f; cat $f)
  • Run M-! in Emacs

Process utils

  • ps -eTp
  • /usr/share/bcc/tools/cachetop
  • /usr/share/bcc/tools/filetop -s writes -p `pgrep mongod`
  • sudo perf record -F99 -p `pgrep mongod`-g -- sleep 120
  • procs as a replacement for ps

Check the uptime of a process using ps

ps -eo pid,comm,lstart,etime,time,args | grep <process>

Dealing with processes in the foreground and background

  • List all jobs: jobs -l
  • Starting a process in the background: emacs &
  • Bringing it to the foreground:
    • jobs -l to see jobs
    • fg %<job-num> to bring it in the foreground.
  • Pushing a job to the background
    • C-z to stop the job
    • jobs -l to see jobs
    • bg %<job-num> to push the job in the background and start it again.

Sum numbers in shell

  • <cmd> | paste -sd+ - | bc

SSH

Creating a new key pair

ssh-keygen -f full_output_file_path.id_rsa
# Fill out the necessary values to override the defaults The -f is
# provided here, because if you make a mistake this command will
# override your id_rsa file.

Adding a key to ssh-agent

ssh-add -t 86400 -k ~/.ssh/id_rsa
# -t = time for which key should be stored in agent
# -k = key to add to agent

SSH break bad connection

  • Press ~. on the shell

NTP: fix the date on your machine

  • Install ntp and ntpdate
    • sudo apt install ntp ntpdate
  • Sync time with ntpdate
    • sudo ntpdate 0.us.pool.ntp.org
  • If you see an error that ntp socket is already in use, turn off ntpd (that is generally what is running on 123 port)
    • ps ax | grep ntpd
    • sudo kill <pid>
  • This should sync your time correctly with the ntp servers.

Locate

Updating locate db on Mac

sudo /usr/libexec/locate.updatedb

Updating locate db on Ubuntu

sudo updatedb

Dig

  • The following notes are from: https://www.thegeekstuff.com/2012/02/dig-command-examples/
  • By default, the ANSWER section returns A records. Eg: dig redhat.com +noall +answer
  • You can ask dig to return different record types using -t flag.
    • Eg: dig -t MX redhat.com +noall +answer
    • dig -t NS redhat.com +noall +answer
  • You can view all records with ANY type.
    • dig -t ANY redhat.com +noall +answer
  • You can do a reverse lookup on an IP address using -x
    • dig -x 209.132.183.81 +noall +answer
  • You can use a specific DNS server to answer your query.
    • dig @ns1.redhat.com redhat.com
  • You can pass a newline separated domain name file to dig for bulk queries
    • dig -f bulk.txt +noall +answer
    • where bulk.txt has one domain name per line
  • Default dig options can be stored in the $HOME/.digrc file
    • cat $HOME/.digrc prints +noall +answer

wget notes

Recursively fetch data from a webpage.

I’ve recently read through wget’s man-page and boy is it a super tool! I wish someone had taught me about it earlier.

wget --directory-prefix=/Users/vedang/src/data/PBT/ -e robots=off --page-requisites --adjust-extension --convert-links --recursive -l 1 --span-hosts --accept-regex property-based-testing https://wickstrom.tech/blog.html
  • page-requisites downloads all the images, js, stylesheets and stuff needed to render the page properly.
  • adjust-extension adds extensions to filenames based on content-type (in case the proper extension does not already exist)
  • convert-links adjusts any links on the page to point to the locally downloaded resources (if the resource has been downloaded).
  • recursive -l 1 says to recursively download content upto a depth of 1 from the parent page.
  • span-hosts says that it’s okay to visit other websites when recursively downloading data.
  • accept-regex gives a regular expression to filter a subset of URLs to download in the recursive descent.

All of this put together means that I can download the exact 4 posts I want in this example for easy offline reading.

Download from wget such that you can resume the download

Shell Script

To read the scripting manual, run man bash.

Conditionals and tests in Shell Script

To understand all the conditional flags that you can use in a shell script, read man test.

Observing traffic on the network

ngrep

  • Any UDP from any local interfaces to 91.22.38.4:12201 
    ngrep -W byline -d any udp and host 91.22.38.4 and dst port 12201

tcpdump

  • Show UDP packet header and data parts 
    tcpdump -i lo -n udp port 8125 -X
  • Save packets to pcap file for inspection in wireshark 
    tcpdump -i lo -n udp port 14550 -w packets.pcap

Sending traffic to a socket

nc (netcat)

  • nc is great because you can pipe stdin / stdout to it.
  • Start a server: 
    nc -l localhost 3000
  • Start a client: 
    nc localhost 3000

Ripgrep (rg) notes

Ripgrep: Searching in the entire dir, including in ignore files

rg --no-ignore <text to search>
rg -uu <text to search>

From the man page:

-u, –unrestricted Reduce the level of “smart” searching. A single -u won’t respect .gitignore (etc.) files. Two -u flags will additionally search hidden files and directories. Three -u flags will additionally search binary files.

‘rg -uuu’ is roughly equivalent to ‘grep -r’.

Remove color markers from command output

Commands which print output in color make it difficult to work with data using pipes etc. Here is a simple Sed script to remove the markers: 
gsed -r "s/\x1B\[([0-9]{1,3}(;[0-9]{1,2})?)?[mGK]//g"

Cron and Crontab

Some basic commands:
  • crontab -l to list all the current running cron jobs
  • crontab -e to add a new job to the list.