Linux tips

From GHER

Revision as of 13:21, 11 March 2013 by Charles (Talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Here you'll find answers to "How to" questions related using command line

Contents

Download files from a ftp

FTP = file transfer protocol.

Examples of commands can be found here: http://linux.about.com/od/commands/a/blcmdl1_ftpx.htm

  • Connect to the server:
ftp ftp.aviso.oceanobs.com

then type you username and password if necessary.

  • Navigate in the directories
ls 
cd global
  • Download the file you want:
get file_name

or for multiple files:

mget *.nc

to download all the .nc files.

  • Close the session:
ftp bye

Find files containing a given text string?

Use the find command:

find . -type f -name "*.m" -exec grep -l StringBuffer {} \;

where 'StringBuffer' is the text string you are looking for.

In my case, useful when searching for a matlab file that created a given figure.

Source: http://www.devdaily.com/unix/edu/examples/find.shtml

Find large files on your machine?

find . -type f -size +500M -exec ls -lh {} \; | awk '{ print $8 ": " $5 }'

where your can modify the minimum size. Useful if your disk is full.

Source: http://www.cyberciti.biz/faq/find-large-files-linux/

Edit multiple files with vi

[charles@gher13 divastripped]$ vi diva*
54 files to edit

:n --> go to next file

:w --> write file

Source: http://www.bo.infn.it/alice/alice-doc/mll-doc/linux/vi-ex/node27.html

Remove all files except one (or one type of file)

ls *|grep -v file2keep.txt|xargs rm -rfv

If we want to apply this recursively:

ls */*|grep -v dineof.init|xargs rm -rfv

Note that the symbolic link are not removed with these options.

Transfer files to another machine but not the symbolic links?

Example: you want to upload files to NIC3. If you use a classic scp, then the symbolic links will be uploaded as files as well. This can be an issue if the links point to large files.

Solution: use the command rsync

rsync -azuv -e ssh QuikSCAT_Canary2009_UVwind_300e360e0n60n ctroupin@nic3:/scratch/ctroupin/DINEOF/

For more information about rsync + ssh: http://troy.jdmz.net/rsync/index.html

Create file names with the date using bash

[charles@gher13 ~]$ date "+%Y_%m_%d_%Hh%M"
2012_01_26_14h02
[charles@gher13 ~]$ date "+%F"
2012-01-26

Source: http://www.cyberciti.biz/faq/unix-linux-appleosx-bsd-shell-appending-date-to-filename/

Replace a given string in several files

Use the sed command:

sed -i 's/old-word/new-word/g' *.txt

Source: http://www.cyberciti.biz/faq/unix-linux-replace-string-words-in-many-files/

Copy files from multiple directories into a single directory

Use the find command combined with a cp command:

find /home/charles/ -name '*.mp3' -exec cp -v {} /home/charles/music/ \;

This command will copy all the mp3 files located in /home/charles/ into the directory /home/charles/music/

Sources:

Remove white space from file name

You can a find a little bash script here: http://design.liberta.co.za/articles/how-to-remove-spaces-from-filenames-in-linuxunix/

#!/bin/bash
find . -name '* *' | while read file;
do
        target=`echo "$file" | sed 's/ /_/g'`;
        echo "Renaming '$file' to '$target'";
        mv "$file" "$target";
done;

Remove empty files from a directory

find . -empty -exec rm -f {} \;^C

Source: http://ayaz.wordpress.com/2008/02/05/bash-quickly-deleting-empty-files-in-a-directory/


Copy large amount of file from one machine to another

If you want to copy or move a large amount of data from one computer to another (e.g., your computer to /shared), it is better to use rsync than cp or mv.

rsync seems to put less load on the system. It is still a good idea to do these tasks overnight or over the weekend.

The other advantage of rsync is that if something happens in the middle of transferring a file, rsync will continue directly from where it left off.

rsync -v --progress -r -t -l -H -p -o -g /home/emason/somefolder/\* . ; date ;  
rsync -v --progress -r -t -l -H -p -o -g /home/emason/somefolder/\* . ; date ;  
rsync -v --progress -r -t -l -H -p -o -g -c  /home/emason/somefolder/\* . ; date

Note that there are 3 rsync commands (1) rsync....., (2) rsync..... exactly the same as the first, (3) rsync..... exactly the same as the first two but with an extra -c flag.

You can find out about the function of each of the flags by studying 'man rsync'.

The last two commands (2 and 3) should not give much output on the screen, but -c involves computing md5sums for each file and then comparing the match and this can take a long time.

Personal tools