File System -‐ Overview - the Rostlab!

File System -‐ Overview - the Rostlab!

File  System  -­‐  Overview   / sbin bin tmp usr X11 bin bin var proc dev home a b c sbin http://en.wikipedia.org/wiki/Filesystem_Hierarch...

2MB Sizes 0 Downloads 7 Views

Recommend Documents

The Google File System
Master stores three major types of metadata: •File and chunk namespaces. •Mapping from files to chunks. •Location

Exploring the file system
Rename the file myfile.txt and the subdirectory X d. Copy myfile.txt into ... Solution: Explore the file system. 2. ....

The Unix file system.
Oct 19, 1998 - File Systems in Unix ... There are four types of files in the Unix file system. ... You can use this comm

The Unix File System
The inode is the building block of the UNIX file system. They contain all kinds of information about the directory entry

File system
setgid bit on a directory causes the newly created files in directory to take group ownership of directory. • Sticky b

CV Burkhard Rost - the Rostlab!
Jan 19, 2016 - Burkhard Rost obtained his doctoral degree (Dr. rer. nat.) from the Univ. of Heidelberg (Germany) in the

11. File system Implementation
File-System Structure (3). ▫ File control block – storage structure consisting of information about a file. ◇ owne

Unix File System
May have setuid (sticky) bit set. • time of last access, and modification. • number of links (aliases) to the file.

Journaling file system forensics
Time from file systems and less conventional sources. ... Time Size MAC Permission Owner Group File name ... Size MAC Pe

General Pr0ken File System
Apr 8, 2015 - User Space Utilities: - More than 200 binaries / shell scripts. - Symlinked or plain copies. - Some setuid

File  System  -­‐  Overview   / sbin

bin

tmp usr

X11 bin

bin

var

proc

dev

home a

b

c

sbin

http://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard Practical: EidPB WS2013/14

Command  Overview  

● 

● 

● 

● 

File  System:   cd,  ls,  mkdir,  rmdir,  ln,  touch,  mv,  pwd,  rm,  cp,   chmod,  chown,  chgrp   Files:   cat,  less,  head,  tail,  wc,  strings,  cut,  paste,  join,   uniq,  file     Processes   ps,  top,  kill,  killall   Search   grep,  find,  apropos,  locate   Practical: EidPB WS2013/14

Command  Overview  2   ● 

● 

● 

● 

Archiving   (g)zip,  tar   System  status   df,  du,  env,  date,  who,  sGy,  mount   DocumentaIon   man,  info   User  administraIon   passwd,  id   Practical: EidPB WS2013/14

Log  into  the  System   ● 

● 

Get  yourself  access  to  the  system  (Login):   - 

Enter  your  Login  name  (user  name  issued  by  the   systems  administrator)  

- 

Enter  your  password  (invisible)  

Change  your  password  with  the  command   passwd:   passwd  Username   old  password   new  password  (2x)   Practical: EidPB WS2013/14

InformaIons  about  Users   ●  ● 

● 

Id  login   gives  informaIonen  about  user  name,  user  id,   primary  and  addiIonal  group  id’s  and  group   names   informaIon  about  currently  logged-­‐in  users:   command  who

Practical: EidPB WS2013/14

Online  DocumentaIon   ● 

● 

man:  manual   - 

original  Unix  help  system  

- 

Manpages  are  organized  in  9  secIons  

- 

indexed  in  its  own  database  

- 

RestricIon  to  certain  secIons  or  with  keyword   possible  

apropos:  searches  to  index  with  a  keyword,   yields  all  occurences  in  document  Itles   Practical: EidPB WS2013/14

Online  DocumentaIon  II   ● 

info:  GNU  documentaIon  tool   - 

hyper  text  system  

- 

handling  needs  geVng  used  to,  but  very  powerful  

- 

indexed  in  own  database  

- 

 actual  informaIon  source  for  the  big  soWware   packages  provided  by  GNU  like  GNU  compiler,  Latex,   etc  

Practical: EidPB WS2013/14

File  System  1   ● 

● 

get  the  current  directory   pwd:  print  working  directory   change  to  another  directory   cd:  change  directory   cd    changes  to  your  home  directory   cd ..    changes  one  level  toward  root   cd  Path_to_Dest  changes  to  directory  Dest  

Practical: EidPB WS2013/14

File  System  2   ● 

● 

List  the  content  of  a  directory:   ls:  list  directory   important  opIons:   -­‐a  all  files  (included  hidden  files)     -­‐l    with  comprehensive  informaIon   -­‐d  do  not  list  directories   -­‐t    sort  according  to  date  of  last  modificaIon   -­‐r    reverse  sorIng   -­‐h  „human  readable ,  size  in  Kilo-­‐        Mega  byte   -­‐S  sorts  according  to  file  size    

Practical: EidPB WS2013/14

File  System  3   ● 

● 

Create  a  directory:   mkdir:  make  directory   mkdir  directoryName   Delete  a  directory:   rmdir:  remove  directory   rmdir  directoryName   works  only  if  the  directory  is  empty  

Practical: EidPB WS2013/14

File  System  4   ● 

● 

Copying  files:   cp:  copy  files   cp  fromFile  toFile     cp  fromFile  toDirectory     opIons:   -­‐p  keeps  all  informaIons,  permissions,  etc.   -­‐r  -­‐R  recursive,  if  you  want  to  copy  whole      directories        

Practical: EidPB WS2013/14

File  System  5   ● 

● 

moving  resp.  renaming  of    files:   mv:  move  files   mv  fromFile  toFile     mv  fromFile  toDirectory   mv  fromDirectory  toDirectory   Warning:   -­‐exisIng  files  are  overwriGen   -­‐exisIng  directories  collect  moved  directories       Practical: EidPB WS2013/14

File  System  6   ● 

● 

● 

DeleIng  Files:   rm:  remove  files   rm    file     rm –r  directory   Op=ons:   -­‐r    recursive  deleIon  of  directories   -­‐f    suppress  request  for  confirmaIon  when      deleIng  write-­‐protected  files   -­‐i    interacIve,  user  confirms  deleIon   WARNING:  DeleIon  is  final  and  irrevocable,   Use  rm -rf  only  with  extreme  cauIon   Practical: EidPB WS2013/14

File  System  7   ● 

● 

● 

CreaIon  of  links:   ln:  link  files  (alias,  nic  name)   ln  (path/)file  path/(LinkName)   OpIons:   -­‐s    creates  a  symbolic  link,  i.e.  an  own  link  file   -­‐f    suppress  request  for  confirmaIon  if      exisIng  files  are  overwriGen   symbolic  links  are  safer  in  respect  to  undesired   deleIon  of  the  target  file   Practical: EidPB WS2013/14

Files  and  Permissions   ● 

● 

● 

Each  user  has  an  ID  and  is  member  of  at  least   one  group   Permissions  can  be  granted  on  owner  (user),   group  or  others.   The  three  groups  are  represented  by  u  (owner),  g   (group)  and  o  (other)  as  well  as    a  (all).    

Practical: EidPB WS2013/14

Files  and  Permissions  2   ● 

● 

● 

the  following  permission  can  be  granted:   r    read    read  permission   w    write    write  permission   x    execute    execute  permission   finer  granularity  can  be  achieved  with  the  use  of   ACL  (Access  Control  Lists)   grant  a  permission  with   + ,  revoke  with   -­‐   grant  exactly  with  “=“   specify  by  leGers  or  octal  numbers   Practical: EidPB WS2013/14

Files  and  Permissions  2a   ● 

there  are  following  special  permissions:   - 

s(et  user  id):  a  file  with  s  is  executed  with  the   effecIve  uid  of  the  owner;  for  a  directory  gives  new   files  the  ownership  of  the  directory  owner  

- 

s(et  group  id)  :  a  file  with  s  is  executed  with  the   effecIve  gid  of  the  owning  group;  for  a  directory   gives  new  files  the  ownership  of  the  directory  owning   group  

- 

t  (sIcky  bit):  a  program  file  is  kept  in  main  memory   (out  of  use);  only  the  owner  of  a  file  or  directory  can   delete  it;  protects  files  in  shared  directories  

Practical: EidPB WS2013/14

Files  and  Permissions  3  

● 

● 

● 

● 

ls  -­‐l  lists  these  permissions:   -rw-r--r--  i.e.  the  owner  has  w/r  perm.  for   this  file,  the  remainder  only  read  permissions   octal  644   grant  write  permission  to  the  group:   chmod g+w  filei  resp.  chmod 664  file   grant  execuIon  permission  to  all:   chmod a+x  file  bzw.  chmod 775  file   grant  special  permissions:  fourth  octal,  sIcky=1,   sgid=2,  suid=4;  u/g+-­‐=s;  o+-­‐=t   Practical: EidPB WS2013/14

Files  and  Permissions  4   ● 

●  ● 

● 

execuIon  permission  on  directories  allows  to   change  to  that  directory   read  permission  is  not  required  to  change  into     deleIng  a  write  protected  file  is  possible  if  you   have  write  permission  to  the  directory!     OpIons:   -­‐R  recursive  change  of  permission  in  the  file      system  tree   Practical: EidPB WS2013/14

Files  and  Permissions  5   ● 

● 

assignment  (hand  over)    of  files:   chown:  change  owner   chown  user.group  fileName   chgrp  group  fileName   OpIons:   -­‐R  recursive  assignment  in  the  file  system  tree  

Practical: EidPB WS2013/14

Files   ● 

● 

● 

Displaying  of  text  files:   cat:  catenate   cat  fileName   The  file  is  displayed  in  one  piece  to  standard  out.   Comfortably  reading  text  file:   less fileName   Allows  paging  and  searching  for  key  words   Determine  the  format  (file  type)  of  an  unknown   file:   file  fileName  

Practical: EidPB WS2013/14

File  EdiIng   ● 

● 

● 

● 

ExtracIng  strings  from  binary  files:   strings  fileName   Output  the  first  n  lines  of  a  file:   head  –n  file   Output  the  last  n  lines  of  a  file:   tail  –n  file   OpIon  -­‐f:  displays  always  the  last  lines  of  a  file,   useful  for  monitoring  log  files   Practical: EidPB WS2013/14

File  EdiIng  II   ● 

● 

● 

●  ● 

Count  lines,  words,  characters:   wc  Dateiname  (word  count)   SorIng  words:   sort  fileName   Spilt  a  file  into  columns:   cut Joining  file  using  a  common  key  column:  join columnwise  merging  of  files:   paste     Practical: EidPB WS2013/14

EdiIng  Files  III   ● 

● 

Remove  mulIply  occurring  lines:   uniq Create  an  empty  file  or  update  the  Ime  stamp:   touch  fileName  

Practical: EidPB WS2013/14

System  Status   ● 

● 

● 

Get  the  seizure  (use  space)  of  a  hard  disk:   df:    disk  free   shows  the  available  space  on  file  systems  and  the   used  space   space  consumpIon  of  files  and  directories:   du:  disk  usage   shows  you  the  space  consumpIon  of  files,  many   opIons  to  improve  the  readability   mounted  file  systems  (drives):   mount Practical: EidPB WS2013/14

System  Status  II   ● 

● 

● 

●  ● 

Display  the  environment  variables:   env:    environment   displays  the  name  and  the  values  of  the   environment  variables  of  your  shell   show  the  sytem  Ime:   date Ime  consumpIon  of  a  program:   time logged-­‐in  user  and  processes:  who terminal  seVngs:  stty if  your  terminal  is  misconfigured:  stty sane

Practical: EidPB WS2013/14

Processes   ● 

● 

●  ● 

process:  AcIvity  which  gets  compuIng  Ime  from   the  operaIng  system   characterized  by:  ownn  ID,  parent-­‐ID,  consumed   cpu  Ime,  used  (virtual)  memory,  status   can  run  in  front  or  as  background   can  run  without  control  terminal  (demon   process)  

Practical: EidPB WS2013/14

Processes  2   ● 

● 

ps:  report  process  status   ps  [opIons]   OpIons:   -­‐u    UID    prozesses  for  user  UID,      w/o  UID  for  all  users   -­‐a    all  processes  having  a  terminal   -­‐x      shows  processes  w/o  control  terminal   -­‐l    verbose  informaIons  (long)   -­‐f    display  as  tree   Practical: EidPB WS2013/14

Prozesses  3  

● 

● 

● 

● 

kill:  kill  processes   kill  -­‐Signal  PID  

OpIons:   Signal    15  (terminate,  controlled  stop),          9  (abort,  immediate  stop)   killall  ProgName   finishes  all  processes  of  program  ProgName,   assumed  you  have  permission  to  do  so   top:  tool  for  online  display  of  acIve  processes   Practical: EidPB WS2013/14

Piping   ● 

● 

Many  tools  use  as  standard  output  device  the   screen  and  as  standard  input  device  the   keyboard   These  standard  channels  can  be  redirected  and   combined  into  a  series  of  connected  processing   steps  

Practical: EidPB WS2013/14

Simple  In/Output  RedirecIon   ● 

● 

● 

Redirect  screen  output  into  a  file:   myProgram  >  myPutputFile   RedirecIon  of  keyboard  input  from  a  file:   myProgram  <  mInputFile   CombinaIon  of  both:   mProgram  >  mOutputFile  <  mInputFile  

Practical: EidPB WS2013/14

Piping   ● 

● 

Pipes  is  a  mechanism  provided  by  the  operaIng   system  to  automaIcally  connect  programs  into  a   series:   Prog1 | Prog2 | Prog3   i.e.  putput  of  program  1  is  directly  used  as  input   for  program  2  and  the  output  of  program  2  is   used  by  program  3   Goal:  many  small  but  specialized  tools  can  be   connected  to  fullfill  complex  tasks   Practical: EidPB WS2013/14

Filtering   ● 

In  a  series  of  connected  programs  normally  there   is  a  processing  at  each  step.  I.e.  certain  data  are   modified  or  filtered  so  this  is  also  called  filtering.  

Practical: EidPB WS2013/14

grep   ● 

grep:  get  regular  expression  

● 

Search  for  paGerns  in  input:  

●  ● 

- 

PaGern  are  regular  expressions  

- 

regular  expression  similar  to  those  in  Perl  

Input:  files  or  standard  input   OpIons:   -­‐c    counts  occurrences  (count)   -­‐r    search  directories  recursively   -­‐e  specify  a  paGern   -­‐v  all  lines  that  do  not  match   ....   Practical: EidPB WS2013/14

find   ●  ● 

●  ● 

find    find  files   searches  the  file  system  starIng  at  a  certain   point  for  (specified)  files   the  desired  files  a  specified  by  condiIons   CondiIons  /  OpIons:   -­‐name,  -­‐type,  -­‐size,  -­‐cIme,  -­‐mIme,  -­‐depth.  -­‐ follow,  etc.    

Practical: EidPB WS2013/14

locate  /  which   ● 

● 

locate:    finding  files   - 

uses  its  own  database  

- 

the  database  is  regularly  updated  by  a  cron  job  

- 

supports  search  paGern,  too  

- 

faster  than  find,  but  depends  from  its  database  

which:  finds  executable  program  in  the  search   path,  programs  are  found  if  they  reside  in  a   directory  listed  in  the  PATH  variable  

Practical: EidPB WS2013/14

Archiving  

● 

● 

(g)zip:    compressing  of  files   (g)unzip:  resp.  for  uncompressing   tar  :  tape  archive   tar -directive [options]   - 

combine  with  g(un)zip  to  compress  archives  

- 

most  important  direcIves:  c,x,t   c  create  a  new  archive   x  extract  an  exisIng  archive   t  list  the  contents  of  an  exisIng  archive  

- 

Ex.:  tar  -­‐czf  archiv.tgz  directory/  to  create  an  compres-­‐ sed  archive  containing  directory  in  the  file  archiv.tgz   tar  -­‐xzf  archiv.tgz  unpack  and  uncompress  archiv.tgz  to   directory  directory  

Practical: EidPB WS2013/14

Examples  for  Piping   ● 

● 

find /usr/local/include -name "*.h" | grep gnu |wc –l counts  the  number  of  header  files  having  “gnu” in  the  filename   “grep gnu *” would  search  the  file  contents   for “gnu” cd ; ls .* | wc –l counts  hidden  files  in  your  home  directory  

Practical: EidPB WS2013/14

Examples  for  Piping  2   ● 

● 

cat /etc/passwd/ | grep system | less searches /etc/passwd for  all  line  containing   “system”  and  display  them  with  the  pager  less   cd; find -type f | grep -c mp3 counts  all  mp3  file  in  your  home  directory  

Practical: EidPB WS2013/14

AddiIonal  Tools   ● 

● 

tee  fileName:  read  from  standard  input  and   writes  to  standard  output  and  to  the  file   fileName   you  can  “tap”  the  pipeline  at  this  point   sort:  input  lines  are  sorted  numerically  or   alphanumerically  (default),  depending  on  the   selected  opIon  (n:  numerically,  r:  reverse)  

Practical: EidPB WS2013/14

More  Examples   ●  ●  ● 

● 

cat /etc/passwd | cut -f1,5 -d:     find . -name *.pl | wc cd; du -h --max-depth=3 | sort -rn | less cd; grep -lvir gnu * | wc –l searches  files  in  your  home  for  the  occurence  of “gnu” and  counts  the  lines  

Practical: EidPB WS2013/14

Shell  Programming  in  10  seconds   ● 

So  far  covered:  Single  instrucIons,  instrucIon   sequences  (piping)  

● 

SIll  missing:  Variables,  control  structures  

● 

These  are  provided  by  the  shell  

● 

several  shell  flavors:  sh,  ksh,  zsh,  csh,  tcsh,  bash  

● 

determine  your  shell:  echo $SHELL!

Practical: EidPB WS2013/14

Shell  Programming  II   ● 

● 

make  a  script  executable:   #!/bin/bash  (she-­‐bang  line)  and  set  x-­‐bit  or   bash  yourScriptfile! How  to  define  a  variable:     VariableName=Value  (no  whitespace   around  =)   VariableName:  [a-­‐zA-­‐Z_]  [a-­‐zA-­‐Z0-­‐9_]*   Value:  unquoted  string  without  whitespace  or    quoted  string  if  it  contains  whitespace  or $VariableName! Practical: EidPB WS2013/14

Shell  Programming  III   ● 

simple  output:  echo!

● 

formaGed  output:  printf!

● 

● 

enable  trace  mode  with  set  –x,  even  more   verbose  with  –v   special  variables:   - 

$#  number  of  arguments  (command  line  &  funcIon)  

- 

$*,[email protected]  list  of  all  command  line  arguments  

- 

“$*”  one  string,  “[email protected]”  list  of  protected  strings  

Practical: EidPB WS2013/14

Shell  Programming  IV   ● 

●  ● 

shift  return  first  posiIonal  parameter   (deleIng)   arithmeIcs  within  $(( )),  operators  like  in  C   Exit  status:     Exits  status  0  indicates  success!   $?  holds  the  exit  status  of  the  last  command   exit  n  returns  immediately  exit  status  n!

Practical: EidPB WS2013/14

Shell  Programming  V   ●  ● 

if-­‐elif-­‐else-­‐fi   if  pipeline  [pipeline...]   then   command-­‐if-­‐true-­‐1   [elif  pipeline]  [pipeline...]   then   command-­‐if-­‐true-­‐2   [else  commands-­‐in-­‐all-­‐other-­‐cases]   fi     Practical: EidPB WS2013/14  

Shell  Programming  VI   ● 

test  [expression] or   [  [expression]  ]!

● 

-d file  is  this  a  directory?!

● 

-e file  does  file  exists?!

● 

-h file  is  this  a  symbolic  link?  

● 

-s file  is  not  empty?  

● 

-x file  is  executable?  

● 

...!

Practical: EidPB WS2013/14

Shell  Programming  VII   ●  ● 

● 

Loops:  for  and  while  loops   for  i  in  list   do   commands...   done! while  condiIon  /  until  condiIon   do
 commands   done! Practical: EidPB WS2013/14

Vi  I   ● 

very  powerful  text  editor  

● 

available  on  (nearly)  all  UNIX  machines  

● 

nowadays:  vim  (vi  improved)  

● 

now  graphical  user  interface  needed  

● 

needs  liGle  resource  

● 

sIll  works  when  systems  responds  slowly  

Practical: EidPB WS2013/14

Vi  –  Sources  for  Help   ● 

hGp://www.vim.org  

● 

man  page  

● 

Help  system:  “:help”  

● 

vimtutor  

 

Practical: EidPB WS2013/14

Vi  –  Modes  and  Basics   ● 

command  mode  (normal  mode)  

● 

command  line  mode  (ex  mode)  

● 

insert  mode  

● 

visual  mode  (vim  only)  

● 

command  mode  -­‐>  insert  mode:  insert/edit   command  

● 

insert  mode  -­‐>  command  mode:  ESCAPE,  CTRL-­‐C  

● 

command  line  mode  command  mode:  RETURN  

Practical: EidPB WS2013/14

Vi  –  navigaIon   ● 

●  ● 

● 

navigaIon:  cursor  keys  or  h,j,k,l  (leW,  down,  up   right)   gg:  start  of  document,  G:  end  of  document   0  begin  of  line,  w  next  word,  e  end  of  word,  $   end  of  line   preceded  by  repeater  number  like:  5w,  2j  

Practical: EidPB WS2013/14

Vi  –  edit   ● 

Insert:  i  –  aWer  cursor,  I  –  start  of  line  

● 

Erase:  x  –  erase  at  cursor,  X  –  erase  before  cursor  

● 

● 

Append:  a  –  append  at  cursor,  A  –  append  at  end   of  line   o:  append  in  new  line  aWer,  O:  append  in  new   line  before  cursor  

● 

r:  replace  single  character,  c+posiIoner:  change    

● 

u:  undo,  “.”:  repeat  last  command   Practical: EidPB WS2013/14

More  complex  commands   ● 

full  command  syntax:   [“register][counter]command [cursor positioner]

● 

save  and  exit:  ZZ  or  :wq  

● 

exit  without  save:  “:q!”  (force  quit)  

Practical: EidPB WS2013/14

Make  Life  easier   ● 

:set showcmd to  show  your  typed   commands!

● 

:set nu display  line  numers!

● 

:syntax on to  enable  syntax  highlighIng!

● 

:set hls / nohls to  enable  search   highlighIng!

Practical: EidPB WS2013/14

Version  Control   ● 

What  is  version  control  

● 

Why  control  is  necessary  and  useful  

● 

Version  control  tools:   cvs,  svn,  git     hGp://cvs.nongnu.org/   hGp://subversion.apache.org/   hGp://git-­‐scm.com/documentaIon   Practical: EidPB WS2013/14

Version  Control  II   ● 

● 

● 

Problem:   Two  or  more  people  ediIng  the  same  file  at  the   same  Ime  can  cause  problems,  called  lost   update  phenomenon   Version  control  helps  to  resolve  this  by  several   strategies:  check  out  with  locking  (outdated)  or   copy,  modify,  merge   prevents  lost  update  phenomena,  allows  rollback   to  unspoiled  predecessor  versions   Practical: EidPB WS2013/14

Version  Control  III   ● 

CVS  and  SVN  using  a  centralized  repositotory  

● 

git  is  fully  decentralized  

● 

typical  funcIons:  check-­‐out,  update,  check-­‐in,   merge,  commit  

Practical: EidPB WS2013/14

Thank  you  for  your  aGenIon  

Practical: EidPB WS2013/14