Basic commands that will be used while working on Hadoop in Ubuntu envronment. Slides will be updated.

1. LINUX COMANDS FOR
HADOOP
Prepared By PM VENKATESH BABU

2. UNIX FOLDER STRUCTURE
• Bin -> All the executable files are present in this folder
• Sys -> The folder where all the System files are located
• Log -> All the log files are present in this folder
• Temp -> Processing data is present in this folder
• Usr -> Application Installation is present in this folder
• Opt -> Application Installation is present in this folder
• Etc -> Client Configuration data is available in this folder
• Proc -> System calls details present in this folder
• Home -> Application installation details and user working directory info
• Lib -> System Library files are available in this folder

3. COMMONLY USED HADOOP COMMANDS
SL NO COMMAND DESCRIPTION
1 Pwd Present working directory
2 Ls Lists and displays all files in the current folder
3 Ls –ltr Lists and displays the files along with the details of permissions rw-r-1
4 Ls –a Lists and displays the hidden files such as .ssh files and .vim files
5 Mkdir To create a folder (eg) mkdir hadoop -> creates a folder called hadoop
6 Cd .. Takes the user to the root directory
7 Cd ~ Takes the user to the root/home directory
8 Cat > file name.txt Creates a file name and allows the user to insert/write content to the file
9 Ctrl+d or Ctrl+c Is used to save the text file and its contents saved
10 Cat filename.txt Used to view the contents of the text file created

4. COMMONLY USED HADOOP COMMANDS….Cont
SL NO COMMAND DESCRIPTION
11 Cp filename.txt This will copy the filename.txt file from one folder to another folder called Hadoop
with the same file name. This will retain the copy of the original file in the origin
folder.
Eg) Cp filename.txt /home/Ubuntu/Hadoop
12 Cp filename.txt Cp filename.txt /home/Ubuntu/Hadoop/newfilename.txt ( This will copy the
filename.txt file from one folder to another folder called Hadoop with the changed
name called newfilename.txt
13 Mv filename.txt This will move a file from one folder to another folder. Mv command is like ctrl+x
and paste, which will be like cut a file from one folder and paste it to the destination
folder
14 Rm filename.txt This will completely remove the file specified from the folder.
Eg) Rm filename.txt /home/Ubuntu/Hadoop/
15 Cp –r
-r does the Recursive action
Will copy one folder to another folder. Eg. Cp /home/Ubuntu/hadoop
home/Ubuntu/bigdata (will copy the entire folder called Hadoop along with the
contents to another folder called bigdata.
16 Rm -r Will delete the folder called Hadoop
Eg) rm –r /home/Ubuntu/bigdata/Hadoop

5. COMMONLY USED HADOOP COMMANDS….Cont
SL NO COMMAND DESCRIPTION
21 Ps -es This command will display all the processes currently running
22 Ps –ef |grep java This command is used to view if a specific process is running or not. Here we are
using the | ‘pipe’ operator to filter for a specific process. In this command we are
looking if the process ‘Java’ is running or not.
23 netstat This command is used to view the network statistics and port numbers
24 Netstat |grep
<portnumber>
This command is used to view the network statistics of a specific portnumber.
|grep is used for pattern searches. Used for logical searches
Eg . Netstat |grep 11868 ( will display the network stats of port number 1186 )
25 Tail <filename.txt> This command is used to find the last line in a large text file
26 Jps This command stands for Java Virtual machine status tool. This command gives the
status of the daemons which run Hadoop cluster
27 Start-all.sh Starts the Hadoop and its daemons.
Namenode
SecondaryNamenode
Datanode
JobTracker

6. COMMONLY USED HADOOP COMMANDS….Cont
SL NO COMMAND DESCRIPTION
31 Cat <filename> The cat command will display the contents of the filename given.
Eg. Cat sales_tran.csv will display the contents of the specified file.
32 cat<filename> | grep
<specific word to search >
The ‘Grep’ command will be used to search for a specific word in a file.
Eg. Cat myfile.txt | grep is (This will search for the word ‘is’ in the text file called
myfile and display all the sentences
33 Du <filename> The ‘du’ command will specify the size of the filename given.
Eg . Du myfile.txt The output will be something like “29 myfile.txt” This indicates
that the file size of myfile.txt is 29 bytes.