Pandas are a general-purpose high-performance open-source python library which is used for the analysis of different data type. Pandas are using very powerful underlying data structure to process heavy data.

Pandas provide a variety of functionality.

1. Loading data from different data sources
2. Indexing and slicing of data
3. Transformation of data
4. Insert new data or delete existing data.

Before going next it is recommended that you should have knowledge of numpy. You can read it from our tutorial of numpy.

Data Structures in pandas

Pandas provide mainly 3 types of data Structure.  Pandas data structure is built on numpy so we can understand how fast they are.
1. Series
2. Dataframe
3. Panel

Series

It is a one-dimensional homogeneous array. Series data looks like one-dimensional array like below:
Numeric data: 1,5,8,6,7,9,11

String data: data1,data2,data3,data4,data5,data6,data7,data8

Dataframe

They are a two-dimensional tabular data structure which is capable of storing a heterogeneous data type with different columns.

Example for Dataframe is

Panels

They are Three dimensional labeled array. As panels are 3 dimensional, it is difficult to show an example here. We can say that panels are the containers for the data frame.

How to install panda/numpy?
1) pip install panda
2) pip install numpy

[root]# python
Python 2.7.5 (default, Oct 30 2018, 23:45:53) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> import panda
>>> 

Example1: Creating a simple series using numpy array.

#!/usr/bin/python
#import the pandas and numpy library
import pandas as pd
import numpy as np
# Creating a numpy array with some numeric data
numericdata = np.array([100,200,300,400])

#Assigning value to series
series = pd.Series(numericdata)

print(series)


===========Output===========
0    100
1    200
2    300
3    400
dtype: int64

In the above example, we can see the index is auto assign by pandas, But pandas provide flexibility to assign our own indexes.

#!/usr/bin/python
#import the pandas and numpy library
import pandas as pd
import numpy as np
# Creating a numpy array with some numeric data
numericdata = np.array([100,200,300,400])

#Assigning value and index to series
series = pd.Series(numericdata, index=[7,8,9,22])

print(series)

========output==========
7     100
8     200
9     300
22    400
dtype: int64

in the above example indexes are assigned manually.

Attributes of series.

Series provides the following attributes.
1. Axes
2. Size
3. Value
4. Head
5. Tail
6. empty

Let’s see an example of the above attributes:

#!/usr/bin/python
#import the pandas and numpy library
import pandas as pd
import numpy as np
# Creating a numpy array with some numeric data
numericdata = np.array([100,200,304,400])

#Assigning value to series
series = pd.Series(numericdata)

print("axis of series {0}".format(series.axes))

print("Is series empty {0}".format(series.empty))

print("Size of series {0}".format(series.size))

print("values in series {0}".format(series.values))

# head provides first n elements of series
print("Head of series {0}".format(series.head(2)))

# Tail provides last n elements of series
print("Tail of series {0}".format(series.tail(2)))


=======output=======
axis of series [RangeIndex(start=0, stop=4, step=1)]
Is series empty False
Size of series 4
values in series [100 200 304 400]
Head of series 0    100
1    200
dtype: int64
Tail of series 2    304
3    400
dtype: int64

Attributes of Data frames

Data frames provide the following attributes
1. Transpose
2. Empty
3. Shape
4. Size
5. Values
6. Head
7. Tail

Let’s take an example of all the above:

#!/usr/bin/python
#import the pandas and numpy library
import pandas as pd
import numpy as np
# Creating a numpy array with some numeric data
data = {"ID":[4,3,2,1],
"Name":["Name1","name2","name3","Name4"],
"Age":[10,20,30,40]
}

# Create a dataframe
df =pd.DataFrame(data)
print("Actual data")
print(df)

#Transpose of data
print("Transpose of data")
print(df.T)

# Axes are
print("Axes of data")
print(df.axes)

# empty tell us is data frame has data or not
print("Is empty ?")
print(df.empty)

# shape provides rows and columns contained in the data frame
print("Shape data")
print(df.shape)

# Head and tail get data from the top or bottom of the data frame
print("Top 2 records from data frame")
print(df.head(2))

print("last 2 records from dataframe")
print(df.tail(2))


=======output======
Actual data
   Age  ID   Name
0   10   4  Name1
1   20   3  name2
2   30   2  name3
3   40   1  Name4
Transpose of data
          0      1      2      3
Age      10     20     30     40
ID        4      3      2      1
Name  Name1  name2  name3  Name4
Axes of data
[RangeIndex(start=0, stop=4, step=1), Index([u'Age', u'ID', u'Name'], dtype='object')]
Is empty ?
False
Shape data
(4, 3)
Top 2 records from data frame
   Age  ID   Name
0   10   4  Name1
1   20   3  name2
last 2 records from dataframe
   Age  ID   Name
2   30   2  name3
3   40   1  Name4

Pandas provide some math function some are listed below:

1. sum()
2. count()
3. mean()
4. min()
5. max()
6. Prod

Working with text data

Pandas provide very good functionality to deal with text data. When we process any text data we need to split, sort, count and many such kinds of function which make our task easy.

Pandas provide these kinds of functions some of them are listed below:
1. lower()
2. upper()
3. len()
4. split()
5. replace()
6. count()
7. find()
8. Islower
9. Isupper
10. isnumeric()

Example of above functions:

!/usr/bin/python
#import the pandas and numpy library
import pandas as pd

s = pd.Series(['Raj kumar', 'Amit singh', 'John', 'katie', 20, '897456','Steve','smith'])

print("All text is in lower case")
print(s.str.lower())

print("All text is in upper case")
print(s.str.upper())

print("Length of each text in string")
print(s.str.len())

print("split string by space")
print(s.str.split(' '))

print("check string contains a specific data or not it return true and false")
print(s.str.contains(' '))

print("count a specific word and character in string ")
print(s.str.count('R'))

=======output==========
All text is in lower case
0     raj kumar
1    amit singh
2          john
3         katie
4           NaN
5        897456
6         steve
7         smith
dtype: object
All text is in upper case
0     RAJ KUMAR
1    AMIT SINGH
2          JOHN
3         KATIE
4           NaN
5        897456
6         STEVE
7         SMITH
dtype: object
Length of each text in string
0     9.0
1    10.0
2     4.0
3     5.0
4     NaN
5     6.0
6     5.0
7     5.0
dtype: float64
split string by space
0     [Raj, kumar]
1    [Amit, singh]
2           [John]
3          [katie]
4              NaN
5         [897456]
6          [Steve]
7          [smith]
dtype: object
check string contains a specific data or not it return true and false
0     True
1     True
2    False
3    False
4      NaN
5    False
6    False
7    False
dtype: object
count a specific word and character in string 
0    1.0
1    0.0
2    0.0
3    0.0
4    NaN
5    0.0
6    0.0
7    0.0
dtype: float64

SQL operations on pandas

We can perform many SQL operations on pandas some of the examples are listed below.

An example is mainly focused on Select, where, limit statement How these SQL Query can be used with a data frame with some tricks.

An example is below:

CSV File: (http://insight.dev.schoolwires.com/HelpAssets/C2Assets/C2Files/C2ImportGroupsSample.csv)

GroupName,Groupcode ,GroupOwner,GroupCategoryID 
System Administrators,sysadmin,13456,100
Independence High Teachers,HS Teachers,,101
John Glenn Middle Teachers,MS Teachers,13458,102
Liberty Elementary Teachers,Elem Teachers,13559,103
1st Grade Teachers,1stgrade,,104
2nd Grade Teachers,2nsgrade,13561,105
3rd Grade Teachers,3rdgrade,13562,106
Guidance Department,guidance,,107
Independence Math Teachers,HS Math,13660,108
Independence English Teachers,HS English,13661,109
John Glenn 8th Grade Teachers,8thgrade,,110
John Glenn 7th Grade Teachers,7thgrade,13452,111
Elementary Parents,Elem Parents,,112
Middle School Parents,MS Parents,18001,113
High School Parents,HS Parents,18002,114

#!/usr/bin/python
#import the pandas and numpy library
import pandas as pd

# Read CSV
url = 'http://insight.dev.schoolwires.com/HelpAssets/C2Assets/C2Files/C2ImportGroupsSample.csv'

csvdata=pd.read_csv(url)
print(csvdata.head())

# select some specfic columns in pandas like
print("Select some specific columns")
print(csvdata[["GroupName","GroupOwner"]].head())

# Filter data like as Sql WHERE
print("Filer record")
print(csvdata[csvdata["GroupName"]=="System Administrators"])

# select top n rows like as SQL
print("Select only 2 rows like as SQL")
print(csvdata.head(2))

========output==========

                    GroupName     Groupcode   GroupOwner  GroupCategoryID 
0        System Administrators       sysadmin     13456.0               100
1   Independence High Teachers    HS Teachers         NaN               101
2   John Glenn Middle Teachers    MS Teachers     13458.0               102
3  Liberty Elementary Teachers  Elem Teachers     13559.0               103
4           1st Grade Teachers       1stgrade         NaN               104
Select some specific columns
                     GroupName  GroupOwner
0        System Administrators     13456.0
1   Independence High Teachers         NaN
2   John Glenn Middle Teachers     13458.0
3  Liberty Elementary Teachers     13559.0
4           1st Grade Teachers         NaN
Filer record
               GroupName Groupcode   GroupOwner  GroupCategoryID 
0  System Administrators   sysadmin     13456.0               100
Select only 2 rows like as SQL
                    GroupName   Groupcode   GroupOwner  GroupCategoryID 
0       System Administrators     sysadmin     13456.0               100
1  Independence High Teachers  HS Teachers         NaN               101

There is a variety of other functionality provided by pandas.

Like merging, grouping, missing data, Date functionality, time delta and many more.

The above article is mainly focused on the basics of pandas and how to start working with pandas.

For detail information, you can read the official document of pandas (https://pandas.pydata.org/pandas-docs/stable/)