AdSense

Showing posts with label Logistic Regression. Show all posts
Showing posts with label Logistic Regression. Show all posts

Monday, June 15, 2020

Multinomial Logistic Regression with L2 (or L1) Regularitation of Standardized Training Data (plus Test Data) in Python

Multinomial Logistic Regression with L2 (or L1) Regularitation of Standardized Training Data (plus Test Data) in Python

0_MacOS_Python_setup.txt
# Install on Terminal of MacOS

#pip3 install -U scikit-learn

#pip3 install -U matplotlib

#pip3 install -U numpy

#pip3 install -U pandas

#pip3 install -U seaborn


1_MacOS_Terminal.txt
########## Run Terminal on MacOS and execute
### TO UPDATE
cd "YOUR_WORKING_DIRECTORY"


python3 mlgstcreg.py l2 1
#python3 mlgstcreg.py l2 10
#python3 mlgstcreg.py l2 100
#python3 mlgstcreg.py l2 1000




Data files

X.csv
4.494746752403546,2.565907751154284
3.3841445311320393,5.169101554140335
4.641125838188323,0.617920368071555
3.3436279255182804,1.7859485832624662
1.854026499900972,2.580673970131322
2.203708371908368,4.056653316946923
3.076269872380446,2.172074458526128
2.6277174035873467,2.471886759188408
4.11294688851731,1.7098624009494876
2.4427445896156783,0.7921262219144254
-1.6104728221529445,2.9243562822910114
3.2224973962333587,0.9504201626223048
5.20991777250232,-0.05678366166746862
2.064712315761789,1.7352820606362547
4.167677153069308,4.077987100384831
2.2191287508753765,2.5348025640025753
0.7444813552199567,-0.8012692296630051
1.5079779199085814,2.2211108325699005
3.7398937663463703,3.7004218892750256
1.4522371617508665,1.5724793501936696
0.5171221759355953,-0.008208625571572536
-0.4130304446548192,4.758813021080532
1.279242972473774,1.3804693813343385
0.2282798109393096,3.0995374058317706
-0.28239622430120104,1.6991398104583528
0.7336190445083068,2.5471627597885957
1.2776124467202452,0.33033395303991564
1.960144310465579,2.6057527405007535
2.094069558025663,2.427759860020327
1.1029330922034843,1.4870065234299592
1.0489973145957656,1.4915150425552421
0.8500374997394751,-0.4413322687061778
2.2509184566947953,1.4317959509113165
-0.30544861163758075,2.654472942190161
0.7168863479702925,2.073461883237738
3.0310897612296968,2.1824093817074885
3.6113559010583245,0.2536925776873813
2.568997005660739,1.0315322817422565
0.7685068615511554,1.181382953515044
1.5593981836737902,2.079429788716632
0.35222929296732475,3.2739610351955215
2.65854613775456,-0.172576656243288
4.104706436697249,4.681192185498803
3.6670460565823193,1.745547856986421
0.4857271207132452,3.491219933093745
1.4298216935543513,3.728798397790962
2.294545298699255,3.3811761709382586
2.503978192058076,2.999245357265979
2.0148492717085826,4.52560227312357
2.1794808027306107,2.5684988097132244
9.26171268561018,4.665612829739954
3.7994554323686414,8.679044351219694
3.9680906587940683,10.366450644211836
5.283590910349264,5.705370290222793
9.330633289873484,9.564326840121662
9.234707005589334,8.569315382059349
4.508313356701943,10.308329544524192
5.53580454491248,8.389895248377957
7.640688535753728,6.731514643080693
7.063616669319316,8.597308810232208
6.6519881452276515,5.095781972826784
6.5165636704814665,9.29736776350474
4.7969731776409015,6.74082537358526
5.246291939324073,10.203018734320477
7.1644486768077105,7.705744602514913
4.66646624149673,7.934006997257856
4.8320215705383225,7.0551321441644586
4.898682286454355,8.171616834778561
6.998684589554707,6.639215972185542
6.685903746468299,5.106761931080872
3.4170660822610364,7.761048751014304
6.288686962260316,8.099906713100598
10.12772783185664,8.635886458173985
4.4189455272526255,8.934728963864012
3.7207815069376498,6.200512012469322
5.881802072387676,9.967596644675044
4.710046809030184,5.568566461709612
5.829475225561735,5.850821898302097
7.9513906587206815,5.129503758820057
4.012525993931379,6.241673437930561
5.137382491582886,10.342047552043745
7.644445075358459,7.151643198344877
3.8774834199866217,8.46247957515756
4.267576199811119,4.324377974317278
8.05772796125969,7.548960706937425
7.594974269386494,7.552052488674493
7.484074153295792,5.872390595376473
4.2086388506648875,8.180556335824212
4.608453642359058,5.805664750588825
5.210994559408913,7.030274791504974
5.386864560138938,4.618514501867448
4.885220225607101,3.1489527746384947
7.082932639669101,4.22515474383443
4.087151945077339,7.090352567660865
4.719039314852367,9.672579676064801
3.760706145442498,7.462545673918223
5.931960162965485,4.976802713921405
6.906341762455969,6.702873038471027
7.336780447503662,8.426351034909207
9.746834572913645,9.314934314159679



y.csv
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0




Python files


mlgstcreg.py
#################### Multinomial Logistic Regression with L2 (or L1) Regularitation of Standardized Training Data (plus Test Data) in Python ####################
#
#Run this script on Terminal of MacOS as follows:
#python3 mlgstcreg.py l2 1
##python3 mlgstcreg.py (penalty/regularization of LogisticRegression()) (parameter c, 1 by default)
#
#Reference
#http://ailaby.com/logistic_reg/



##########Overview
'''
We have to avoid overfitting, i.e., letting a model learn outliers and noises of training data too well.
Reasons of overfitting are
(1) variables are too many,
(2) parameters are too big/impactful,  and/or
(3) data is not enough.

We can use L1 regularization (e.g., linear Lasso Regression) to eliminate redundant / unnecessary parameters as in (1).
Also, L2 regularization (e.g., linear Ridge Regression) is to avoid (2).
'''



########## import
'''
from sklearn import datasets
import matplotlib.pyplot as plt
import seaborn as sns
'''

#from sklearn.cross_validation import train_test_split
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
import sys
import math


########## arguments
pnlty = str(sys.argv[1])    #l2 or l1
c = int(sys.argv[2])    #C = 1 by default. The larger the C, the weaker regularization is going to be.


########## Multinomial Logistic Regression with Regularitation

''' 
##### generate dataset with TWO classes as the mixture of training data and test data
np.random.seed(seed=0)

X_0 = np.random.multivariate_normal( [2,2],  [[2,0],[0,2]],  50 )

y_0 = np.zeros(len(X_0))

X_1 = np.random.multivariate_normal( [6,7],  [[3,0],[0,3]],  50 )
y_1 = np.ones(len(X_1))
 
X = np.vstack((X_0, X_1))
#print(X)
#print(type(X))
#<class 'numpy.ndarray'>

y = np.append(y_0, y_1)
#print(y)
#print(type(y))
#<class 'numpy.ndarray'>


##### save dataset
pd.DataFrame(data=X).to_csv("X.csv", header=False, index=False)
pd.DataFrame(data=y).to_csv("y.csv", header=False, index=False)
'''



##### load raw dataset
X = pd.read_csv('X.csv', header=None).values
y = pd.read_csv('y.csv', header=None).values.ravel()

#print(type(X))
#<class 'numpy.ndarray'>

#print(type(y))
#<class 'numpy.ndarray'> 



##### plot raw data
plt.scatter(X[y==0, 0], X[y==0, 1], c='blue', marker='*', label='raw data 0')
plt.scatter(X[y==1, 0], X[y==1, 1], c='red', marker='*', label='raw data 1')

plt.legend(loc='upper left')
plt.title('Raw Data')
plt.xlabel('X1: Raw Data')
plt.ylabel('X2: Raw Data')

plt.savefig('Figure_1_Raw_Data.png')
plt.show()
plt.close()


##### splitting Training Data and Test Data
#X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=3)
#random_state=3 is to fix results. You can change this number to, say, 1, 2, or any other integer you like.



##### Standardization of Training and Test Data (Average=0, SD=1)
sc = StandardScaler()
X_train_std = sc.fit_transform(X_train)
X_test_std = sc.transform(X_test)


#####  max and min of X and y
xmax = max(
    max(X_train_std[y_train==0, 0]),
    max(X_train_std[y_train==1, 0]),
    max(X_test_std[y_test==0, 0]), 
    max(X_test_std[y_test==1, 0])
)
#
xmin = min(
    min(X_train_std[y_train==0, 0]),
    min(X_train_std[y_train==1, 0]),
    min(X_test_std[y_test==0, 0]), 
    min(X_test_std[y_test==1, 0])
)
#
#
ymax = max(
    max(X_train_std[y_train==0, 1]),
    max(X_train_std[y_train==1, 1]),
    max(X_test_std[y_test==0, 1]),
    max(X_test_std[y_test==1, 1])
)
#
ymin = min(
    min(X_train_std[y_train==0, 1]),
    min(X_train_std[y_train==1, 1]),
    min(X_test_std[y_test==0, 1]),
    min(X_test_std[y_test==1, 1])
)


##### plot trainging and test Data
plt.xlim([math.floor(xmin), math.ceil(xmax)])
plt.ylim([math.floor(ymin), math.ceil(ymax)])

plt.scatter(X_train_std[y_train==0, 0], X_train_std[y_train==0, 1], c='blue', marker='x', label='train 0')
plt.scatter(X_train_std[y_train==1, 0], X_train_std[y_train==1, 1], c='red', marker='x', label='train 1')
plt.scatter(X_test_std[y_test==0, 0], X_test_std[y_test==0, 1], c='blue', marker='o', s=60, label='test 0')
plt.scatter(X_test_std[y_test==1, 0], X_test_std[y_test==1, 1], c='red', marker='o', s=60, label='test 1')
 
plt.legend(loc='upper left')
plt.title('Training Data and Test Data')
plt.xlabel('X1: Training Data and Test Data')
plt.ylabel('X2: Training Data and Test Data')

plt.savefig('Figure_2_Traing_Data_and_Test_Data.png')
plt.show()
plt.close()

 
########## Logistic Regression built by Standardized Training Data

#lr = LogisticRegression()
lr = LogisticRegression(C=c, penalty=pnlty)

lr.fit(X_train_std, y_train)


#check predict and score
#print(lr.predict(X_test_std))
#[1. 1. 0. 1. 1. 0. 1. 1. 0. 0. 1. 0. 1. 1. 0. 1. 1. 0. 1. 0.]
#
#print(y_test)
#[1. 1. 0. 1. 1. 1. 1. 0. 0. 0. 1. 0. 1. 1. 0. 1. 1. 0. 1. 0.]
#
#print(lr.predict(X_test_std))
#[1. 1. 0. 1. 1. 0. 1. 1. 0. 0. 1. 0. 1. 1. 0. 1. 1. 0. 1. 0.]
#
#[T  T  T  T  T  F  T  F  T  T  T  T  T  T  T  T  T  T  T  T ]
#Number of T (  correct answers) = 18
#Number of F (incorrect answers) = 2 
#18/(18+2) = 0.9

#print(lr.score(X_test_std, y_test))
#0.9


'''
A result of Logistic Regression is weights w0, w1, and w2 of a boundary line below:
w0+w1x+w2y=0

w0 is stored in intercept_ while w1 and w2 are included in coef_.

'''

#print (lr.intercept_)
#[-0.09150732]
 
#print (lr.coef_)
#[[1.99471124 2.32334603]]
 
w_0 = lr.intercept_[0]
w_1 = lr.coef_[0,0]
w_2 = lr.coef_[0,1]


# a boundary line 
#   w_0 + (w_1 * x) + (w_2 * y) = 0
#   y = ((-w_1 * x) - w_0) / w_2
#
print("y = ((-w_1 * x) - w_0) / w_2")
print("y = (-w_1/w_2) * x - w_0/w_2")
print("y: x2")
print("x: x1")
print("w_0 = ", w_0)
print("w_1 = ", w_1)
print("w_2 = ", w_2)
print("(-w_1/w_2) = ", (-w_1/w_2))
print("(-w_0/w_2) = ", (-w_0/w_2))

'''
y = ((-w_1 * x) - w_0) / w_2
y = (-w_1/w_2) * x - w_0/w_2
y: x2
x: x1
w_0 =  -0.09150731939635004
w_1 =  1.9947112354879184
w_2 =  2.3233460327656656
(-w_1/w_2) =  -0.8585510756283915
(-w_0/w_2) =  0.03938600540162393
'''


# plotting a boundary line
#plt.plot([-2,2], map(lambda x: (-w_1 * x - w_0)/w_2, [-2,2]))
#plt.plot([-2,2], list(map(lambda x: (-w_1 * x - w_0)/w_2, [-2,2])))
plt.plot([math.floor(xmin) - 1, math.ceil(xmax) + 1], list(map(lambda x: (-w_1 * x - w_0)/w_2, [math.floor(xmin) - 1, math.ceil(xmax) + 1])))


# plotting Training Data and Test Data
plt.xlim([math.floor(xmin), math.ceil(xmax)])
plt.ylim([math.floor(ymin), math.ceil(ymax)])

plt.scatter(X_train_std[y_train==0, 0], X_train_std[y_train==0, 1], c='blue', marker='x', label='train 0')
plt.scatter(X_train_std[y_train==1, 0], X_train_std[y_train==1, 1], c='red', marker='x', label='train 1')
plt.scatter(X_test_std[y_test==0, 0], X_test_std[y_test==0, 1], c='blue', marker='o', s=60, label='test 0')
plt.scatter(X_test_std[y_test==1, 0], X_test_std[y_test==1, 1], c='red', marker='o', s=60, label='test 1')

plt.legend(loc='upper left')
plt.title('Training Data and Test Data plus a Boundary Line')
plt.xlabel('X1: Training Data and Test Data')
plt.ylabel('X2: Training Data and Test Data')

#plt.text(-2,-2, 'Boundary Line: x2 = (-w_1/w_2) * x1 - w_0/w_2 = ' + str(-w_1/w_2) + ' * x1 + ' + str(-w_0/w_2), size=10)
plt.text(math.floor(xmin) + (math.ceil(xmax) - math.floor(xmin)) * 0.05, math.floor(ymin) + (math.ceil(ymax) - math.floor(ymin)) * 0.20, 'Regularization: ' + str(pnlty), size=9)
plt.text(math.floor(xmin) + (math.ceil(xmax) - math.floor(xmin)) * 0.05, math.floor(ymin) + (math.ceil(ymax) - math.floor(ymin)) * 0.15, 'c = ' + str(c), size=9)
plt.text(math.floor(xmin) + (math.ceil(xmax) - math.floor(xmin)) * 0.05, math.floor(ymin) + (math.ceil(ymax) - math.floor(ymin)) * 0.10, 'Test score = ' + str(round(lr.score(X_test_std, y_test),3)*100) + '%', size=9)
plt.text(math.floor(xmin) + (math.ceil(xmax) - math.floor(xmin)) * 0.05, math.floor(ymin) + (math.ceil(ymax) - math.floor(ymin)) * 0.05, 'Boundary Line: x2 = ' + str(-w_1/w_2) + ' * x1 + ' + str(-w_0/w_2), size=9)

plt.savefig('Figure_3_Traing_Data_and_Test_Data_plus_Boundary_Line.png')
plt.show()
plt.close()


##### Confusion Matrix

cm = confusion_matrix(y_test, lr.predict(X_test_std))

print ("Confusion Matrix : \n", cm)
'''
Confusion Matrix :
Confusion Matrix : 
 [[ 7  1]
 [ 0 12]]
'''
'''
Out of 20 :
TruePostive + TrueNegative = 12 + 7
FalsePositive + FalseNegative = 1 + 0

 [[TrueNegative(TN)  FalsePositive(FP)]
 [ FalseNegative(FN) TruePositive(TP)]]


                            Predicted Labels:
                            Negative              Positive
Actual Results: Negative    TrueNegative(TN)     FalsePositive(FP)
                Positive    FalseNegative(FN)    TruePositive(TP)
'''
#use 1, 2 as there are two independent variables x1 and x2
class_names=[1,2]
#fig, ax = plt.subplots()
fig, ax = plt.subplots(figsize=(6,5))
tick_marks = np.arange(len(class_names))
plt.xticks(tick_marks, class_names)
plt.yticks(tick_marks, class_names)
#sns.heatmap(pd.DataFrame(cm), annot=True, cmap="Blues" ,fmt='g')
sns.heatmap(pd.DataFrame(cm), annot=True, cmap="coolwarm", fmt='g')
ax.xaxis.set_label_position("top")
#plt.tight_layout()
plt.tight_layout(pad=3.00)
plt.title('Confusion Matrix (Test Data)')
plt.xlabel('Predicted Labels')
plt.ylabel('Actual Results')
#print("Accuracy:",metrics.accuracy_score(y_test, y_pred))
plt.savefig("Figure_4_Confusion_Matrix_Test_Data.png")
plt.show()
plt.close()


########## Logistic Regression built by Standardized Training Data (+ Regularization parameter C)

'''
The larger the C, the weaker regularization is going to be.

C is 1.0 by default. We try 1, 10, 100, 1000 here.
'''


fig, axs = plt.subplots(2, 2, sharex=True, sharey=True)

plt.xlim([math.floor(xmin), math.ceil(xmax)])
plt.ylim([math.floor(ymin), math.ceil(ymax)])

plt.subplots_adjust(wspace=0.1, hspace=0.6)
c_params = [1.0, 10.0, 100.0, 1000.0]

#print(c_params)
#[1.0, 10.0, 100.0, 1000.0]
#
#print(type(c_params))
#<class 'list'>

#print(enumerate(c_params))
#<enumerate object at 0x127bec500>

for i, c in enumerate(c_params):
    #print(i, c)
    #
    #lr = LogisticRegression(C=c)
    lr = LogisticRegression(C=c, penalty=pnlty)
    lr.fit(X_train_std, y_train)
    #
    w_0 = lr.intercept_[0]
    w_1 = lr.coef_[0,0]
    w_2 = lr.coef_[0,1]
    score = lr.score(X_test_std, y_test)
    #
    #print(i/2, i%2)
    #print(math.floor(i/2), i%2)
    #####axs[i/2, i%2].set_title('C=' + str(c))
    axs[math.floor(i/2), i%2].set_title('C=' + str(c))
    #
    #####axs[i/2, i%2].plot([-2,2], map(lambda x: (-w_1 * x - w_0)/w_2, [-2,2]))
    axs[math.floor(i/2), i%2].plot([math.floor(xmin)-1, math.ceil(xmax)+1], list(map(lambda x: (-w_1 * x - w_0)/w_2, [math.floor(xmin)-1, math.ceil(xmax)+1])))
    #
    #####axs[i/2, i%2].scatter(X_train_std[y_train==0, 0], X_train_std[y_train==0, 1], c='red', marker='x', label='train 0')
    #axs[math.floor(i/2), i%2].scatter(X_train_std[y_train==0, 0], X_train_std[y_train==0, 1], c='red', marker='x', label='train 0')
    axs[math.floor(i/2), i%2].scatter(X_train_std[y_train==0, 0], X_train_std[y_train==0, 1], c='blue', marker='x', label='train 0')
    #
    #####axs[i/2, i%2].scatter(X_train_std[y_train==1, 0], X_train_std[y_train==1, 1], c='blue', marker='x', label='train 1')
    #axs[math.floor(i/2), i%2].scatter(X_train_std[y_train==1, 0], X_train_std[y_train==1, 1], c='blue', marker='x', label='train 1')
    axs[math.floor(i/2), i%2].scatter(X_train_std[y_train==1, 0], X_train_std[y_train==1, 1], c='red', marker='x', label='train 1')
    #
    #####axs[i/2, i%2].scatter(X_test_std[y_test==0, 0], X_test_std[y_test==0, 1], c='red', marker='o', s=60, label='test 0')
    #axs[math.floor(i/2), i%2].scatter(X_test_std[y_test==0, 0], X_test_std[y_test==0, 1], c='red', marker='o', s=60, label='test 0')
    axs[math.floor(i/2), i%2].scatter(X_test_std[y_test==0, 0], X_test_std[y_test==0, 1], c='blue', marker='o', s=60, label='test 0')
    #
    #####axs[i/2, i%2].scatter(X_test_std[y_test==1, 0], X_test_std[y_test==1, 1], c='blue', marker='o', s=60, label='test 1')
    #axs[math.floor(i/2), i%2].scatter(X_test_std[y_test==1, 0], X_test_std[y_test==1, 1], c='blue', marker='o', s=60, label='test 1')
    axs[math.floor(i/2), i%2].scatter(X_test_std[y_test==1, 0], X_test_std[y_test==1, 1], c='red', marker='o', s=60, label='test 1')
    #
    #
    if (i < 2):
        #####axs[i/2, i%2].text(0,-2.7, 'score ' + str(round(score,3)*100) + '%', size=13)
        axs[math.floor(i/2), i%2].text(math.floor(xmin) + (math.ceil(xmax) - math.floor(xmin)) * 0.10, math.ceil(ymax) - (math.ceil(ymax) - math.floor(ymin)) * 0.10, 'score ' + str(round(score,3)*100) + '%', size=10)
    else:
        #####axs[i/2, i%2].text(0,-3.3, 'score ' + str(round(score,3)*100) + '%', size=13)
        axs[math.floor(i/2), i%2].text(math.floor(xmin) + (math.ceil(xmax) - math.floor(xmin)) * 0.10, math.ceil(ymax) - (math.ceil(ymax) - math.floor(ymin)) * 0.10, 'score ' + str(round(score,3)*100) + '%', size=10)


plt.savefig('Figure_5_Traing_Data_and_Test_Data_plus_Boundary_Line_for_various_Cs.png')
plt.show()
plt.close()



Figures
Figure_1_Raw_Data.png



Figure_2_Traing_Data_and_Test_Data.png



Figure_3_Traing_Data_and_Test_Data_plus_Boundary_Line.png



Figure_4_Confusion_Matrix_Test_Data.png



Figure_5_Traing_Data_and_Test_Data_plus_Boundary_Line_for_various_Cs.png






- No comments:
Labels: , , ,

Multinomial Logistic Regression in Python (Probabilities in Each Class for New Observations)

Multinomial Logistic Regression in Python (Probabilities in Each Class for New Observations)

0_MacOS_Python_setup.txt
# Install on Terminal of MacOS

#pip3 install -U pandas

#pip3 install -U numpy

#pip3 install -U matplotlib

#pip3 install -U scikit-learn

#pip3 install -U seaborn


1_MacOS_Terminal.txt
########## Run Terminal on MacOS and execute
### TO UPDATE
cd "YOUR_WORKING_DIRECTORY"


python3 mlrprob.py Xy.csv Species XTEST.csv sepal_length petal_length



Data files

Xy.csv
sepal_length,sepal_width,petal_length,petal_width,Species
5.1,3.5,1.4,0.2,0
4.9,3.0,1.4,0.2,0
4.7,3.2,1.3,0.2,0
4.6,3.1,1.5,0.2,0
5.0,3.6,1.4,0.2,0
5.4,3.9,1.7,0.4,0
4.6,3.4,1.4,0.3,0
5.0,3.4,1.5,0.2,0
4.4,2.9,1.4,0.2,0
4.9,3.1,1.5,0.1,0
5.4,3.7,1.5,0.2,0
4.8,3.4,1.6,0.2,0
4.8,3.0,1.4,0.1,0
4.3,3.0,1.1,0.1,0
5.8,4.0,1.2,0.2,0
5.7,4.4,1.5,0.4,0
5.4,3.9,1.3,0.4,0
5.1,3.5,1.4,0.3,0
5.7,3.8,1.7,0.3,0
5.1,3.8,1.5,0.3,0
5.4,3.4,1.7,0.2,0
5.1,3.7,1.5,0.4,0
4.6,3.6,1.0,0.2,0
5.1,3.3,1.7,0.5,0
4.8,3.4,1.9,0.2,0
5.0,3.0,1.6,0.2,0
5.0,3.4,1.6,0.4,0
5.2,3.5,1.5,0.2,0
5.2,3.4,1.4,0.2,0
4.7,3.2,1.6,0.2,0
4.8,3.1,1.6,0.2,0
5.4,3.4,1.5,0.4,0
5.2,4.1,1.5,0.1,0
5.5,4.2,1.4,0.2,0
4.9,3.1,1.5,0.2,0
5.0,3.2,1.2,0.2,0
5.5,3.5,1.3,0.2,0
4.9,3.6,1.4,0.1,0
4.4,3.0,1.3,0.2,0
5.1,3.4,1.5,0.2,0
5.0,3.5,1.3,0.3,0
4.5,2.3,1.3,0.3,0
4.4,3.2,1.3,0.2,0
5.0,3.5,1.6,0.6,0
5.1,3.8,1.9,0.4,0
4.8,3.0,1.4,0.3,0
5.1,3.8,1.6,0.2,0
4.6,3.2,1.4,0.2,0
5.3,3.7,1.5,0.2,0
5.0,3.3,1.4,0.2,0
7.0,3.2,4.7,1.4,1
6.4,3.2,4.5,1.5,1
6.9,3.1,4.9,1.5,1
5.5,2.3,4.0,1.3,1
6.5,2.8,4.6,1.5,1
5.7,2.8,4.5,1.3,1
6.3,3.3,4.7,1.6,1
4.9,2.4,3.3,1.0,1
6.6,2.9,4.6,1.3,1
5.2,2.7,3.9,1.4,1
5.0,2.0,3.5,1.0,1
5.9,3.0,4.2,1.5,1
6.0,2.2,4.0,1.0,1
6.1,2.9,4.7,1.4,1
5.6,2.9,3.6,1.3,1
6.7,3.1,4.4,1.4,1
5.6,3.0,4.5,1.5,1
5.8,2.7,4.1,1.0,1
6.2,2.2,4.5,1.5,1
5.6,2.5,3.9,1.1,1
5.9,3.2,4.8,1.8,1
6.1,2.8,4.0,1.3,1
6.3,2.5,4.9,1.5,1
6.1,2.8,4.7,1.2,1
6.4,2.9,4.3,1.3,1
6.6,3.0,4.4,1.4,1
6.8,2.8,4.8,1.4,1
6.7,3.0,5.0,1.7,1
6.0,2.9,4.5,1.5,1
5.7,2.6,3.5,1.0,1
5.5,2.4,3.8,1.1,1
5.5,2.4,3.7,1.0,1
5.8,2.7,3.9,1.2,1
6.0,2.7,5.1,1.6,1
5.4,3.0,4.5,1.5,1
6.0,3.4,4.5,1.6,1
6.7,3.1,4.7,1.5,1
6.3,2.3,4.4,1.3,1
5.6,3.0,4.1,1.3,1
5.5,2.5,4.0,1.3,1
5.5,2.6,4.4,1.2,1
6.1,3.0,4.6,1.4,1
5.8,2.6,4.0,1.2,1
5.0,2.3,3.3,1.0,1
5.6,2.7,4.2,1.3,1
5.7,3.0,4.2,1.2,1
5.7,2.9,4.2,1.3,1
6.2,2.9,4.3,1.3,1
5.1,2.5,3.0,1.1,1
5.7,2.8,4.1,1.3,1
6.3,3.3,6.0,2.5,2
5.8,2.7,5.1,1.9,2
7.1,3.0,5.9,2.1,2
6.3,2.9,5.6,1.8,2
6.5,3.0,5.8,2.2,2
7.6,3.0,6.6,2.1,2
4.9,2.5,4.5,1.7,2
7.3,2.9,6.3,1.8,2
6.7,2.5,5.8,1.8,2
7.2,3.6,6.1,2.5,2
6.5,3.2,5.1,2.0,2
6.4,2.7,5.3,1.9,2
6.8,3.0,5.5,2.1,2
5.7,2.5,5.0,2.0,2
5.8,2.8,5.1,2.4,2
6.4,3.2,5.3,2.3,2
6.5,3.0,5.5,1.8,2
7.7,3.8,6.7,2.2,2
7.7,2.6,6.9,2.3,2
6.0,2.2,5.0,1.5,2
6.9,3.2,5.7,2.3,2
5.6,2.8,4.9,2.0,2
7.7,2.8,6.7,2.0,2
6.3,2.7,4.9,1.8,2
6.7,3.3,5.7,2.1,2
7.2,3.2,6.0,1.8,2
6.2,2.8,4.8,1.8,2
6.1,3.0,4.9,1.8,2
6.4,2.8,5.6,2.1,2
7.2,3.0,5.8,1.6,2
7.4,2.8,6.1,1.9,2
7.9,3.8,6.4,2.0,2
6.4,2.8,5.6,2.2,2
6.3,2.8,5.1,1.5,2
6.1,2.6,5.6,1.4,2
7.7,3.0,6.1,2.3,2
6.3,3.4,5.6,2.4,2
6.4,3.1,5.5,1.8,2
6.0,3.0,4.8,1.8,2
6.9,3.1,5.4,2.1,2
6.7,3.1,5.6,2.4,2
6.9,3.1,5.1,2.3,2
5.8,2.7,5.1,1.9,2
6.8,3.2,5.9,2.3,2
6.7,3.3,5.7,2.5,2
6.7,3.0,5.2,2.3,2
6.3,2.5,5.0,1.9,2
6.5,3.0,5.2,2.0,2
6.2,3.4,5.4,2.3,2
5.9,3.0,5.1,1.8,2



XTEST.csv
sepal_length,sepal_width,petal_length,petal_width
5.0,2.5,3.0,1.5
7.0,3.0,4.0,1.0




Python files

mlrprob.py
#################### Multinomial Logistic Regression in Python (Probabilities in Each Class for New Observations) ####################
#
#Run this script on Terminal of MacOS as follows:
#python3 mlrprob.py Xy.csv Species XTEST.csv sepal_length petal_length
#python3 mlrprob.py (Training Data File that has X and y) (y column name of Training Data File for labelling) (Test Data X without y) (x1 name) (x2 name)
#
#Reference
#https://chrisalbon.com/machine_learning/naive_bayes/multinomial_logistic_regression/
#https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_iris.html



########## import
from sklearn.linear_model import LogisticRegression
from sklearn import datasets
from sklearn.preprocessing import StandardScaler
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import sys



########## arguments
TrainFname = str(sys.argv[1])
yname = str(sys.argv[2])
TestFname = str(sys.argv[3])
x1name = str(sys.argv[4])
x2name = str(sys.argv[5])


#################### Multinomial Logistic Regression (Probabilities in Each Class for New Observations) ####################


########## Generate Training Data (without a header)

'''
iris = datasets.load_iris()
with open('iris.DESCR.txt', 'w') as f:
        print(iris.DESCR, file=f)
'''

#print(type(iris))
#<class 'sklearn.utils.Bunch'>

#X = iris.data
#y = iris.target
#
#print(type(X))
#<class 'numpy.ndarray'>
#
#print(type(y))
#<class 'numpy.ndarray'>
#
#pd.DataFrame(data=X).to_csv("X.csv", header=False, index=False)
#pd.DataFrame(data=y).to_csv("y.csv", header=False, index=False)
#
#X = pd.read_csv('X.csv', header=None).values
#y = pd.read_csv('y.csv', header=None).values.ravel()
#
#print(type(X))
#<class 'numpy.ndarray'>
#
#print(type(y))
#<class 'numpy.ndarray'>

#print(pd.DataFrame(X).head())    # a header is added with values 0, 1, 2, and 3
'''
     0    1    2    3
0  5.1  3.5  1.4  0.2
1  4.9  3.0  1.4  0.2
2  4.7  3.2  1.3  0.2
3  4.6  3.1  1.5  0.2
4  5.0  3.6  1.4  0.2
'''
#print(pd.DataFrame(X).tail())    # a header is added with values 0, 1, 2, and 3
'''
       0    1    2    3
145  6.7  3.0  5.2  2.3
146  6.3  2.5  5.0  1.9
147  6.5  3.0  5.2  2.0
148  6.2  3.4  5.4  2.3
149  5.9  3.0  5.1  1.8
'''
#
#print(pd.DataFrame(y).head())    # a header is added with value 0
'''
   0
0  0
1  0
2  0
3  0
4  0
'''
#print(pd.DataFrame(y).tail())    # a header is added with value 0
'''
     0
145  2
146  2
147  2
148  2
149  2
'''
#

#print(iris.feature_names)
#['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']

#print(iris.target_names)
#['setosa' 'versicolor' 'virginica']
#0(=setosa), 1(=versicolor), 2(=virginica)

#print(iris.data.shape)
#(150, 4)


########## Load Training Data (with a header)

Xy = pd.read_csv(TrainFname, header=0)

y = Xy[yname]

#X = Xy[Xy.columns[Xy.columns != yname]]
X = Xy.drop(yname, axis=1)

#print(X.describe())
'''
       sepal_length   sepal_width   petal_length   petal_width
count           150.000000           150.000000            150.000000           150.000000
mean              5.843333             3.057333              3.758000             1.199333
std               0.828066             0.435866              1.765298             0.762238
min               4.300000             2.000000              1.000000             0.100000
25%               5.100000             2.800000              1.600000             0.300000
50%               5.800000             3.000000              4.350000             1.300000
75%               6.400000             3.300000              5.100000             1.800000
max               7.900000             4.400000              6.900000             2.500000
'''

#print(type(X))
#<class 'pandas.core.frame.DataFrame'>

#print(type(y))
#<class 'pandas.core.frame.DataFrame'>

#print(X.head())
'''
   sepal_length   sepal_width   petal_length   petal_width
0                  5.1                  3.5                   1.4                  0.2
1                  4.9                  3.0                   1.4                  0.2
2                  4.7                  3.2                   1.3                  0.2
3                  4.6                  3.1                   1.5                  0.2
4                  5.0                  3.6                   1.4                  0.2
'''

#print(y.head())
'''
0         0
1         0
2         0
3         0
4         0
'''


###### Two Specified X columns

#If you comment this out, then all Xs will be considered.
X = Xy.loc[:, [x1name, x2name]]



###### Data Conversion From pd.DataFrame to np.ndarray

Xnp = X.values
ynp = y.values.ravel()

#print(type(Xnp))
#<class 'numpy.ndarray'>
#
#print(type(ynp))
#<class 'numpy.ndarray'>

#print(pd.DataFrame(Xnp).head())
'''
     0    1
0  5.1  1.4
1  4.9  1.4
2  4.7  1.3
3  4.6  1.5
4  5.0  1.4
'''
#
#print(pd.DataFrame(ynp).head())
'''
   0
0  0
1  0
2  0
3  0
4  0
'''




########## Standardize Features X

scaler = StandardScaler()    # Standardize Features X
#
#scaler = MinMaxScaler()    # Normalizing Features X
#
#print(type(scaler))
#<class 'sklearn.preprocessing._data.StandardScaler'>


X_std = scaler.fit_transform(Xnp)
#X_std = scaler.inverse_transform(Xnp)



########## Create Multinomial Logistic Regression
clf = LogisticRegression(random_state=0, multi_class='multinomial', solver='newton-cg')



########## Train Multinomial Logistic Regression (by using all features X)
# Train model
model = clf.fit(X_std, ynp)


########## Test Data without Answers: Previously Unseen Observations
#XTEST = pd.read_csv('XTEST.csv', header=0)
XTEST = pd.read_csv(TestFname, header=0)

#If you comment this out, then all Xs will be considered.
XTEST = XTEST.loc[:, [x1name, x2name]]

XTESTnp = XTEST.values

#print(type(XTESTnp))
#<class 'numpy.ndarray'>

#print(XTESTnp)
'''
[[5. 3.]
 [7. 4.]]
'''

print(pd.DataFrame(XTESTnp).head())
'''
     0    1
0  5.0  3.0
1  7.0  4.0
'''



########## Predict Observations’ Class
#model.predict(new_observation)
model.predict(XTESTnp)



########## Probabilities in Each Class for Test Data

#model.predict_proba(new_observation)
#print(model.predict_proba(new_observation))
print(model.predict_proba(XTESTnp))
'''
[[5.93479270e-03 6.37847758e-01 3.56217450e-01]
 [7.50267624e-17 7.48974219e-07 9.99999251e-01]]
'''
'''
[[5.93479270e-03 63.7847758 35.6217450]
 [7.50267624e-17 7.48974219e-07 99.9999251]]
'''
#0(=setosa), 1(=versicolor), 2(=virginica)


with open('model.predict_proba.txt', 'w') as f:
        print(model.predict_proba(XTESTnp), file=f)


#print(iris.target_names)
#['setosa' 'versicolor' 'virginica']
#0(=setosa), 1(=versicolor), 2(=virginica)


yclass = pd.DataFrame(ynp).drop_duplicates()
yclass = yclass.reset_index(drop=True)
#
#print(type(yclass))
#<class 'pandas.core.frame.DataFrame'>
#
#print(yclass)
'''
   0
0  0
1  1
2  2
'''



########## Pair Plot

sns.pairplot(Xy, hue=yname, height=2)
plt.legend()
plt.savefig('Figure_1_Pair_Plot.png')
plt.show()



#################### Multinominal Logistic Regression (Two Independent X) ####################


########## Data For Logistic Regression

#print(Xy)
#print(Xy.columns)
'''
Index(['sepal_length', 'sepal_width', 'petal_length', 'petal_width',
       'Species'],
      dtype='object')
'''
#print(Xy[x1name])
#print(Xy[x2name])

#print(Xy.loc[:, [x1name, x2name]])
#X = Xy[:, [x1name, x2name]]
X = Xy.loc[:, [x1name, x2name]]
#print(X)
#print(type(X))
#<class 'pandas.core.frame.DataFrame'>


X = X.values
#print(type(X))
#<class 'numpy.ndarray'>
#print(X)

y = y.values.ravel()
#print(type(y))
#<class 'numpy.ndarray'>



########## A Function for Decision Boundary Drawing

# graph common settings
h = .02  # step size in the mesh
x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

#def decision_boundary(clf, X, y, ax, title):
def decision_boundary(clf, X, y, ax, title, x1name, x2name):
    clf.fit(X, y)

    # Plot the decision boundary. For that, we will assign a color to each
    # point in the mesh [x_min, x_max]x[y_min, y_max].  
    Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])

    # Put the result into a color plot
    Z = Z.reshape(xx.shape)
    ax.pcolormesh(xx, yy, Z, cmap=plt.cm.Paired)

    # Plot also the training points
    ax.scatter(X[:, 0], X[:, 1], c=y, edgecolors='k', cmap=plt.cm.Paired)
    #ax.scatter(X[:, 0], X[:, 1], c=y, label=y, edgecolors='k', cmap=plt.cm.Paired)
    #ax.scatter(X[:, 0], X[:, 1], c=y, label=ax, edgecolors='k', cmap=plt.cm.Paired)
    #
    #ax.legend()
    #plt.legend()
    #
    #print(X[:, 0])
    #print(X[:, 1])
    #print(y)
   
    # label
    ax.set_title(title)
    #
    #ax.set_xlabel('sepal length')
    ax.set_xlabel(x1name)
    #ax.set_ylabel('petal length')
    ax.set_ylabel(x2name)
   
    #Test Data
    #ax.scatter(XTESTnp[:, 0], XTESTnp[:, 1], c=y, edgecolors='k', cmap=plt.cm.Paired)
    ax.scatter(XTESTnp[:, 0], XTESTnp[:, 1])



########## Logistic Regression


fig, axes = plt.subplots(1, 3, figsize=(12, 4))
#fig, axes = plt.subplots(1, 1, figsize=(10, 3))

#C is a parameter for regularization. Larger C means weak regularization and could cause over-fitting.
for ax, C in zip(axes, [0.01, 1, 100]):
    title = "C=%s"% (C)
    #
    clf = LogisticRegression(C=C)
    #print(type(clf))
    #<class 'sklearn.linear_model._logistic.LogisticRegression'>
    #
    #decision_boundary(clf, X, y, ax, title)
    decision_boundary(clf, X, y, ax, title, x1name, x2name)
    #ax.legend()
    #plt.legend()

#ax.legend()
#plt.legend()
#ax.scatter(XTESTnp[:, 0], XTESTnp[:, 1], c=y, edgecolors='k', cmap=plt.cm.Paired)
plt.savefig('Figure_2_Logistic_Regression.png')
plt.show()



Figures
- No comments:
Labels: ,

Saturday, June 13, 2020

Logistic Regression in Python (x1, x2: Two Independent Variables, y: One Dependent Variable, and various metrics)

Logistic Regression in Python (x1, x2: Two Independent Variables, y: One Dependent Variable, and various metrics)

0_MacOS_Python_setup.txt
# Install on Terminal of MacOS

#pip3 install -U pandas

#pip3 install -U numpy

#pip3 install -U matplotlib

#pip3 install -U scikit-learn

#pip3 install -U seaborn


1_MacOS_Terminal.txt
########## Run Terminal on MacOS and execute
### TO UPDATE
cd "YOUR_WORKING_DIRECTORY"


#Two numerical arguments after this script name are arbitrary.
#2 and 3 in this case mean that the third (x1 = 2) and the fourth (x2 = 3) columns of User_Data.csv will be used for this logistics regression to predict y_pred; y_pred (0 or 1) will be compared with actual results y (0 or 1).

python3 lgrs02.py 2 3 4 User_Data.csv



Data files

User_Data.csv
User ID,Gender,Age,EstimatedSalary,Purchased
15624510,Male,19,19000,0
15810944,Male,35,20000,0
15668575,Female,26,43000,0
15603246,Female,27,57000,0
15804002,Male,19,76000,0
15728773,Male,27,58000,0
15598044,Female,27,84000,0
15694829,Female,32,150000,1
15600575,Male,25,33000,0
15727311,Female,35,65000,0
15570769,Female,26,80000,0
15606274,Female,26,52000,0
15746139,Male,20,86000,0
15704987,Male,32,18000,0
15628972,Male,18,82000,0
15697686,Male,29,80000,0
15733883,Male,47,25000,1
15617482,Male,45,26000,1
15704583,Male,46,28000,1
15621083,Female,48,29000,1
15649487,Male,45,22000,1
15736760,Female,47,49000,1
15714658,Male,48,41000,1
15599081,Female,45,22000,1
15705113,Male,46,23000,1
15631159,Male,47,20000,1
15792818,Male,49,28000,1
15633531,Female,47,30000,1
15744529,Male,29,43000,0
15669656,Male,31,18000,0
15581198,Male,31,74000,0
15729054,Female,27,137000,1
15573452,Female,21,16000,0
15776733,Female,28,44000,0
15724858,Male,27,90000,0
15713144,Male,35,27000,0
15690188,Female,33,28000,0
15689425,Male,30,49000,0
15671766,Female,26,72000,0
15782806,Female,27,31000,0
15764419,Female,27,17000,0
15591915,Female,33,51000,0
15772798,Male,35,108000,0
15792008,Male,30,15000,0
15715541,Female,28,84000,0
15639277,Male,23,20000,0
15798850,Male,25,79000,0
15776348,Female,27,54000,0
15727696,Male,30,135000,1
15793813,Female,31,89000,0
15694395,Female,24,32000,0
15764195,Female,18,44000,0
15744919,Female,29,83000,0
15671655,Female,35,23000,0
15654901,Female,27,58000,0
15649136,Female,24,55000,0
15775562,Female,23,48000,0
15807481,Male,28,79000,0
15642885,Male,22,18000,0
15789109,Female,32,117000,0
15814004,Male,27,20000,0
15673619,Male,25,87000,0
15595135,Female,23,66000,0
15583681,Male,32,120000,1
15605000,Female,59,83000,0
15718071,Male,24,58000,0
15679760,Male,24,19000,0
15654574,Female,23,82000,0
15577178,Female,22,63000,0
15595324,Female,31,68000,0
15756932,Male,25,80000,0
15726358,Female,24,27000,0
15595228,Female,20,23000,0
15782530,Female,33,113000,0
15592877,Male,32,18000,0
15651983,Male,34,112000,1
15746737,Male,18,52000,0
15774179,Female,22,27000,0
15667265,Female,28,87000,0
15655123,Female,26,17000,0
15595917,Male,30,80000,0
15668385,Male,39,42000,0
15709476,Male,20,49000,0
15711218,Male,35,88000,0
15798659,Female,30,62000,0
15663939,Female,31,118000,1
15694946,Male,24,55000,0
15631912,Female,28,85000,0
15768816,Male,26,81000,0
15682268,Male,35,50000,0
15684801,Male,22,81000,0
15636428,Female,30,116000,0
15809823,Male,26,15000,0
15699284,Female,29,28000,0
15786993,Female,29,83000,0
15709441,Female,35,44000,0
15710257,Female,35,25000,0
15582492,Male,28,123000,1
15575694,Male,35,73000,0
15756820,Female,28,37000,0
15766289,Male,27,88000,0
15593014,Male,28,59000,0
15584545,Female,32,86000,0
15675949,Female,33,149000,1
15672091,Female,19,21000,0
15801658,Male,21,72000,0
15706185,Female,26,35000,0
15789863,Male,27,89000,0
15720943,Male,26,86000,0
15697997,Female,38,80000,0
15665416,Female,39,71000,0
15660200,Female,37,71000,0
15619653,Male,38,61000,0
15773447,Male,37,55000,0
15739160,Male,42,80000,0
15689237,Male,40,57000,0
15679297,Male,35,75000,0
15591433,Male,36,52000,0
15642725,Male,40,59000,0
15701962,Male,41,59000,0
15811613,Female,36,75000,0
15741049,Male,37,72000,0
15724423,Female,40,75000,0
15574305,Male,35,53000,0
15678168,Female,41,51000,0
15697020,Female,39,61000,0
15610801,Male,42,65000,0
15745232,Male,26,32000,0
15722758,Male,30,17000,0
15792102,Female,26,84000,0
15675185,Male,31,58000,0
15801247,Male,33,31000,0
15725660,Male,30,87000,0
15638963,Female,21,68000,0
15800061,Female,28,55000,0
15578006,Male,23,63000,0
15668504,Female,20,82000,0
15687491,Male,30,107000,1
15610403,Female,28,59000,0
15741094,Male,19,25000,0
15807909,Male,19,85000,0
15666141,Female,18,68000,0
15617134,Male,35,59000,0
15783029,Male,30,89000,0
15622833,Female,34,25000,0
15746422,Female,24,89000,0
15750839,Female,27,96000,1
15749130,Female,41,30000,0
15779862,Male,29,61000,0
15767871,Male,20,74000,0
15679651,Female,26,15000,0
15576219,Male,41,45000,0
15699247,Male,31,76000,0
15619087,Female,36,50000,0
15605327,Male,40,47000,0
15610140,Female,31,15000,0
15791174,Male,46,59000,0
15602373,Male,29,75000,0
15762605,Male,26,30000,0
15598840,Female,32,135000,1
15744279,Male,32,100000,1
15670619,Male,25,90000,0
15599533,Female,37,33000,0
15757837,Male,35,38000,0
15697574,Female,33,69000,0
15578738,Female,18,86000,0
15762228,Female,22,55000,0
15614827,Female,35,71000,0
15789815,Male,29,148000,1
15579781,Female,29,47000,0
15587013,Male,21,88000,0
15570932,Male,34,115000,0
15794661,Female,26,118000,0
15581654,Female,34,43000,0
15644296,Female,34,72000,0
15614420,Female,23,28000,0
15609653,Female,35,47000,0
15594577,Male,25,22000,0
15584114,Male,24,23000,0
15673367,Female,31,34000,0
15685576,Male,26,16000,0
15774727,Female,31,71000,0
15694288,Female,32,117000,1
15603319,Male,33,43000,0
15759066,Female,33,60000,0
15814816,Male,31,66000,0
15724402,Female,20,82000,0
15571059,Female,33,41000,0
15674206,Male,35,72000,0
15715160,Male,28,32000,0
15730448,Male,24,84000,0
15662067,Female,19,26000,0
15779581,Male,29,43000,0
15662901,Male,19,70000,0
15689751,Male,28,89000,0
15667742,Male,34,43000,0
15738448,Female,30,79000,0
15680243,Female,20,36000,0
15745083,Male,26,80000,0
15708228,Male,35,22000,0
15628523,Male,35,39000,0
15708196,Male,49,74000,0
15735549,Female,39,134000,1
15809347,Female,41,71000,0
15660866,Female,58,101000,1
15766609,Female,47,47000,0
15654230,Female,55,130000,1
15794566,Female,52,114000,0
15800890,Female,40,142000,1
15697424,Female,46,22000,0
15724536,Female,48,96000,1
15735878,Male,52,150000,1
15707596,Female,59,42000,0
15657163,Male,35,58000,0
15622478,Male,47,43000,0
15779529,Female,60,108000,1
15636023,Male,49,65000,0
15582066,Male,40,78000,0
15666675,Female,46,96000,0
15732987,Male,59,143000,1
15789432,Female,41,80000,0
15663161,Male,35,91000,1
15694879,Male,37,144000,1
15593715,Male,60,102000,1
15575002,Female,35,60000,0
15622171,Male,37,53000,0
15795224,Female,36,126000,1
15685346,Male,56,133000,1
15691808,Female,40,72000,0
15721007,Female,42,80000,1
15794253,Female,35,147000,1
15694453,Male,39,42000,0
15813113,Male,40,107000,1
15614187,Male,49,86000,1
15619407,Female,38,112000,0
15646227,Male,46,79000,1
15660541,Male,40,57000,0
15753874,Female,37,80000,0
15617877,Female,46,82000,0
15772073,Female,53,143000,1
15701537,Male,42,149000,1
15736228,Male,38,59000,0
15780572,Female,50,88000,1
15769596,Female,56,104000,1
15586996,Female,41,72000,0
15722061,Female,51,146000,1
15638003,Female,35,50000,0
15775590,Female,57,122000,1
15730688,Male,41,52000,0
15753102,Female,35,97000,1
15810075,Female,44,39000,0
15723373,Male,37,52000,0
15795298,Female,48,134000,1
15584320,Female,37,146000,1
15724161,Female,50,44000,0
15750056,Female,52,90000,1
15609637,Female,41,72000,0
15794493,Male,40,57000,0
15569641,Female,58,95000,1
15815236,Female,45,131000,1
15811177,Female,35,77000,0
15680587,Male,36,144000,1
15672821,Female,55,125000,1
15767681,Female,35,72000,0
15600379,Male,48,90000,1
15801336,Female,42,108000,1
15721592,Male,40,75000,0
15581282,Male,37,74000,0
15746203,Female,47,144000,1
15583137,Male,40,61000,0
15680752,Female,43,133000,0
15688172,Female,59,76000,1
15791373,Male,60,42000,1
15589449,Male,39,106000,1
15692819,Female,57,26000,1
15727467,Male,57,74000,1
15734312,Male,38,71000,0
15764604,Male,49,88000,1
15613014,Female,52,38000,1
15759684,Female,50,36000,1
15609669,Female,59,88000,1
15685536,Male,35,61000,0
15750447,Male,37,70000,1
15663249,Female,52,21000,1
15638646,Male,48,141000,0
15734161,Female,37,93000,1
15631070,Female,37,62000,0
15761950,Female,48,138000,1
15649668,Male,41,79000,0
15713912,Female,37,78000,1
15586757,Male,39,134000,1
15596522,Male,49,89000,1
15625395,Male,55,39000,1
15760570,Male,37,77000,0
15566689,Female,35,57000,0
15725794,Female,36,63000,0
15673539,Male,42,73000,1
15705298,Female,43,112000,1
15675791,Male,45,79000,0
15747043,Male,46,117000,1
15736397,Female,58,38000,1
15678201,Male,48,74000,1
15720745,Female,37,137000,1
15637593,Male,37,79000,1
15598070,Female,40,60000,0
15787550,Male,42,54000,0
15603942,Female,51,134000,0
15733973,Female,47,113000,1
15596761,Male,36,125000,1
15652400,Female,38,50000,0
15717893,Female,42,70000,0
15622585,Male,39,96000,1
15733964,Female,38,50000,0
15753861,Female,49,141000,1
15747097,Female,39,79000,0
15594762,Female,39,75000,1
15667417,Female,54,104000,1
15684861,Male,35,55000,0
15742204,Male,45,32000,1
15623502,Male,36,60000,0
15774872,Female,52,138000,1
15611191,Female,53,82000,1
15674331,Male,41,52000,0
15619465,Female,48,30000,1
15575247,Female,48,131000,1
15695679,Female,41,60000,0
15713463,Male,41,72000,0
15785170,Female,42,75000,0
15796351,Male,36,118000,1
15639576,Female,47,107000,1
15693264,Male,38,51000,0
15589715,Female,48,119000,1
15769902,Male,42,65000,0
15587177,Male,40,65000,0
15814553,Male,57,60000,1
15601550,Female,36,54000,0
15664907,Male,58,144000,1
15612465,Male,35,79000,0
15810800,Female,38,55000,0
15665760,Male,39,122000,1
15588080,Female,53,104000,1
15776844,Male,35,75000,0
15717560,Female,38,65000,0
15629739,Female,47,51000,1
15729908,Male,47,105000,1
15716781,Female,41,63000,0
15646936,Male,53,72000,1
15768151,Female,54,108000,1
15579212,Male,39,77000,0
15721835,Male,38,61000,0
15800515,Female,38,113000,1
15591279,Male,37,75000,0
15587419,Female,42,90000,1
15750335,Female,37,57000,0
15699619,Male,36,99000,1
15606472,Male,60,34000,1
15778368,Male,54,70000,1
15671387,Female,41,72000,0
15573926,Male,40,71000,1
15709183,Male,42,54000,0
15577514,Male,43,129000,1
15778830,Female,53,34000,1
15768072,Female,47,50000,1
15768293,Female,42,79000,0
15654456,Male,42,104000,1
15807525,Female,59,29000,1
15574372,Female,58,47000,1
15671249,Male,46,88000,1
15779744,Male,38,71000,0
15624755,Female,54,26000,1
15611430,Female,60,46000,1
15774744,Male,60,83000,1
15629885,Female,39,73000,0
15708791,Male,59,130000,1
15793890,Female,37,80000,0
15646091,Female,46,32000,1
15596984,Female,46,74000,0
15800215,Female,42,53000,0
15577806,Male,41,87000,1
15749381,Female,58,23000,1
15683758,Male,42,64000,0
15670615,Male,48,33000,1
15715622,Female,44,139000,1
15707634,Male,49,28000,1
15806901,Female,57,33000,1
15775335,Male,56,60000,1
15724150,Female,49,39000,1
15627220,Male,39,71000,0
15672330,Male,47,34000,1
15668521,Female,48,35000,1
15807837,Male,48,33000,1
15592570,Male,47,23000,1
15748589,Female,45,45000,1
15635893,Male,60,42000,1
15757632,Female,39,59000,0
15691863,Female,46,41000,1
15706071,Male,51,23000,1
15654296,Female,50,20000,1
15755018,Male,36,33000,0
15594041,Female,49,36000,1



Python files

lgrs02.py
#################### Logistic Regression in Python (x1, x2: Two Independent Variables, y: One Dependent Variable, and various metrics) ####################

##### Data
#
# Download csv data from the following website and save it to your working directory (where this lgrs01.py script is saved):
# https://drive.google.com/open?id=1Upqoz2gIAYq6LByD7YjHU-9_9K4EOhOh
# User_Data.csv
#
#User Database – This dataset contains information of users from a companies database. It contains information about UserID, Gender, Age, EstimatedSalary, Purchased. We are using this dataset for predicting that a user will purchase the company’s newly launched product or not.

##### Run this script on Terminal of MacOs as follows:
#
#python3 lgrs01.py 2 3 4 User_Data.csv
# Three numerical arguments after this script name are arbitrary.
# 2 and 3 in this case mean that the third (nx1 = 2) and the fourth (nx2 = 3) columns of User_Data.csv will be used for this logistics regression to predict y_pred.
# y_pred will be compared with actual results y (ny = 4 in this case).
#
# Generally, this is described as follows.
#python3 lgrs01.py (nx1) (nx2) (ny) (dt: dataset)


#Reference:
#ML | Logistic Regression using Python
#https://www.geeksforgeeks.org/ml-logistic-regression-using-python/


import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import sys


### columns of User_Data.csv
#Two numerical arguments (x1 and x2) after this script name (lgrs01.py) are arbitrary.
#2 and 3 in this case mean that the third (x1 = 2) and the fourth (x2 = 3) columns of User_Data.csv will be used for this logistics regression to predict y_pred; y_pred (0 or 1) will be compared with actual results y (0 or 1).
nx1 = int(sys.argv[1])
nx2 = int(sys.argv[2])
ny = int(sys.argv[3])
dt = str(sys.argv[4])


# Loading dataset – User_Data
#dataset = pd.read_csv('User_Data.csv')
dataset = pd.read_csv(dt)

#print(dataset)
'''
      User ID  Gender  Age  EstimatedSalary  Purchased
0    15624510    Male   19            19000          0
1    15810944    Male   35            20000          0
2    15668575  Female   26            43000          0
3    15603246  Female   27            57000          0
4    15804002    Male   19            76000          0
..        ...     ...  ...              ...        ...
395  15691863  Female   46            41000          1
396  15706071    Male   51            23000          1
397  15654296  Female   50            20000          1
398  15755018    Male   36            33000          0
399  15594041  Female   49            36000          1

[400 rows x 5 columns]
'''


#Now, to predict whether a user will purchase the product or not, one needs to find out the relationship between Age and Estimated Salary. Here User ID and Gender are not important factors for finding out this.

# input
#x = dataset.iloc[:, [2, 3]].values
x = dataset.iloc[:, [nx1, nx2]].values
# ignore the following
#     0: User ID,
#     1: Gender,
# and use the following
#     2: Age (to be used as x1),
#     3: EstimatedSalary (to be used as x2),

#print(x)
'''
[[    19  19000]
 [    35  20000]
(omitted)
 [    49  36000]]
'''

 
# output
#y = dataset.iloc[:, 4].values
y = dataset.iloc[:, ny].values
# This shows the results whether or not a user purchased.
#     4: Purchased (to be used as y)

#print(y)
'''
[0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 1 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0
 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0
 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 1 0 1 0 1 1 0 0 0 1 0 0 0 1 0 1
 1 1 0 0 1 1 0 1 1 0 1 1 0 1 0 0 0 1 1 0 1 1 0 1 0 1 0 1 0 0 1 1 0 1 0 0 1
 1 0 1 1 0 1 1 0 0 1 0 0 1 1 1 1 1 0 1 1 1 1 0 1 1 0 1 0 1 0 1 1 1 1 0 0 0
 1 1 0 1 1 1 1 1 0 0 0 1 1 0 0 1 0 1 0 1 1 0 1 0 1 1 0 1 1 0 0 0 1 1 0 1 0
 0 1 0 1 0 0 1 1 0 0 1 1 0 1 1 0 0 1 0 1 0 1 1 1 0 1 0 1 1 1 0 1 1 1 1 0 1
 1 1 0 1 0 1 0 0 1 1 0 1 1 1 1 1 1 0 1 1 1 1 1 1 0 1 1 1 0 1]
'''



# Splitting the dataset to train and test.
# 75% of data is used for training the model and 25% of it is used to test the performance of our model.

#from sklearn.cross_validation import train_test_split    #changed to the following new module
from sklearn.model_selection import train_test_split
xtrain, xtest, ytrain, ytest = train_test_split(x, y, test_size = 0.25, random_state = 0)


# saving raw data
xtrain_raw = xtrain
xtest_raw = xtest
ytrain_raw = ytrain
ytest_raw = ytest

# raw Training Data and Test Data output
pd.DataFrame(data=xtrain_raw).to_csv("xtrain_raw.csv", header=False, index=False)
pd.DataFrame(data=xtest_raw).to_csv("xtest_raw.csv", header=False, index=False)
pd.DataFrame(data=ytrain_raw).to_csv("ytrain_raw.csv", header=False, index=False)
pd.DataFrame(data=ytest_raw).to_csv("ytest_raw.csv", header=False, index=False)

# Training Data
trainx1mean = xtrain_raw[:,0].mean()
trainx2mean = xtrain_raw[:,1].mean()
trainx1std  = xtrain_raw[:,0].std()
trainx2std  = xtrain_raw[:,1].std()

# Training Data
testx1mean  = xtest_raw[:,0].mean()
testx2mean  = xtest_raw[:,1].mean()
testx1std   = xtest_raw[:,0].std()
testx2std   = xtest_raw[:,1].std()


# Standardization
#Now, it is very important to perform feature scaling here because Age and Estimated Salary values lie in different ranges. If we don’t scale the features then Estimated Salary feature will dominate Age feature when the model finds the nearest neighbor to a data point in data space.

from sklearn.preprocessing import StandardScaler
sc_x = StandardScaler()
xtrain = sc_x.fit_transform(xtrain)    # standardization model building and calculating standardized training data (xtrain)
#print(xtrain)
#print(len(xtrain))
#300
xtest = sc_x.transform(xtest)    # NO standarization model building, only calculating standardized test data (xtest)
#print(xtest)
#100
#print(len(xtest))

#print (xtrain[0:10, :])
'''
[[ 0.58164944 -0.88670699]
 [-0.60673761  1.46173768]
 [-0.01254409 -0.5677824 ]
 [-0.60673761  1.89663484]
 [ 1.37390747 -1.40858358]
 [ 1.47293972  0.99784738]
 [ 0.08648817 -0.79972756]
 [-0.01254409 -0.24885782]
 [-0.21060859 -0.5677824 ]
 [-0.21060859 -0.19087153]]
'''
#Here once see that Age and Estimated salary features values are sacled and now there in the -1 to 1. Hence, each feature will contribute equally in decision making i.e., finalizing the hypothesis.



#Finally, we are training our Logistic Regression model.
from sklearn.linear_model import LogisticRegression
classifier = LogisticRegression(random_state = 0)
classifier.fit(xtrain, ytrain)


#After training the model, it time to use it to do prediction on testing data.
y_pred = classifier.predict(xtest)


# Confusion Matrix
# Let’s test the performance of our model.
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(ytest, y_pred)
 
#print ("Confusion Matrix : \n", cm)
'''
Confusion Matrix :
 [[65  3]
 [ 8 24]]
'''
'''
Out of 100 :
TruePostive + TrueNegative = 24 + 65
FalsePositive + FalseNegative = 3 + 8

 [[TrueNegative(TN)  FalsePositive(FP)]
 [ FalseNegative(FN) TruePositive(TP)]]


                            Predicted Labels:
                            Negative              Positive
Actual Results: Negative    TrueNegative(TN)     FalsePositive(FP)
                Positive    FalseNegative(FN)    TruePositive(TP)
'''




#sklearn.metrics
#
#
# 1. Accuracy
# = (TruePostive + TrueNegative)/(All Dataset)
# = (TruePostive + TrueNegative)/(TruePostive + TrueNegative + FalsePositive + FalseNegative)
# = (TP + TN)/(TP + TN + FP + FN)
# = (24 + 65)/(24 + 65 + 3 + 8)
# = 0.89
from sklearn.metrics import accuracy_score
print ("Accuracy : ", accuracy_score(ytest, y_pred))
#Accuracy :  0.89
#
#
# 2. Precision
# = (TruePostive)/(TruePostive + FalsePositive)
# = (TP)/(TP + FP)
# = (24)/(24 + 3)
# = 0.888888888888889
from sklearn.metrics import precision_score
print ("Precision : ", precision_score(ytest, y_pred))
#Precision :  0.8888888888888888
#
#
#3. Recall
# = (TruePostive)/(TruePostive + FalseNegative)
# = (TP)/(TP + FN)
# = (24)/(24 + 8)
# = 0.75
from sklearn.metrics import recall_score
print ("Recall : ", recall_score(ytest, y_pred))
#Recall :  0.75
#
#
#4. F1-measure
# = (2 * Precision * Recall)/(Precision + Recall)
# = (2 * TruePostive)/(2 * TruePostive + FalsePositive + FalseNegative)
# = (2 * TP)/(2 * TP + FP + FN)
# = (2 * 24)/(2 * 24 + 3 + 8)
# = 0.813559322033898
from sklearn.metrics import f1_score
print ("F1-measure : ", f1_score(ytest, y_pred))
#F1-measure :  0.8135593220338982


##### Visualization (Standardized Test set)
# Visualizing the performance of our model.

from matplotlib.colors import ListedColormap

### Standardized Test Data set
X_set, y_set = xtest, ytest

pd.DataFrame(data=xtest).to_csv("xtest.csv", header=False, index=False)
pd.DataFrame(data=ytest).to_csv("ytest.csv", header=False, index=False)


X1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1,
                               stop = X_set[:, 0].max() + 1, step = 0.01),
                     np.arange(start = X_set[:, 1].min() - 1,
                               stop = X_set[:, 1].max() + 1, step = 0.01))

#size of Figure 1
plt.figure(figsize=(9, 9))


#This is the line applying the classifier on all the pixel observation points.
#It colors all the red pixel points and the green pixel points.
#contour function make the contour between red and green regions.
plt.contourf(X1, X2, classifier.predict(
             np.array([X1.ravel(), X2.ravel()]).T).reshape(
             X1.shape), alpha = 0.75, cmap = ListedColormap(('red', 'green')))
 
plt.xlim(X1.min(), X1.max())
plt.ylim(X2.min(), X2.max())



for i, j in enumerate(np.unique(y_set)):
    plt.scatter(X_set[y_set == j, 0], X_set[y_set == j, 1],
                c = ListedColormap(('yellow', 'blue'))(i), label = str(j) + " (Test Data)")



##### Visualization (Standardized Training set)
### Standardized Training Data set
X_set, y_set = xtrain, ytrain

for i, j in enumerate(np.unique(y_set)):
    plt.scatter(X_set[y_set == j, 0], X_set[y_set == j, 1],
                c = ListedColormap(('red', 'green'))(i), label = str(j) + " (Training Data)")



##### Figure 1: Logistic Regression
plt.title('Logistic Regession (Scatter Plot: Training Data and Test Data)')
plt.xlabel('x1 (Standardized)')
plt.ylabel('x2 (Standardized)')
plt.legend()

plt.text(-2.0, -2.0, 'Training Data: x1 mean = ' + str(round(trainx1mean, 2)) + ', x1 std = ' + str(round(trainx1std, 2)),fontsize=10)
plt.text(-2.0, -2.2, '      Test Data: x1 mean = ' + str(round(testx1mean, 2)) + ', x1 std = ' + str(round(testx1std, 2)),fontsize=10)
plt.text(-2.9, -2.5, 'Note: The model built by Standardized Training Data is used to fill both green region (1) and red region (0).',fontsize=10)

plt.text(-2.9, 3.0, 'Training Data: x2 mean = ' + str(round(trainx2mean, 2)) + ', x2 std = ' + str(round(trainx2std, 2)),fontsize=10)
plt.text(-2.9, 2.7, '       Test Data: x2 mean = ' + str(round(testx2mean, 2)) + ', x2 std = ' + str(round(testx2std, 2)),fontsize=10)

plt.savefig("Figure_1_Logistic_Regression_Standardized_Training_and_Test_Data.png")    # added to save a figure
plt.show()

#Analysing the performance measures – accuracy and confusion matrix and the graph, we can clearly say that our model is performing really good.



##### Figure 2: Confusion Matrix

#print ("Confusion Matrix : \n", cm)
'''
Confusion Matrix :
 [[65  3]
 [ 8 24]]
'''
'''
Out of 100 :
TruePostive + TrueNegative = 24 + 65
FalsePositive + FalseNegative = 3 + 8

 [[TrueNegative(TN)  FalsePositive(FP)]
 [ FalseNegative(FN) TruePositive(TP)]]


                            Predicted Labels:
                            Negative              Positive
Actual Results: Negative    TrueNegative(TN)     FalsePositive(FP)
                Positive    FalseNegative(FN)    TruePositive(TP)
'''

#use 1, 2 as there are two independent variables x1 and x2
class_names=[1,2]
#fig, ax = plt.subplots()
fig, ax = plt.subplots(figsize=(6,5))
tick_marks = np.arange(len(class_names))
plt.xticks(tick_marks, class_names)
plt.yticks(tick_marks, class_names)
#sns.heatmap(pd.DataFrame(cm), annot=True, cmap="Blues" ,fmt='g')
sns.heatmap(pd.DataFrame(cm), annot=True, cmap="coolwarm" ,fmt='g')
ax.xaxis.set_label_position("top")
#plt.tight_layout()
plt.tight_layout(pad=3.00)
plt.title('Confusion Matrix (Test Data)')
plt.xlabel('Predicted Labels')
plt.ylabel('Actual Results')
#print("Accuracy:",metrics.accuracy_score(y_test, y_pred))
plt.savefig("Figure_2_Confusion_Matrix_Test_Data.png")
plt.show()



Figures
Figure_1_Logistic_Regression_Standardized_Training_and_Test_Data.png

Figure_2_Confusion_Matrix_Test_Data.png




References
Download csv data from the following website and save it to your working directory (where the lgrs01.py script is saved):
https://drive.google.com/open?id=1Upqoz2gIAYq6LByD7YjHU-9_9K4EOhOh
User_Data.csv

- No comments:
Labels: ,
Subscribe to: Posts (Atom)

Deep Learning (Regression, Multiple Features/Explanatory Variables, Supervised Learning): Impelementation and Showing Biases and Weights

Deep Learning (Regression, Multiple Features/Explanatory Variables, Supervised Learning): Impelementation and Showing Biases and Weights ...