[Machine Learning with Python] My First Data Preprocessing Pipeline with Titanic Dataset

The Dataset was acquired from https://www.kaggle.com/c/titanic

For data preprocessing, I firstly defined three transformers:

DataFrameSelector: Select features to handle.
CombinedAttributesAdder: Add a categorical feature Age_cat which divided all passengers into three catagories according to their ages.
ImputeMostFrequent: Since the SimpleImputer( ) method was only suitable for numerical variables, I wrote an transformer to impute string missing values with the mode value. Here I was inspired by https://stackoverflow.com/questions/25239958/impute-categorical-missing-values-in-scikit-learn.

Then I wrote pipelines separately for different features

For numerical features, I applied DataFrameSelector, SimpleImputer and StandardScaler
For categorical features, I applied DataFrameSelector, ImputeMostFrequent and OneHotEncoder
For the new created feature Age_cat, since itself was a category but was derived from a numerical feature, I wrote an individual pipeline to impute the missing values and encode the categories.

Finally, we can build a full pipeline through FeatureUnion. Here is the code:

 # Read data

 import pandas as pd

 import numpy as np

 import os

 titanic_train = pd.read_csv('Dataset/Titanic/train.csv')

 titanic_test = pd.read_csv('Dataset/Titanic/test.csv')

 submission = pd.read_csv('Dataset/Titanic/gender_submission.csv')

 # Divide attributes and labels

 titanic_labels = titanic_train['Survived'].copy()

 titanic = titanic_train.drop(['Survived'],axis=1)

 # Feature Selection

 from sklearn.base import BaseEstimator, TransformerMixin

 class DataFrameSelector(BaseEstimator, TransformerMixin):

     def __init__(self,attribute_name):

         self.attribute_name = attribute_name

     def fit(self, X):

         return self

     def transform (self, X, y=None):

         if 'Pclass' in self.attribute_name:

             X['Pclass'] = X['Pclass'].astype(str)

         return X[self.attribute_name]

 # Feature Creation

 class CombinedAttributesAdder(BaseEstimator, TransformerMixin):

     def fit(self, X, y=None):

         return self  # nothing else to do

     def transform(self, X, y=None):

         Age_cat = pd.cut(X['Age'],[0,18,60,100],labels=['child', 'adult', 'old'])

         Age_cat=np.array(Age_cat)

         return pd.DataFrame(Age_cat,columns=['Age_Cat'])

 # Impute Categorical variables

 class ImputeMostFrequent(BaseEstimator, TransformerMixin):

     def fit(self, X, y=None):

         self.fill = pd.Series([X[c].value_counts().index[0] for c in X],index=X.columns)

         return self

     def transform(self, X, y=None):

         return X.fillna(self.fill)

 #Pipeline

 from sklearn.impute import SimpleImputer # Scikit-Learn 0.20+

 from sklearn.pipeline import Pipeline

 from sklearn.preprocessing import StandardScaler

 from sklearn.preprocessing import OneHotEncoder

 from sklearn.pipeline import FeatureUnion

 num_pipeline = Pipeline([

     ('selector',DataFrameSelector(['Age','SibSp','Parch','Fare'])),

     ('imputer', SimpleImputer(strategy="median")),

     ('std_scaler', StandardScaler()),

 ])

 cat_pipeline = Pipeline([

     ('selector',DataFrameSelector(['Pclass','Sex','Embarked'])),

     ('imputer',ImputeMostFrequent()),

     ('encoder', OneHotEncoder()),

 ])

 new_pipeline = Pipeline([

     ('selector',DataFrameSelector(['Age'])),

     #('imputer', SimpleImputer(strategy="median")),

     ('attr_adder',CombinedAttributesAdder()),

     ('imputer',ImputeMostFrequent()),

     ('encoder', OneHotEncoder()),

 ])

 full_pipeline = FeatureUnion([

     ("num", num_pipeline),

     ("cat", cat_pipeline),

     ("new", new_pipeline),

 ])

 titanic_prepared = full_pipeline.fit_transform(titanic)

Another thing I want to mention is that the output of a pipeline should be a 2D array rather a 1D array. So if you wanna choose only one feature, don't forget to transform the 1D array by reshape() method. Otherwise, you will receive an error like

ValueError: Expected 2D array, got 1D array instead

Specifically, apply reshape(-1,1) for column and reshape(1,-1). More about the issue can be found at https://stackoverflow.com/questions/51150153/valueerror-expected-2d-array-got-1d-array-instead.

巴特西

[Machine Learning with Python] My First Data Preprocessing Pipeline with Titanic Dataset

最新文章

热门文章