{
  "source_file": "0930",
  "source_platform": "Hashnode",
  "author": "Wesley Kambale",
  "title_listed": "Techniques of Feature Extraction in Machine Learning",
  "keywords": "",
  "domain": "数据处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://kambale.dev/feature-extraction-in-ml",
  "publish_date": "2023-05-29",
  "corpus_stage": "",
  "notes": "",
  "id": "6cdf5af5b1cb",
  "collected_at": "2026-10-02T14:11:36",
  "title": "Handling Feature Extraction in Machine Learning",
  "body_markdown": "Techniques of Feature Extraction in Machine Learning\nFrom Numerical Scaling to Image Descriptors: Mastering Feature Extraction for Optimal Machine Learning Models for Enhanced Model Performance\nUpdated\nJuly 18, 2023\n•\n10\nmin read\n•\nView as Markdown\nW\nWesley Kambale\nI'm a Machine Learning Engineer passionate about building production-ready ML systems for the African market.\nWith experience in TensorFlow, Keras, and Python-based workflows, I help teams bridge the gap between machine learning research and real-world deployment—especially on resource-constrained devices.\nI'm also a Google Developer Expert in AI. I regularly speak at tech conferences including PyCon Africa, DevFest Kampala, DevFest Nairobi and more and also write technical articles on AI/ML here.\nPart of series\nMachine Learning\nIntroduction\nFeature extraction is a crucial step in machine learning that involves transforming raw input data into a set of meaningful features that can be used for training models. The goal is to reduce the dimensionality of the data, remove irrelevant information, and extract relevant patterns and characteristics that can improve the model's performance.\nFeature extraction is particularly useful when dealing with high-dimensional data, such as images, text documents, or sensor readings. By selecting and extracting informative features, we can reduce computational complexity, remove noise, and improve the model's ability to generalize. The resulting features should be discriminative, informative, and independent.\nFeature Extraction Libraries\nPython provides several libraries that facilitate feature extraction:\nNumPy\n: For numerical feature extraction and manipulation.\nPandas\n: Ideal for handling tabular data and categorical feature extraction.\nscikit-learn\n: Offers various feature extraction techniques and utilities.\nNLTK (Natural Language Toolkit)\n: Useful for text feature extraction and processing.\nOpenCV\n: Widely used for image feature extraction\nTechniques of Feature Extraction\nThere are several techniques for feature extraction depending on the data that one has. Below, we cover these different techniques and how they are applied in machine learning.\nNumerical Feature Extraction\nScaling\nScaling is essential to bringing numerical features onto a common scale, preventing one feature from dominating others. Common scaling techniques include Min-Max scaling and Z-score normalization. Min-Max scaling scales the features to a specific range, usually between 0 and 1, while Z-score normalization transforms the features to have zero mean and unit variance.\nfrom\nsklearn.preprocessing\nimport\nMinMaxScaler\n\ndata = [[\n10\n,\n0.5\n], [\n20\n,\n0.8\n], [\n15\n,\n0.7\n]]\n# Create an instance of the MinMaxScaler\nscaler = MinMaxScaler()\n# Apply scaling to the data\nscaled_features = scaler.fit_transform(data)\nThe\nMinMaxScaler\nscales the features between 0 and 1, ensuring they are normalized\nBinning\nBinning is useful when dealing with continuous numerical features. It involves dividing the range of values into multiple bins or intervals and assigning each value to a specific bin. Binning helps reduce noise and capture patterns within specific value ranges.\nimport\nnumpy\nas\nnp\n\ndata = [\n2.5\n,\n3.7\n,\n1.9\n,\n4.2\n,\n5.1\n,\n2.8\n]\n# Create four bins from 1 to 6\nbins = np.linspace(\n1\n,\n6\n,\n4\n)\n# Assign each value to a bin\nbinned_features = np.digitize(data, bins)\nThe\nnp.digitize\nfunction assigns each value to a bin based on its position in the specified bins.\nAggregation\nAggregation involves computing statistical summaries of numerical features. Common aggregations include mean, median, standard deviation, minimum, maximum, and various percentiles. Aggregating features can provide insights into the overall distribution and characteristics of the data.\nimport\nnumpy\nas\nnp\n\ndata = [[\n10\n,\n20\n,\n15\n], [\n5\n,\n10\n,\n8\n], [\n12\n,\n18\n,\n20\n]]\n# Compute mean along each column (axis=0)\nmean_features = np.mean(data, axis=\n0\n)\n# Compute median along each column\nmedian_features = np.median(data, axis=\n0\n)\nThe\nnp.mean\nand\nnp.median\nfunctions compute the mean and median values along the specified axis, resulting in summary statistics for each feature.\nPolynomial Features\nPolynomial features allow capturing of nonlinear relationships between numerical features. By creating higher-order combinations, such as squares or interaction terms, we can introduce additional dimensions that may improve the model's ability to capture complex patterns.\nfrom\nsklearn.preprocessing\nimport\nPolynomialFeatures\n\ndata = [[\n2\n,\n3\n], [\n1\n,\n4\n], [\n5\n,\n2\n]]\n# Create polynomial features up to degree 2\npoly = PolynomialFeatures(degree=\n2\n)\n# Generate polynomial features\npolynomial_features = poly.fit_transform(data)\nThe\nPolynomialFeatures\nclass generates polynomial features up to the specified degree, allowing the model to capture nonlinear relationships between the original features.\nCategorical Feature Extraction\nOne-Hot Encoding\nOne-hot encoding transforms categorical variables into binary vectors. Each unique category becomes a separate binary feature, with a value of 1 indicating the presence of that category and 0 otherwise. One-hot encoding is suitable when there is no inherent order or hierarchy among categories.\nfrom\nsklearn.preprocessing\nimport\nOneHotEncoder\n\ndata = [[\n'Red'\n], [\n'Blue'\n], [\n'Green'\n], [\n'Red'\n]]\n# Create an instance of the OneHotEncoder\nencoder = OneHotEncoder()\n# Apply one-hot encoding\nonehot_features = encoder.fit_transform(data).toarray()\nThe\nOneHotEncoder\nencodes categorical features as binary vectors, where each unique category becomes a separate binary feature.\nLabel Encoding\nLabel encoding assigns a unique numeric label to each category. It is useful when there is an ordinal relationship among the categories. However, it is important to note that label encoding may introduce unintended ordinality that could impact model performance.\nfrom\nsklearn.preprocessing\nimport\nLabelEncoder\n\ndata = [\n'Low'\n,\n'High'\n,\n'Medium'\n,\n'Low'\n]\n# Create an instance of the LabelEncoder\nencoder = LabelEncoder()\n# Apply label encoding\nencoded_features = encoder.fit_transform(data)\nThe\nLabelEncoder\nassigns a numerical label to each category, preserving the order of categories.\nFrequency Encoding\nFrequency encoding replaces categorical values with their corresponding frequencies in the dataset. It can help capture the importance or prevalence of each category within the dataset.\nimport\npandas\nas\npd\n\ndata = pd.Series([\n'Apple'\n,\n'Banana'\n,\n'Apple'\n,\n'Orange'\n,\n'Banana'\n])\n# Compute frequency of each category\nfrequency = data.value_counts(normalize=\nTrue\n)\n# Replace categories with frequencies\nencoded_features = data.map(frequency)\nThe\nvalue_counts\nmethod computes the frequency of each category, and the map function replaces the categories with their corresponding frequencies.\nTarget Encoding\nTarget encoding replaces categorical values with the mean (or other statistics) of the target variable for each category. It leverages the relationship between the target variable and the categorical feature, potentially capturing valuable information for predictive modeling.\nimport\npandas\nas\npd\n\ndata = pd.DataFrame({\n'Category'\n: [\n'A'\n,\n'B'\n,\n'A'\n,\n'B'\n],\n'Target'\n: [\n1\n,\n0\n,\n1\n,\n1\n]})\n# Compute mean target value for each category\ntarget_mean = data.groupby(\n'Category'\n)[\n'Target'\n].mean()\n# Replace categories with mean target values\nencoded_features = data[\n'Category'\n].map(target_mean)\nThe\ngroupby\nmethod groups the data by category, and the\nmean\nfunction computes the mean target value for each category. The\nmap\nfunction replaces the categories with their corresponding mean target values.\nText Feature Extraction\nBag-of-Words (BoW)\nThe bag-of-words approach represents text documents by creating a vocabulary of unique words and counting the occurrence of each word in each document. The resulting representation is a count or frequency matrix, where each row corresponds to a document, and each column corresponds to a word in the vocabulary.\nfrom\nsklearn.feature_extraction.text\nimport\nCountVectorizer\n\ndata = [\n'I love dogs'\n,\n'I hate cats'\n,\n'Dogs are cute'\n]\n# Create an instance of CountVectorizer\nvectorizer = CountVectorizer()\n# Apply BoW transformation\nbow_features = vectorizer.fit_transform(data)\nThe\nCountVectorizer\nconverts text into a matrix of token counts, where each row represents a document, and each column represents a unique word in the vocabulary.\nTerm Frequency-Inverse Document Frequency (TF-IDF)\nTF-IDF assigns weights to words based on their frequency within a document and their rarity across the entire document collection. It downplays common words and emphasizes rare and distinctive words, providing a more informative representation of the text.\nfrom\nsklearn.feature_extraction.text\nimport\nTfidfVectorizer\n\ndata = [\n'I love dogs'\n,\n'I hate cats'\n,\n'Dogs are cute'\n]\n# Create an instance of TfidfVectorizer\nvectorizer = TfidfVectorizer()\n# Apply TF-IDF transformation\ntfidf_features = vectorizer.fit_transform(data)\nThe\nTfidfVectorizer\ncomputes TF-IDF weights for each word, where TF measures the word's frequency in a document, and IDF measures its rarity across the document collection.\nWord Embeddings\nWord embeddings represent words as dense vectors in a continuous vector space, capturing semantic relationships between words. Techniques like Word2Vec and GloVe learn representations based on the surrounding context of words, enabling the model to capture word similarities and analogies.\nimport\ngensim\nfrom\ngensim.models\nimport\nWord2Vec\n\ndata = [[\n'I'\n,\n'love'\n,\n'dogs'\n], [\n'I'\n,\n'hate'\n,\n'cats'\n], [\n'Dogs'\n,\n'are'\n,\n'cute'\n]]\n# Create a Word2Vec model\nmodel = Word2Vec(data, min_count=\n1\n)\n# Obtain the word embedding for 'dogs'\nword_embedding = model.wv[\n'dogs'\n]\nThe\nWord2Vec\nmodel learns word embeddings based on the context of words in the provided data. Each word is represented as a dense vector, and we can access the word embeddings using the model's wv property.\nN-grams\nN-grams represent contiguous sequences of N words in a text document. By considering contextual information and preserving word order, N-grams capture richer information about the relationships between words.\nfrom\nsklearn.feature_extraction.text\nimport\nCountVectorizer\n\ndata = [\n'I love dogs'\n,\n'I hate cats'\n,\n'Dogs are cute'\n]\nvectorizer = CountVectorizer(ngram_range=(\n1\n,\n2\n))\nngram_features = vectorizer.fit_transform(data)\nImage Feature Extraction\nHistogram of Oriented Gradients (HOG)\nHOG calculates the distribution of gradient orientations within image patches. It captures local shape information and is commonly used for object detection and recognition tasks.\nimport\ncv2\n\nimage = cv2.imread(\n'image.jpg'\n,\n0\n)\nhog = cv2.HOGDescriptor()\nhog_features = hog.compute(image)\nThe\nHOGDescriptor\ncomputes the HOG features for an image, which represents the local shape and edge information of objects in the image.\nScale-Invariant Feature Transform (SIFT)\nSIFT identifies key points and descriptors that are invariant to scale, rotation, and affine transformations. It is widely used for image matching and recognition, particularly in scenarios with significant variations in lighting and viewpoint.\nimport\ncv2\n\nimage = cv2.imread(\n'image.jpg'\n,\n0\n)\nsift = cv2.SIFT_create()\nkeypoints, descriptors = sift.detectAndCompute(image,\nNone\n)\nThe\nSIFT_create\nfunction creates a SIFT object, and\ndetectAndCompute\nextracts key points and their descriptors from the image, capturing distinctive features invariant to transformations.\nConvolutional Neural Networks (CNNs)\nCNNs are deep learning models designed to automatically learn hierarchical representations from images. They consist of multiple layers, including convolutional layers that extract local features and pooling layers that aggregate information. CNNs have achieved remarkable success in various computer vision tasks.\nimport\ntensorflow\nas\ntf\nfrom\ntensorflow.keras.applications\nimport\nVGG16\n\nimage = tf.keras.preprocessing.image.load_img(\n'image.jpg'\n, target_size=(\n224\n,\n224\n))\nimage = tf.keras.preprocessing.image.img_to_array(image)\nimage = tf.keras.applications.vgg16.preprocess_input(image)\nvgg_model = VGG16(weights=\n'imagenet'\n, include_top=\nFalse\n)\nfeatures = vgg_model.predict(np.expand_dims(image, axis=\n0\n))\nThe\nVGG16\nmodel is a pre-trained CNN that learns hierarchical features from images. We preprocess the image and extract features using the model's predict function.\nPre-trained Models\nPre-trained models, such as VGG, ResNet, or Inception, are deep-learning models trained on large-scale image datasets. Instead of training from scratch, we can utilize these models and extract features from intermediate layers. The features extracted from these models can capture high-level image representations that can be used for transfer learning or as input to other models.\nimport\ntensorflow\nas\ntf\nfrom\ntensorflow.keras.applications\nimport\nVGG16\n\nimage = tf.keras.preprocessing.image.load_img(\n'image.jpg'\n, target_size=(\n224\n,\n224\n))\nimage = tf.keras.preprocessing.image.img_to_array(image)\nimage = tf.keras.applications.vgg16.preprocess_input(image)\nvgg_model = VGG16(weights=\n'imagenet'\n, include_top=\nFalse\n)\nintermediate_layer_model = tf.keras.Model(inputs=vgg_model.input, outputs=vgg_model.get_layer(\n'block4_pool'\n).output)\nfeatures = intermediate_layer_model.predict(np.expand_dims(image, axis=\n0\n))\nThe\nintermediate_layer_model\nis created by specifying the desired intermediate layer of the pre-trained VGG16 model. By extracting features from this intermediate layer, we capture high-level representations that can be used for transfer learning or as inputs to other models.\nFurther Feature Extraction Techniques\nPrincipal Component Analysis (PCA)\nPrincipal Component Analysis (PCA) is a widely used technique for dimensionality reduction. It identifies the directions (principal components) in which the data varies the most and projects the data onto these components, effectively reducing the dimensionality while preserving most of the information.\nfrom\nsklearn.decomposition\nimport\nPCA\n# Assuming X is your input data\n# Specify the number of components you want to extract\npca = PCA(n_components=\n2\n)\n# X_pca contains the extracted features with reduced dimensionality\nX_pca = pca.fit_transform(X)\nIndependent Component Analysis (ICA)\nIndependent Component Analysis (ICA) is another technique for feature extraction that aims to find statistically independent components from the input data. It assumes that the observed data is a linear combination of independent sources and tries to separate these sources.\nfrom\nsklearn.decomposition\nimport\nFastICA\n# Assuming X is your input data\n# Specify the number of components you want to extract\nica = FastICA(n_components=\n2\n)\n# X_ica contains the extracted features with reduced dimensionality\nX_ica = ica.fit_transform(X)\nFeature Selection\nInstead of transforming the input data, feature selection focuses on selecting a subset of the existing features that are most relevant to the prediction task. This approach can be particularly useful when dealing with high-dimensional data.\nHere's an example of using feature selection with scikit-learn:\nfrom\nsklearn.feature_selection\nimport\nSelectKBest, chi2\n# Assuming X and y are your input features and target labels, respectively\n# Select the top 10 features\nselector = SelectKBest(score_func=chi2, k=\n10\n)\n# X_new contains the selected features\nX_new = selector.fit_transform(X, y)\nConclusion\nFeature extraction is a fundamental step in machine learning that significantly influences model performance. By selecting appropriate techniques for numerical, categorical, text, or image data, we can transform raw data into meaningful representations that capture relevant information. Experimenting with different feature extraction methods and understanding their impact on model performance is crucial for building accurate and robust machine learning models. The choice of feature extraction technique depends on the specific problem and the characteristics of your data.\nResources\nAccess the code snippets in a Google Colab notebook\nhere\n(MIT License).\n#\nfeatures\n#\nmachine-learning\n#\ndata",
  "metadata": {
    "title": "Handling Feature Extraction in Machine Learning",
    "description": "From Numerical Scaling to Image Descriptors: Mastering Feature Extraction for Optimal Machine Learning Models for Enhanced Model Performance",
    "keywords": null,
    "author": null,
    "og:title": "Handling Feature Extraction in Machine Learning",
    "og:description": "From Numerical Scaling to Image Descriptors: Mastering Feature Extraction for Optimal Machine Learning Models for Enhanced Model Performance",
    "og:url": "https://kambale.dev/feature-extraction-in-ml",
    "og:site_name": "Wesley Kambale",
    "og:image": "https://cdn.hashnode.com/res/hashnode/image/upload/v1689697953484/cd442115-3da1-4d63-9f4b-a148d07c470c.png",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:creator": "https://twitter.com/WesKambale",
    "twitter:title": "Handling Feature Extraction in Machine Learning",
    "twitter:description": "From Numerical Scaling to Image Descriptors: Mastering Feature Extraction for Optimal Machine Learning Models for Enhanced Model Performance",
    "twitter:image": "https://cdn.hashnode.com/res/hashnode/image/upload/v1689697953484/cd442115-3da1-4d63-9f4b-a148d07c470c.png",
    "article:published_time": "2023-05-29T23:11:17.295Z",
    "article:modified_time": "2023-07-18T16:45:35.471Z",
    "article:author": "https://kambale.dev/@kambale"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/6cdf5af5b1cb/code/body-000.txt",
      "chars": 229,
      "language": "text"
    },
    {
      "file": "attachments/6cdf5af5b1cb/code/body-001.txt",
      "chars": 191,
      "language": "text"
    },
    {
      "file": "attachments/6cdf5af5b1cb/code/body-002.txt",
      "chars": 229,
      "language": "text"
    },
    {
      "file": "attachments/6cdf5af5b1cb/code/body-003.txt",
      "chars": 247,
      "language": "text"
    },
    {
      "file": "attachments/6cdf5af5b1cb/code/body-004.txt",
      "chars": 249,
      "language": "text"
    },
    {
      "file": "attachments/6cdf5af5b1cb/code/body-005.txt",
      "chars": 229,
      "language": "text"
    },
    {
      "file": "attachments/6cdf5af5b1cb/code/body-006.txt",
      "chars": 252,
      "language": "text"
    },
    {
      "file": "attachments/6cdf5af5b1cb/code/body-007.txt",
      "chars": 304,
      "language": "text"
    },
    {
      "file": "attachments/6cdf5af5b1cb/code/body-008.txt",
      "chars": 264,
      "language": "text"
    },
    {
      "file": "attachments/6cdf5af5b1cb/code/body-009.txt",
      "chars": 269,
      "language": "text"
    },
    {
      "file": "attachments/6cdf5af5b1cb/code/body-010.txt",
      "chars": 268,
      "language": "text"
    },
    {
      "file": "attachments/6cdf5af5b1cb/code/body-011.txt",
      "chars": 213,
      "language": "text"
    },
    {
      "file": "attachments/6cdf5af5b1cb/code/body-012.txt",
      "chars": 107,
      "language": "text"
    },
    {
      "file": "attachments/6cdf5af5b1cb/code/body-013.txt",
      "chars": 132,
      "language": "text"
    },
    {
      "file": "attachments/6cdf5af5b1cb/code/body-014.txt",
      "chars": 390,
      "language": "text"
    },
    {
      "file": "attachments/6cdf5af5b1cb/code/body-015.txt",
      "chars": 522,
      "language": "text"
    },
    {
      "file": "attachments/6cdf5af5b1cb/code/body-016.txt",
      "chars": 251,
      "language": "text"
    },
    {
      "file": "attachments/6cdf5af5b1cb/code/body-017.txt",
      "chars": 260,
      "language": "text"
    },
    {
      "file": "attachments/6cdf5af5b1cb/code/body-018.txt",
      "chars": 286,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/res/hashnode/image/upload/v1689697953484/cd442115-3da1-4d63-9f4b-a148d07c470c.png",
      "bytes": 92292,
      "reused": true,
      "local_path": "attachments/6cdf5af5b1cb/images/000.png",
      "src": "https://cdn.hashnode.com/res/hashnode/image/upload/v1689697953484/cd442115-3da1-4d63-9f4b-a148d07c470c.png",
      "alt": "Techniques of Feature Extraction in Machine Learning",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://kambale.dev/feature-extraction-in-ml.md",
      "label": "View as Markdown",
      "type": "a"
    },
    {
      "href": "https://hashnode.com/@kambale",
      "label": "W",
      "type": "a"
    },
    {
      "href": "https://twitter.com/WesKambale",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://linkedin.com/in/WesKambale",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://kambale.dev/series/ml-series",
      "label": "Part of series Machine Learning",
      "type": "a"
    },
    {
      "href": "https://colab.research.google.com/drive/1nPzeVEK63mD2uzcaYEd6SNxLGibc6-of?usp=sharing",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://kambale.dev/tag/features",
      "label": "# features",
      "type": "a"
    },
    {
      "href": "https://kambale.dev/tag/machine-learning",
      "label": "# machine-learning",
      "type": "a"
    },
    {
      "href": "https://kambale.dev/tag/data",
      "label": "# data",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:52Z",
  "source_url": "https://kambale.dev/feature-extraction-in-ml",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 16141,
  "expected_body_images": 1,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/6cdf5af5b1cb/pages/source.html",
  "local_html": "attachments/6cdf5af5b1cb/pages/article.html",
  "body_base_path": "attachments/6cdf5af5b1cb/pages/",
  "body_markdown_path": "attachments/6cdf5af5b1cb/pages/article.md"
}