{ "id": "6d3d9dca0d80", "url": "https://lilianweng.github.io/posts/2017-10-29-object-recognition-part-1/", "source_excel_row": 1602, "source_url": "https://lilianweng.github.io/posts/2017-10-29-object-recognition-part-1/", "is_paid_flag": "", "images": [ { "status": "ok", "url": "https://lilianweng.github.io/posts/2017-10-29-object-recognition-part-1/image-gradient-vector-pixel-location.png", "bytes": 117546, "reused": true, "local_path": "attachments/6d3d9dca0d80/images/body-834dca5b0e8f45a5caae.png", "src": "https://lilianweng.github.io/posts/2017-10-29-object-recognition-part-1/image-gradient-vector-pixel-location.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2017-10-29-object-recognition-part-1/manu-2004.png", "bytes": 2066932, "reused": true, "local_path": "attachments/6d3d9dca0d80/images/body-db79205a3fdc6b1972ac.png", "src": "https://lilianweng.github.io/posts/2017-10-29-object-recognition-part-1/manu-2004.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2017-10-29-object-recognition-part-1/manu-2004-sobel-operator.png", "bytes": 330686, "reused": true, "local_path": "attachments/6d3d9dca0d80/images/body-f4bbdd9860111196027c.png", "src": "https://lilianweng.github.io/posts/2017-10-29-object-recognition-part-1/manu-2004-sobel-operator.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2017-10-29-object-recognition-part-1/HOG-histogram-creation.png", "bytes": 77027, "reused": true, "local_path": "attachments/6d3d9dca0d80/images/body-7663a2768e6a5304e97d.png", "src": "https://lilianweng.github.io/posts/2017-10-29-object-recognition-part-1/HOG-histogram-creation.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2017-10-29-object-recognition-part-1/block_histogram.png", "bytes": 92546, "reused": true, "local_path": "attachments/6d3d9dca0d80/images/body-c0097767c26c92046b06.png", "src": "https://lilianweng.github.io/posts/2017-10-29-object-recognition-part-1/block_histogram.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2017-10-29-object-recognition-part-1/image-segmentation-indoor.png", "bytes": 710268, "reused": true, "local_path": "attachments/6d3d9dca0d80/images/body-6a8653bd53396d46c1fa.png", "src": "https://lilianweng.github.io/posts/2017-10-29-object-recognition-part-1/image-segmentation-indoor.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2017-10-29-object-recognition-part-1/manu-2013.png", "bytes": 1618413, "reused": true, "local_path": "attachments/6d3d9dca0d80/images/body-ec2093ad281b035abb92.png", "src": "https://lilianweng.github.io/posts/2017-10-29-object-recognition-part-1/manu-2013.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2017-10-29-object-recognition-part-1/manu-2013-segmentation.png", "bytes": 292184, "reused": true, "local_path": "attachments/6d3d9dca0d80/images/body-5c3e06f69064d46da7a3.png", "src": "https://lilianweng.github.io/posts/2017-10-29-object-recognition-part-1/manu-2013-segmentation.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2017-10-29-object-recognition-part-1/selective-search-algorithm.png", "bytes": 372187, "reused": true, "local_path": "attachments/6d3d9dca0d80/images/body-f8d914d42f0dceb916fe.png", "src": "https://lilianweng.github.io/posts/2017-10-29-object-recognition-part-1/selective-search-algorithm.png", "alt": "", "downloaded": true } ], "videos": [], "body_markdown": "I’ve never worked in the field of computer vision and has no idea how the magic could work when an autonomous car is configured to tell apart a stop sign from a pedestrian in a red hat. To motivate myself to look into the maths behind object recognition and detection algorithms, I’m writing a few posts on this topic “Object Detection for Dummies”. This post, part 1, starts with super rudimentary concepts in image processing and a few methods for image segmentation. Nothing related to deep neural networks yet. Deep learning models for object detection and recognition will be discussed in\nPart 2\nand\nPart 3\n.\nDisclaimer: When I started, I was using “object recognition” and “object detection” interchangeably. I don’t think they are the same: the former is more about telling whether an object exists in an image while the latter needs to spot where the object is. However, they are highly related and many object recognition algorithms lay the foundation for detection.\nLinks to all the posts in the series:\n[\nPart 1\n]\n[\nPart 2\n]\n[\nPart 3\n]\n[\nPart 4\n].\nImage Gradient Vector\n#\nFirst of all, I would like to make sure we can distinguish the following terms. They are very similar, closely related, but not exactly the same.\nDerivative\nDirectional Derivative\nGradient\nValue type\nScalar\nScalar\nVector\nDefinition\nThe rate of change of a function $f(x,y,z,…)$ at a point $(x_0,y_0,z_0,…)$, which is the slope of the tangent line at the point.\nThe instantaneous rate of change of $f(x,y,z, …)$ in the direction of an unit vector $\\vec{u}$.\nIt points in the direction of the greatest rate of increase of the function, containing all the partial derivative information of a multivariable function.\nIn the image processing, we want to know the direction of colors changing from one extreme to the other (i.e. black to white on a grayscale image). Therefore, we want to measure “gradient” on pixels of colors. The gradient on an image is discrete because each pixel is independent and cannot be further split.\nThe\nimage gradient vector\nis defined as a metric for every individual pixel, containing the pixel color changes in both x-axis and y-axis. The definition is aligned with the gradient of a continuous multi-variable function, which is a vector of partial derivatives of all the variables. Suppose f(x, y) records the color of the pixel at location (x, y), the gradient vector of the pixel (x, y) is defined as follows:\n$$\n\\begin{align*}\n\\nabla f(x, y)\n= \\begin{bmatrix}\n g_x \\\\\n g_y\n\\end{bmatrix}\n= \\begin{bmatrix}\n \\frac{\\partial f}{\\partial x} \\\\[6pt]\n \\frac{\\partial f}{\\partial y}\n\\end{bmatrix}\n= \\begin{bmatrix}\n f(x+1, y) - f(x-1, y)\\\\\n f(x, y+1) - f(x, y-1)\n\\end{bmatrix}\n\\end{align*}\n$$\nThe $\\frac{\\partial f}{\\partial x}$ term is the partial derivative on the x-direction, which is computed as the color difference between the adjacent pixels on the left and right of the target, f(x+1, y) - f(x-1, y). Similarly, the $\\frac{\\partial f}{\\partial y}$ term is the partial derivative on the y-direction, measured as f(x, y+1) - f(x, y-1), the color difference between the adjacent pixels above and below the target.\nThere are two important attributes of an image gradient:\nMagnitude\nis the L2-norm of the vector, $g = \\sqrt{ g_x^2 + g_y^2 }$.\nDirection\nis the arctangent of the ratio between the partial derivatives on two directions, $\\theta = \\arctan{(g_y / g_x)}$.\nTo compute the gradient vector of a target pixel at location (x, y), we need to know the colors of its four neighbors (or eight surrounding pixels depending on the kernel).\nThe gradient vector of the example in is:\n$$\n\\begin{align*}\n\\nabla f \n= \\begin{bmatrix}\n f(x+1, y) - f(x-1, y)\\\\\n f(x, y+1) - f(x, y-1)\n\\end{bmatrix}\n= \\begin{bmatrix}\n 55-105\\\\\n 90-40\n\\end{bmatrix}\n= \\begin{bmatrix}\n -50\\\\\n 50\n\\end{bmatrix}\n\\end{align*}\n$$\nThus,\nthe magnitude is $\\sqrt{50^2 + (-50)^2} = 70.7107$, and\nthe direction is $\\arctan{(-50/50)} = -45^{\\circ}$.\nRepeating the gradient computation process for every pixel iteratively is too slow. Instead, it can be well translated into applying a convolution operator on the entire image matrix, labeled as $\\mathbf{A}$ using one of the specially designed convolutional kernels.\nLet’s start with the x-direction of the example in Fig 1. using the kernel $[-1,0,1]$ sliding over the x-axis; $\\ast$ is the convolution operator:\n$$\n\\begin{align*}\n\\mathbf{G}_x &= \n[-1, 0, 1] \\ast [105, 255, 55] = -105 + 0 + 55 = -50\n\\end{align*}\n$$\nSimilarly, on the y-direction, we adopt the kernel $[+1, 0, -1]^\\top$:\n$$\n\\begin{align*}\n\\mathbf{G}_y &= \n[+1, 0, -1]^\\top \\ast\n\\begin{bmatrix}\n 90\\\\\n 255\\\\\n 40\n\\end{bmatrix} \n= 90 + 0 - 40 = 50\n\\end{align*}\n$$\nTry this in python:\nimport\nnumpy\nas\nnp\nimport\nscipy.signal\nas\nsig\ndata\n=\nnp\n.\narray([[\n0\n,\n105\n,\n0\n], [\n40\n,\n255\n,\n90\n], [\n0\n,\n55\n,\n0\n]])\nG_x\n=\nsig\n.\nconvolve2d(data, np\n.\narray([[\n-\n1\n,\n0\n,\n1\n]]), mode\n=\n'valid'\n)\nG_y\n=\nsig\n.\nconvolve2d(data, np\n.\narray([[\n-\n1\n], [\n0\n], [\n1\n]]), mode\n=\n'valid'\n)\nThese two functions return\narray([[0], [-50], [0]])\nand\narray([[0, 50, 0]])\nrespectively. (Note that in the numpy array representation, 40 is shown in front of 90, so -1 is listed before 1 in the kernel correspondingly.)\nCommon Image Processing Kernels\n#\nPrewitt operator\n: Rather than only relying on four directly adjacent neighbors, the Prewitt operator utilizes eight surrounding pixels for smoother results.\n$$\n\\mathbf{G}_x = \\begin{bmatrix}\n-1 & 0 & +1 \\\\\n-1 & 0 & +1 \\\\\n-1 & 0 & +1\n\\end{bmatrix} \\ast \\mathbf{A} \\text{ and }\n\\mathbf{G}_y = \\begin{bmatrix}\n+1 & +1 & +1 \\\\\n0 & 0 & 0 \\\\\n-1 & -1 & -1\n\\end{bmatrix} \\ast \\mathbf{A}\n$$\nSobel operator\n: To emphasize the impact of directly adjacent pixels more, they get assigned with higher weights.\n$$\n\\mathbf{G}_x = \\begin{bmatrix}\n-1 & 0 & +1 \\\\\n-2 & 0 & +2 \\\\\n-1 & 0 & +1\n\\end{bmatrix} \\ast \\mathbf{A} \\text{ and }\n\\mathbf{G}_y = \\begin{bmatrix}\n+1 & +2 & +1 \\\\\n0 & 0 & 0 \\\\\n-1 & -2 & -1\n\\end{bmatrix} \\ast \\mathbf{A}\n$$\nDifferent kernels are created for different goals, such as edge detection, blurring, sharpening and many more. Check\nthis wiki page\nfor more examples and references.\nExample: Manu in 2004\n#\nLet’s run a simple experiment on the photo of Manu Ginobili in 2004 [[Download Image]({{ ‘/assets/data/manu-2004.jpg’ | relative_url }}){:target=\"_blank\"}] when he still had a lot of hair. For simplicity, the photo is converted to grayscale first. For colored images, we just need to repeat the same process in each color channel respectively.\nManu Ginobili in 2004 with hair. (Image source:\nManu Ginobili's bald spot through the years\n)\nimport\nnumpy\nas\nnp\nimport\nscipy\nimport\nscipy.signal\nas\nsig\n# With mode=\"L\", we force the image to be parsed in the grayscale, so it is\n# actually unnecessary to convert the photo color beforehand.\nimg\n=\nscipy\n.\nmisc\n.\nimread(\n\"manu-2004.jpg\"\n, mode\n=\n\"L\"\n)\n# Define the Sobel operator kernels.\nkernel_x\n=\nnp\n.\narray([[\n-\n1\n,\n0\n,\n1\n],[\n-\n2\n,\n0\n,\n2\n],[\n-\n1\n,\n0\n,\n1\n]])\nkernel_y\n=\nnp\n.\narray([[\n1\n,\n2\n,\n1\n], [\n0\n,\n0\n,\n0\n], [\n-\n1\n,\n-\n2\n,\n-\n1\n]])\nG_x\n=\nsig\n.\nconvolve2d(img, kernel_x, mode\n=\n'same'\n)\nG_y\n=\nsig\n.\nconvolve2d(img, kernel_y, mode\n=\n'same'\n)\n# Plot them!\nfig\n=\nplt\n.\nfigure()\nax1\n=\nfig\n.\nadd_subplot(\n121\n)\nax2\n=\nfig\n.\nadd_subplot(\n122\n)\n# Actually plt.imshow() can handle the value scale well even if I don't do\n# the transformation (G_x + 255) / 2.\nax1\n.\nimshow((G_x\n+\n255\n)\n/\n2\n, cmap\n=\n'gray'\n); ax1\n.\nset_xlabel(\n\"Gx\"\n)\nax2\n.\nimshow((G_y\n+\n255\n)\n/\n2\n, cmap\n=\n'gray'\n); ax2\n.\nset_xlabel(\n\"Gy\"\n)\nplt\n.\nshow()\nApply Sobel operator kernel on the example image.\nYou might notice that most area is in gray. Because the difference between two pixel is between -255 and 255 and we need to convert them back to [0, 255] for the display purpose.\nA simple linear transformation ($\\mathbf{G}$ + 255)/2 would interpret all the zeros (i.e., constant colored background shows no change in gradient) as 125 (shown as gray).\nHistogram of Oriented Gradients (HOG)\n#\nThe Histogram of Oriented Gradients (HOG) is an efficient way to extract features out of the pixel colors for building an object recognition classifier. With the knowledge of image gradient vectors, it is not hard to understand how HOG works. Let’s start!\nHow HOG works\n#\nPreprocess the image, including resizing and color normalization.\nCompute the gradient vector of every pixel, as well as its magnitude and direction.\nDivide the image into many 8x8 pixel cells. In each cell, the magnitude values of these 64 cells are binned and cumulatively added into 9 buckets of unsigned direction (no sign, so 0-180 degree rather than 0-360 degree; this is a practical choice based on empirical experiments).\nFor better robustness, if the direction of the gradient vector of a pixel lays between two buckets, its magnitude does not all go into the closer one but proportionally split between two. For example, if a pixel’s gradient vector has magnitude 8 and degree 15, it is between two buckets for degree 0 and 20 and we would assign 2 to bucket 0 and 6 to bucket 20.\nThis interesting configuration makes the histogram much more stable when small distortion is applied to the image.\nHow to split one gradient vector's magnitude if its degress is between two degree bins. (Image source: https://www.learnopencv.com/histogram-of-oriented-gradients/)\nThen we slide a 2x2 cells (thus 16x16 pixels) block across the image. In each block region, 4 histograms of 4 cells are concatenated into one-dimensional vector of 36 values and then normalized to have an unit weight.\nThe final HOG feature vector is the concatenation of all the block vectors. It can be fed into a classifier like SVM for learning object recognition tasks.\nExample: Manu in 2004\n#\nLet’s reuse the same example image in the previous section. Remember that we have computed $\\mathbf{G}_x$ and $\\mathbf{G}_y$ for the whole image.\nN_BUCKETS\n=\n9\nCELL_SIZE\n=\n8\n# Each cell is 8x8 pixels\nBLOCK_SIZE\n=\n2\n# Each block is 2x2 cells\ndef\nassign_bucket_vals\n(m, d, bucket_vals):\nleft_bin\n=\nint(d\n/\n20.\n)\n# Handle the case when the direction is between [160, 180)\nright_bin\n=\n(int(d\n/\n20.\n)\n+\n1\n)\n%\nN_BUCKETS\nassert\n0\n<=\nleft_bin\n<\nright_bin\n<\nN_BUCKETS\nleft_val\n=\nm\n*\n(right_bin\n*\n20\n-\nd)\n/\n20\nright_val\n=\nm\n*\n(d\n-\nleft_bin\n*\n20\n)\n/\n20\nbucket_vals[left_bin]\n+=\nleft_val\nbucket_vals[right_bin]\n+=\nright_val\ndef\nget_magnitude_hist_cell\n(loc_x, loc_y):\n# (loc_x, loc_y) defines the top left corner of the target cell.\ncell_x\n=\nG_x[loc_x:loc_x\n+\nCELL_SIZE, loc_y:loc_y\n+\nCELL_SIZE]\ncell_y\n=\nG_y[loc_x:loc_x\n+\nCELL_SIZE, loc_y:loc_y\n+\nCELL_SIZE]\nmagnitudes\n=\nnp\n.\nsqrt(cell_x\n*\ncell_x\n+\ncell_y\n*\ncell_y)\ndirections\n=\nnp\n.\nabs(np\n.\narctan(cell_y\n/\ncell_x)\n*\n180\n/\nnp\n.\npi)\nbuckets\n=\nnp\n.\nlinspace(\n0\n,\n180\n, N_BUCKETS\n+\n1\n)\nbucket_vals\n=\nnp\n.\nzeros(N_BUCKETS)\nmap(\nlambda\n(m, d): assign_bucket_vals(m, d, bucket_vals),\nzip(magnitudes\n.\nflatten(), directions\n.\nflatten())\n)\nreturn\nbucket_vals\ndef\nget_magnitude_hist_block\n(loc_x, loc_y):\n# (loc_x, loc_y) defines the top left corner of the target block.\nreturn\nreduce(\nlambda\narr1, arr2: np\n.\nconcatenate((arr1, arr2)),\n[get_magnitude_hist_cell(x, y)\nfor\nx, y\nin\nzip(\n[loc_x, loc_x\n+\nCELL_SIZE, loc_x, loc_x\n+\nCELL_SIZE],\n[loc_y, loc_y, loc_y\n+\nCELL_SIZE, loc_y\n+\nCELL_SIZE],\n)]\n)\nThe following code simply calls the functions to construct a histogram and plot it.\n# Random location [200, 200] as an example.\nloc_x\n=\nloc_y\n=\n200\nydata\n=\nget_magnitude_hist_block(loc_x, loc_y)\nydata\n=\nydata\n/\nnp\n.\nlinalg\n.\nnorm(ydata)\nxdata\n=\nrange(len(ydata))\nbucket_names\n=\nnp\n.\ntile(np\n.\narange(N_BUCKETS), BLOCK_SIZE\n*\nBLOCK_SIZE)\nassert\nlen(ydata)\n==\nN_BUCKETS\n*\n(BLOCK_SIZE\n*\nBLOCK_SIZE)\nassert\nlen(bucket_names)\n==\nlen(ydata)\nplt\n.\nfigure(figsize\n=\n(\n10\n,\n3\n))\nplt\n.\nbar(xdata, ydata, align\n=\n'center'\n, alpha\n=\n0.8\n, width\n=\n0.9\n)\nplt\n.\nxticks(xdata, bucket_names\n*\n20\n, rotation\n=\n90\n)\nplt\n.\nxlabel(\n'Direction buckets'\n)\nplt\n.\nylabel(\n'Magnitude'\n)\nplt\n.\ngrid(ls\n=\n'--'\n, color\n=\n'k'\n, alpha\n=\n0.1\n)\nplt\n.\ntitle(\n\"HOG of block at [\n%d\n,\n%d\n]\"\n%\n(loc_x, loc_y))\nplt\n.\ntight_layout()\nIn the code above, I use the block with top left corner located at [200, 200] as an example and here is the final normalized histogram of this block. You can play with the code to change the block location to be identified by a sliding window.\nDemonstration of a HOG histogram for one block.\nThe code is mostly for demonstrating the computation process. There are many off-the-shelf libraries with HOG algorithm implemented, such as\nOpenCV\n,\nSimpleCV\nand\nscikit-image\n.\nImage Segmentation (Felzenszwalb’s Algorithm)\n#\nWhen there exist multiple objects in one image (true for almost every real-world photos), we need to identify a region that potentially contains a target object so that the classification can be executed more efficiently.\nFelzenszwalb and Huttenlocher (\n2004\n) proposed an algorithm for segmenting an image into similar regions using a graph-based approach. It is also the initialization method for Selective Search (a popular region proposal algorithm) that we are gonna discuss later.\nSay, we use a undirected graph $G=(V, E)$ to represent an input image. One vertex $v_i \\in V$ represents one pixel. One edge $e = (v_i, v_j) \\in E$ connects two vertices $v_i$ and $v_j$. Its associated weight $w(v_i, v_j)$ measures the dissimilarity between $v_i$ and $v_j$. The dissimilarity can be quantified in dimensions like color, location, intensity, etc. The higher the weight, the less similar two pixels are. A segmentation solution $S$ is a partition of $V$ into multiple connected components, $\\{C\\}$. Intuitively similar pixels should belong to the same components while dissimilar ones are assigned to different components.\nGraph Construction\n#\nThere are two approaches to constructing a graph out of an image.\nGrid Graph\n: Each pixel is only connected with surrounding neighbours (8 other cells in total). The edge weight is the absolute difference between the intensity values of the pixels.\nNearest Neighbor Graph\n: Each pixel is a point in the feature space (x, y, r, g, b), in which (x, y) is the pixel location and (r, g, b) is the color values in RGB. The weight is the Euclidean distance between two pixels’ feature vectors.\nKey Concepts\n#\nBefore we lay down the criteria for a good graph partition (aka image segmentation), let us define a couple of key concepts:\nInternal difference\n: $Int(C) = \\max_{e\\in MST(C, E)} w(e)$, where $MST$ is the minimum spanning tree of the components. A component $C$ can still remain connected even when we have removed all the edges with weights < $Int(C)$.\nDifference between two components\n: $Dif(C_1, C_2) = \\min_{v_i \\in C_1, v_j \\in C_2, (v_i, v_j) \\in E} w(v_i, v_j)$. $Dif(C_1, C_2) = \\infty$ if there is no edge in-between.\nMinimum internal difference\n: $MInt(C_1, C_2) = min(Int(C_1) + \\tau(C_1), Int(C_2) + \\tau(C_2))$, where $\\tau(C) = k / \\vert C \\vert$ helps make sure we have a meaningful threshold for the difference between components. With a higher $k$, it is more likely to result in larger components.\nThe quality of a segmentation is assessed by a pairwise region comparison predicate defined for given two regions $C_1$ and $C_2$:\n$$\nD(C_1, C_2) = \n\\begin{cases}\n \\text{True} & \\text{ if } Dif(C_1, C_2) > MInt(C_1, C_2) \\\\\n \\text{False} & \\text{ otherwise}\n\\end{cases}\n$$\nOnly when the predicate holds True, we consider them as two independent components; otherwise the segmentation is too fine and they probably should be merged.\nHow Image Segmentation Works\n#\nThe algorithm follows a bottom-up procedure. Given $G=(V, E)$ and $|V|=n, |E|=m$:\nEdges are sorted by weight in ascending order, labeled as $e_1, e_2, \\dots, e_m$.\nInitially, each pixel stays in its own component, so we start with $n$ components.\nRepeat for $k=1, \\dots, m$:\nThe segmentation snapshot at the step $k$ is denoted as $S^k$.\nWe take the k-th edge in the order, $e_k = (v_i, v_j)$.\nIf $v_i$ and $v_j$ belong to the same component, do nothing and thus $S^k = S^{k-1}$.\nIf $v_i$ and $v_j$ belong to two different components $C_i^{k-1}$ and $C_j^{k-1}$ as in the segmentation $S^{k-1}$, we want to merge them into one if $w(v_i, v_j) \\leq MInt(C_i^{k-1}, C_j^{k-1})$; otherwise do nothing.\nIf you are interested in the proof of the segmentation properties and why it always exists, please refer to the\npaper\n.\nAn indoor scene with segmentation detected by the grid graph construction in Felzenszwalb's graph-based segmentation algorithm (k=300).\nExample: Manu in 2013\n#\nThis time I would use the photo of old Manu Ginobili in 2013 [[Image]({{ ‘/assets/data/manu-2013.jpg’ | relative_url }})] as the example image when his bald spot has grown up strong. Still for simplicity, we use the picture in grayscale.\nManu Ginobili in 2013 with bald spot. (Image source:\nManu Ginobili's bald spot through the years\n)\nRather than coding from scratch, let us apply\nskimage.segmentation.felzenszwalb\nto the image.\nimport\nskimage.segmentation\nfrom\nmatplotlib\nimport\npyplot\nas\nplt\nimg2\n=\nscipy\n.\nmisc\n.\nimread(\n\"manu-2013.jpg\"\n, mode\n=\n\"L\"\n)\nsegment_mask1\n=\nskimage\n.\nsegmentation\n.\nfelzenszwalb(img2, scale\n=\n100\n)\nsegment_mask2\n=\nskimage\n.\nsegmentation\n.\nfelzenszwalb(img2, scale\n=\n1000\n)\nfig\n=\nplt\n.\nfigure(figsize\n=\n(\n12\n,\n5\n))\nax1\n=\nfig\n.\nadd_subplot(\n121\n)\nax2\n=\nfig\n.\nadd_subplot(\n122\n)\nax1\n.\nimshow(segment_mask1); ax1\n.\nset_xlabel(\n\"k=100\"\n)\nax2\n.\nimshow(segment_mask2); ax2\n.\nset_xlabel(\n\"k=1000\"\n)\nfig\n.\nsuptitle(\n\"Felsenszwalb's efficient graph based image segmentation\"\n)\nplt\n.\ntight_layout()\nplt\n.\nshow()\nThe code ran two versions of Felzenszwalb’s algorithms as shown in The left k=100 generates a finer-grained segmentation with small regions where Manu’s bald spot is identified. The right one k=1000 outputs a coarser-grained segmentation where regions tend to be larger.\nFelsenszwalb's efficient graph-based image segmentation is applied on the photo of Manu in 2013.\nSelective Search\n#\nSelective search is a common algorithm to provide region proposals that potentially contain objects. It is built on top of the image segmentation output and use region-based characteristics (NOTE: not just attributes of a single pixel) to do a bottom-up hierarchical grouping.\nHow Selective Search Works\n#\nAt the initialization stage, apply Felzenszwalb and Huttenlocher’s graph-based image segmentation algorithm to create regions to start with.\nUse a greedy algorithm to iteratively group regions together:\nFirst the similarities between all neighbouring regions are calculated.\nThe two most similar regions are grouped together, and new similarities are calculated between the resulting region and its neighbours.\nThe process of grouping the most similar regions (Step 2) is repeated until the whole image becomes a single region.\nThe detailed algorithm of Selective Search.\nConfiguration Variations\n#\nGiven two regions $(r_i, r_j)$, selective search proposed four complementary similarity measures:\nColor\nsimilarity\nTexture\n: Use algorithm that works well for material recognition such as\nSIFT\n.\nSize\n: Small regions are encouraged to merge early.\nShape\n: Ideally one region can fill the gap of the other.\nBy (i) tuning the threshold $k$ in Felzenszwalb and Huttenlocher’s algorithm, (ii) changing the color space and (iii) picking different combinations of similarity metrics, we can produce a diverse set of Selective Search strategies. The version that produces the region proposals with best quality is configured with (i) a mixture of various initial segmentation proposals, (ii) a blend of multiple color spaces and (iii) a combination of all similarity measures. Unsurprisingly we need to balance between the quality (the model complexity) and the speed.\nCited as:\n@article{weng2017detection1,\n title = \"Object Detection for Dummies Part 1: Gradient Vector, HOG, and SS\",\n author = \"Weng, Lilian\",\n journal = \"lilianweng.github.io\",\n year = \"2017\",\n url = \"https://lilianweng.github.io/posts/2017-10-29-object-recognition-part-1/\"\n}\nReferences\n#\n[1] Dalal, Navneet, and Bill Triggs.\n“Histograms of oriented gradients for human detection.”\nComputer Vision and Pattern Recognition (CVPR), 2005.\n[2] Pedro F. Felzenszwalb, and Daniel P. Huttenlocher.\n“Efficient graph-based image segmentation.”\nIntl. journal of computer vision 59.2 (2004): 167-181.\n[3]\nHistogram of Oriented Gradients by Satya Mallick\n[4]\nGradient Vectors by Chris McCormick\n[5]\nHOG Person Detector Tutorial by Chris McCormick", "repair_version": 2, "repaired_at": "2026-10-14T11:44:55Z", "title": "Object Detection for Dummies Part 1: Gradient Vector, HOG, and SS | Lil'Log", "audio": [], "code_blocks": [ { "file": "attachments/6d3d9dca0d80/code/body-000.txt", "chars": 239, "language": "text" }, { "file": "attachments/6d3d9dca0d80/code/body-001.txt", "chars": 840, "language": "text" }, { "file": "attachments/6d3d9dca0d80/code/body-002.txt", "chars": 1510, "language": "text" }, { "file": "attachments/6d3d9dca0d80/code/body-003.txt", "chars": 655, "language": "text" }, { "file": "attachments/6d3d9dca0d80/code/body-004.txt", "chars": 547, "language": "text" }, { "file": "attachments/6d3d9dca0d80/code/body-005.txt", "chars": 283, "language": "text" } ], "papers": [ { "status": "download_error", "url": "http://cvcl.mit.edu/SUNSeminar/Felzenszwalb_IJCV04.pdf", "reason": "ConnectTimeout: HTTPConnectionPool(host='cvcl.mit.edu', port=80): Max retries exceeded with url: /SUNSeminar/Felzenszwalb_IJCV04.pdf (Caused by ConnectTimeoutError(