![Revisit consent button](../images/body-df29beacd7496220cc94.svg) We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg) We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data. * Cookie BCTempID * Duration 10 minutes * Description No description available. * Cookie __cf_bm * Duration 1 hour * Description This cookie, set by Cloudflare, is used to support Cloudflare Bot Management. * Cookie AWSALBCORS * Duration 7 days * Description Amazon Web Services set this cookie for load balancing. * Cookie _cfuvid * Duration session * Description Cloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services * Cookie li_gc * Duration 6 months * Description Linkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes. * Cookie __hssrc * Duration session * Description This cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session. * Cookie __hssc * Duration 1 hour * Description HubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie. * Cookie wpEmojiSettingsSupports * Duration session * Description WordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly. * Cookie BCSessionID * Duration 1 year 1 month 4 days * Description Blueconic sets this cookie as a unique identifier for the BlueConic profile. * Cookie _octo * Duration 1 year * Description No description available. * Cookie logged_in * Duration 1 year * Description No description available. * Cookie __Secure-YEC * Duration past * Description YouTube sets this cookie to stores the user's video player preferences using embedded YouTube video * Cookie __eoi * Duration 6 months * Description Description is currently not available. * Cookie AWSALBTGCORS * Duration 7 days * Description No description available. * Cookie login-status-p * Duration past * Description Description is currently not available. * Cookie AWSALBTG * Duration 7 days * Description No description available. * Cookie csrf_token * Duration session * Description No description available. * Cookie token_v2 * Duration 1 day * Description Description is currently not available. * Cookie D * Duration 1 year * Description Description is currently not available. * Cookie PHPSESSID * Duration session * Description This cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed. * Cookie VISITOR_PRIVACY_METADATA * Duration 6 months * Description YouTube sets this cookie to store the user's cookie consent state for the current domain. * Cookie cookietest * Duration session * Description The cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience. * Cookie __Host-airtable-session * Duration 1 year * Description This cookie is used to enable us to integrate the services of Airtable. * Cookie __Host-airtable-session.sig * Duration 1 year * Description This cookie is used to enable us to integrate the services of Airtable. * Cookie m * Duration 1 year 1 month 4 days * Description Stripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly. * Cookie BIGipServer* * Duration session * Description Marketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. * Cookie __cfruid * Duration session * Description Cloudflare sets this cookie to identify trusted web traffic. * Cookie _GRECAPTCHA * Duration 6 months * Description Google Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks. * Cookie __Secure-YNID * Duration 6 months * Description Google cookie used to protect user security and prevent fraud, especially during the login process. * Cookie cookieyes-consent * Duration 1 year * Description CookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors. Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features. * Cookie lidc * Duration 1 day * Description LinkedIn sets the lidc cookie to facilitate data center selection. * Cookie brw * Duration 1 year * Description No description available. * Cookie brwConsent * Duration 5 minutes * Description Description is currently not available. * Cookie WMF-Uniq * Duration 1 year * Description Description is currently not available. * Cookie loom_anon_comment * Duration 1 year * Description No description available. * Cookie loom_referral_video * Duration session * Description Description is currently not available. * Cookie VISITOR_INFO1_LIVE * Duration 6 months * Description A cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface. * Cookie yt-remote-connected-devices * Duration Never Expires * Description YouTube sets this cookie to store the user's video preferences using embedded YouTube videos. * Cookie ytidb::LAST_RESULT_ENTRY_KEY * Duration Never Expires * Description The cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future. * Cookie yt-remote-device-id * Duration Never Expires * Description YouTube sets this cookie to store the user's video preferences using embedded YouTube videos. * Cookie yt-remote-session-name * Duration session * Description The yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video. * Cookie yt-remote-fast-check-period * Duration session * Description The yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos. * Cookie yt-remote-session-app * Duration session * Description The yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player. * Cookie yt-remote-cast-available * Duration session * Description The yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player. * Cookie yt-remote-cast-installed * Duration session * Description The yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video. * Cookie cp_session * Duration 3 months * Description Codepen sets this cookie for Help systems found in the website. * Cookie loid * Duration 1 year 1 month 4 days * Description This cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform. Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc. * Cookie __hstc * Duration 6 months * Description Hubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session). * Cookie hubspotutk * Duration 6 months * Description HubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts. * Cookie _ga * Duration 1 year 1 month 4 days * Description Google Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors. * Cookie _ga_* * Duration 1 year 1 month 4 days * Description Google Analytics sets this cookie to store and count page views. * Cookie __Host-psifi.analyticsTrace * Duration 6 hours * Description Description is currently not available. * Cookie __Host-psifi.analyticsTraceV2 * Duration 6 hours * Description Description is currently not available. * Cookie _gh_sess * Duration session * Description GitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form. * Cookie YSC * Duration session * Description YSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages. * Cookie ajs_anonymous_id * Duration 1 year * Description This cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before. * Cookie vuid * Duration 1 year 1 month 4 days * Description Vimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors. * Cookie AWSALB * Duration 7 days * Description AWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target. * Cookie acq * Duration past * Description Description is currently not available. * Cookie acq.sig * Duration past * Description Description is currently not available. * Cookie ptc * Duration 2 years * Description No description available. Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns. * Cookie muc_ads * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to collect user behaviour and interaction data to optimize the website. * Cookie guest_id_marketing * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to identify and track the website visitor. * Cookie guest_id_ads * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to identify and track the website visitor. * Cookie personalization_id * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting. * Cookie guest_id * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences. * Cookie bcookie * Duration 1 year * Description LinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs. * Cookie __Secure-ROLLOUT_TOKEN * Duration 6 months * Description YouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment. * Cookie yt.innertube::nextId * Duration Never Expires * Description YouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen. * Cookie yt.innertube::requests * Duration Never Expires * Description YouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen. * Cookie session_tracker * Duration session * Description This cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website. * Cookie edgebucket * Duration session * Description Reddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content. * Cookie did * Duration 1 year * Description Arbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year. Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation [](https://towardsdatascience.com/) Publish AI, ML & data-science insights to a global community of data professionals. Sign in[Submit an Article](https://write.insightmediagroup.io) [Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/) * * * [Write For TDS](https://towardsdatascience.com/submissions/) [LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search [Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/) # FPN Paper Walkthrough: Leveraging the Internal Pyramid Understanding how FPN allows deep learning models detecting small objects and how to implement it from scratch [Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/) June 4, 202618 min read ![](../images/000.jpg) Photo by In my previous article I was talking about YOLOv3 [1]. One of the factors that makes this YOLO version better than its predecessors is its ability in detecting small objects thanks to its FPN-like neck adopted by the model. Unfortunately, my explanation about FPN in that article was not quite thorough since I was focusing more on YOLOv3 itself. Thus, in this article I decided to write specifically about FPN from its original paper titled “ _Feature Pyramid Networks for Object Detection_ ” [2] so that you can get a better understanding of what it actually is and how it works. Not only that, here I will also demonstrate how to implement FPN from scratch and how to connect it with a CNN backbone and an RPN head. ··· ## Backbone, Neck, and Head Before we get into FPN, we first need to know that the structure of an object detection model is different from that of the classification model, in which the main difference lies in the very last layer. In a typical classification model, the last layer comprises a number of neurons where each of them corresponds to every single class available in the dataset. Or, in the case of binary classification, the output layer only consists of a single neuron, which is responsible to predict whether a sample belongs to class 0 or 1. This kind of output layer is actually not suitable for detection task since it also requires neurons dedicated for predicting the location and the size of an object in addition to its class. So, in order for a model to be able to predict object location and size, we need to replace the output layer, i.e., the _classification head_ , with the so-called _detection head_. The remaining layers themselves (everything except the _head_) are commonly called _backbone_. Some models that use this structure are YOLOv1 and YOLOv2, where they use a stack of convolution layers as the _backbone_ and a specific _head_ for predicting object location and size within an image as well as its class. Learn this step by step with the interactive [Machine Learning](https://roadmap.sh/machine-learning) roadmap. Older object detection models like YOLOv1 and YOLOv2 mentioned above only consist of a _backbone_ and a _head_. As time went on, researchers found that this structure is still not quite optimal, hence they finally came up with an idea by adding a new component called _neck_. As the name suggests, this is essentially something we place between the _backbone_ and the _head_. And FPN, which we are going to talk about in this article, is one of the earliest _necks_ proposed for object detection models. Look at the Figure 1 below to see the high-level architectural view of the older and modern object detection models. ![](../images/001.png) Figure 1. The architecture of an object detection model in general [3]. The _backbone_ of a model is mainly responsible for performing feature extraction, while the _neck_ is useful for enhancing feature quality, and the _head_ is for making predictions. Based on this notion, we can say that by applying FPN, a network can potentially achieve better accuracy thanks to the feature enhancement mechanism performed by the _neck_. ### The Evolution of Multi-Scale Detection Mechanism Previously I mentioned that using _backbone_ and detection _head_ without _neck_ is not quite optimal. This is especially regarding its capability in detecting small objects. Now let’s take a look at the Figure 2 below. The first two YOLO versions I mentioned earlier utilize the structure in image (b), where the bounding box and the object class predictions are made solely on top of the feature map produced by the deepest layer in the backbone. This method is valid, but is effective only for large objects. The reason is pretty straightforward: as an image gets deeper into a network, the spatial dimension shrinks, and more importantly, the pixel information contained in the deeper feature maps becomes a representation of several neighboring pixels in the shallower ones, which causes the spatial information to blend. By doing this, feature maps from deeper layers got a large receptive field, allowing large objects to be detected and recognized easily. However, the degradation of spatial information as we get deeper prevents us from detecting small objects accurately because we do need a detailed pixel location in order to predict the exact coordinates of the objects. Additionally, the receptive field size of a feature map is positively correlated with the amount of semantic information it contains. In the figure below, a feature map of high semantic information is indicated by a thick blue outline. This is essentially why the deepest feature map in (b) has the thickest outline. ![](../images/body-e4168a51bb6e659fc555.png) Figure 2. Comparison of different feature pyramid architectures [2]. The most straightforward approach to allow a network to simultaneously detect large and small objects is by using _featurized image pyramid_ (a). This method is able to achieve high accuracy because we can make predictions from different image resolutions. What’s essentially done here is that we rescale our input image into multiple scales, perform feature extraction independently on each scale, and make predictions on the resulting feature maps. The smaller feature map is responsible to detect large objects, whereas the larger one is specialized to detect small objects thanks to its detailed spatial information. However, this method is computationally expensive since we need to process multiple raw images of different scales at once. Another solution was proposed by the authors of SSD (Single Shot Multibox Detector), which in Figure 2 above is the one referred to as _pyramidal feature hierarchy_ (c). So, instead of feeding the network with the same image of different sizes, the authors of SSD attempted to use only the largest image and utilize the internal pyramidal structure of the CNN backbone to make predictions of varying scales. This approach allows the system to be computationally more feasible than option (a). Nevertheless, here we actually got a tradeoff like (b), where the feature map from the deeper layer contains a large amount of semantic information yet having minimal spatial information, while the feature map from the shallower layer has a lot of spatial information but it does not have that much semantic information. It is important to note that a detailed spatial information might not be quite important for large objects as we can just approximate the general shape of that object. However, both spatial and semantic information are necessary for detecting small objects because not only the detailed coordinates, but the model also needs to understand what’s actually inside the bounding box. So, while it is true that method (c) is indeed able to detect both large and small objects, but its ability in detecting the latter is not yet optimal. And here’s where FPN comes as a solution. If we take a look at image (d) in Figure 2, we can see that the predictions are made on top of the corresponding feature maps which are all semantically rich. This essentially allows objects of varying scales, including the smaller ones, to be detected accurately. We are going to talk about the details of how FPN enriches feature maps in the subsequent section. ··· ## How FPN Works The idea of FPN is to inject information from the deeper feature maps into the shallower ones, and by doing so we will have the shallower feature maps not only containing high spatial information but also high semantic information coming from the deeper part of the network. In theory, this should result in a better detection accuracy on small objects since the large feature maps are now enriched with a lot of semantic information. In order to achieve this, they introduce the so-called _top-down pathway_ and _lateral connections_. You can see the complete FPN architecture in Figure 3 below, which is essentially the detailed version of the one in Figure 2 (d). ![](../images/body-76aa4d1ace7e6b83a894.png) Figure 3. The detailed FPN architecture [3]. The authors of this paper decided to use ResNet-50 and ResNet-101 as the backbone. Suppose we were to use the former, we would later have the _conv2_ , _conv3_ , _conv4_ , and _conv5_ layers repeated 3, 4, 6, and 3 times, respectively, as suggested by the architectural details of ResNet in Figure 4. _C2_ , _C3_ , _C4_ , and _C5_ , which are the tensors produced by the last layer of the corresponding stage, are going to be transferred to the _top-down pathway_ through _lateral connections_ , i.e., the arrows going out from the backbone. ![](../images/body-6e5a7f00da4be75aea9a.png) Figure 4. The ResNet architecture [3, 4]. The _top-down pathway_ is used to transfer semantic information from deeper layers, whereas _lateral connections_ are used to preserve spatial information. We aggregate the two by performing element-wise summation, which the detailed process is given in Figure 5 below. For the tensors that come from the backbone (_C_), we first need to apply 1×1 conv to them. This convolution layer is responsible for adjusting the number of channels so that it matches with the tensor coming from the top-down pathway. The tensor from the top-down pathway itself (_M+1_) undergoes 2× nearest-neighbor upsampling. These processes are essentially done because we need both tensors to have the exact same dimension so that element-wise summation can be performed. As the summation is done, the resulting tensor is now referred to as _M_. This tensor has some aliasing effect due to the upsampling process we did earlier, hence we need to apply a 3×3 convolution to reduce that effect. Finally, we got the _P_ tensor, which is ready to be forwarded to the detection head. ![](../images/body-cfb1a470dcca88dcafe2.png) Figure 5. How feature maps from lateral connection (C) and the top-down pathway (M+1) are aggregated [3]. Keep in mind that all processes described in Figure 5 above only apply to _M2_ , _M3_ , and _M4_. Computing _M5_ is actually much simpler (see Figure 6), where the only thing we need to do is just to adjust the number of channels using 1×1 conv to make it uniform with the tensors in the other lateral connections. The _M5_ tensor itself does not need to be processed further with 3×3 conv because there is nothing to be smoothed out due to the absence of upsampling mechanism. And so, we can basically say that _P5_ is the exact same tensor as _M5_. ![](../images/body-076dd85e523657447a62.png) Figure 6. The way to compute M5 and P5 is slightly different from that of M2-M4 and P2-P4 [3]. And well I think that’s everything about the theory behind FPN. In the next section I am going to bring you into the lower-level view of the architecture by implementing it from scratch with PyTorch. ··· ## FPN From Scratch ### CNN Backbone So as seen in the Codeblock 1 below, the very first thing we need to do in the code is to import the required modules. python ``` # Codeblock 1 import torch import torch.nn as nn ``` Since the focus of this article is FPN, here I will use a dummy model for the backbone instead of using the actual ResNet in order to simplify things. But still, the layers in the code are named according to Figure 3 and 4: `conv1`, `conv2`, `conv3`, `conv4` and `conv5` as shown in Codeblock 2 below. The output tensor dimension of each stage is also set according to the original ResNet architecture. So, although this backbone is just a plain CNN-based model, you can think of this like a normal ResNet. Next, what we do inside the `forward()` method is to connect all the layers. If you take a closer look at the code, you will notice that each convolution layer is followed by a ReLU activation function and a maxpooling layer. The maxpooling layer itself is set to have the stride of 2, effectively halves the spatial dimension of the feature map. By repeating maxpooling layers multiple times, we will have our feature map gradually gets smaller as we get deeper into the network. This essentially creates a pyramidal structure within the CNN backbone which is leveraged by FPN to achieve high detection accuracy on varying object scales. In CNN, reducing spatial dimension like this is a standard practice to reduce computational complexity to compensate the increase of the number of channels. Still within the `forward()` method, don’t forget to clone the main tensor `x` as shown at the lines marked with `#(1)`, `#(2)`, and `#(3)`. The copied tensors, which are named `c2`, `c3`, and `c4`, will then be the return values of the `CNN` class alongside the feature map from the main flow (`c5`) (`#(4)`). python ``` # Codeblock 2 class CNN(nn.Module): def __init__(self): super().__init__() self.relu = nn.ReLU() self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1) self.conv1 = nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3, padding=1) self.conv2 = nn.Conv2d(in_channels=64, out_channels=256, kernel_size=3, padding=1) self.conv3 = nn.Conv2d(in_channels=256, out_channels=512, kernel_size=3, padding=1) self.conv4 = nn.Conv2d(in_channels=512, out_channels=1024, kernel_size=3, padding=1) self.conv5 = nn.Conv2d(in_channels=1024, out_channels=2048, kernel_size=3, padding=1) def forward(self, x): print(f'original\t: {x.size()}\n') x = self.relu(self.conv1(x)) print(f'after conv1\t: {x.size()}') x = self.maxpool(x) print(f'after maxpool\t: {x.size()}\n') x = self.relu(self.conv2(x)) print(f'after conv2\t: {x.size()}') x = self.maxpool(x) print(f'after maxpool\t: {x.size()}\n') c2 = x.clone() #(1) x = self.relu(self.conv3(x)) print(f'after conv3\t: {x.size()}') x = self.maxpool(x) print(f'after maxpool\t: {x.size()}\n') c3 = x.clone() #(2) x = self.relu(self.conv4(x)) print(f'after conv4\t: {x.size()}') x = self.maxpool(x) print(f'after maxpool\t: {x.size()}\n') c4 = x.clone() #(3) x = self.relu(self.conv5(x)) print(f'after conv5\t: {x.size()}') c5 = self.maxpool(x) print(f'after maxpool\t: {c5.size()}\n') return c2, c3, c4, c5 #(4) ``` As the `CNN` class is done, we will now try to pass a dummy RGB image of size 224×224 through the network. This tensor dimension is chosen based on the input shape of the original ResNet. python ``` # Codeblock 3 cnn = CNN() x = torch.randn(1, 3, 224, 224) out_cnn = cnn(x) ``` And below is what the output looks like. Here we can see that the number of channels after each conv layer matches exactly with the ResNet structure given in Figure 4. Not only that, it is also seen that the spatial dimension of our dummy tensor successfully halved after each stage thanks to the maxpooling layers. This essentially indicates that our simple CNN model really mimics the general structure of a ResNet model. markup ``` # Codeblock 3 Outputoriginal : torch.Size([1, 3, 224, 224])after conv1 : torch.Size([1, 64, 224, 224])after maxpool : torch.Size([1, 64, 112, 112])after conv2 : torch.Size([1, 256, 112, 112])after maxpool : torch.Size([1, 256, 56, 56])after conv3 : torch.Size([1, 512, 56, 56])after maxpool : torch.Size([1, 512, 28, 28])after conv4 : torch.Size([1, 1024, 28, 28])after maxpool : torch.Size([1, 1024, 14, 14])after conv5 : torch.Size([1, 2048, 14, 14])after maxpool : torch.Size([1, 2048, 7, 7]) ``` We can also check what the returned tensors look like by running the code below. You can see in the resulting output that the `c2` tensor has the shape of 256×56×56, `c3` is of shape 512×28×28, and so on. By the way, you can just ignore the number 1 in the 0th axis since it only indicates the number of samples we pass within a single batch. python ``` # Codeblock 4 c2, c3, c4, c5 = out_cnn print(c2.shape) print(c3.shape) print(c4.shape) print(c5.shape) ``` markup ``` # Codeblock 4 Outputtorch.Size([1, 256, 56, 56])torch.Size([1, 512, 28, 28])torch.Size([1, 1024, 14, 14])torch.Size([1, 2048, 7, 7]) ``` ··· ### FPN Neck As the CNN backbone is completed, now let’s move on to the FPN neck. In the Codeblock 5 below, we first initialize the upsample layer (`#(1)`) which we will use every time we want to double the spatial dimension of the _M_ tensor. Here I set the `mode` parameter to `nearest` as suggested in the paper, which is actually a very simple interpolation method, allowing the process to be fast. Take a look at Figure 7 to see what a nearest-neighbor interpolation looks like. python ``` # Codeblock 5 class FPN(nn.Module): def __init__(self): super().__init__() self.upsample = nn.Upsample(scale_factor=2, mode='nearest') #(1) self.lateral_c5 = nn.Conv2d(in_channels=2048, out_channels=256, kernel_size=1) self.lateral_c4 = nn.Conv2d(in_channels=1024, out_channels=256, kernel_size=1) self.lateral_c3 = nn.Conv2d(in_channels=512, out_channels=256, kernel_size=1) self.lateral_c2 = nn.Conv2d(in_channels=256, out_channels=256, kernel_size=1) self.smooth_m4 = nn.Conv2d(in_channels=256, out_channels=256, kernel_size=3, padding=1) self.smooth_m3 = nn.Conv2d(in_channels=256, out_channels=256, kernel_size=3, padding=1) self.smooth_m2 = nn.Conv2d(in_channels=256, out_channels=256, kernel_size=3, padding=1) def forward(self, c2, c3, c4, c5): m5 = self.lateral_c5(c5) p5 = m5 m4 = self.upsample(m5) + self.lateral_c4(c4) p4 = self.smooth_m4(m4) m3 = self.upsample(m4) + self.lateral_c3(c3) p3 = self.smooth_m3(m3) m2 = self.upsample(m3) + self.lateral_c2(c2) p2 = self.smooth_m2(m2) return p2, p3, p4, p5 ``` ![](../images/body-fea0ee17592f14e8b54e.png) Figure 7. An example of a 2x upsampling process with nearest-neighbor interpolation method [3]. If you go back to Codeblock 4, you will see that the `c{5,4,3,2}`tensors returned by the backbone have different number of channels. This is basically the reason that we initialize the `lateral_c{5,4,3,2}`layers to process these tensors so that the resulting channel counts will be uniform. According to the paper, we need to set these convolution layers to produce 256 output channels, which is the reason why we use that number for the `out_channels`parameter. Next, based on the Figure 7, you can just imagine how pixelated the resulting feature maps are after being upsampled. Thus, we need to process the `m{4,3,2}`tensors further with the 3×3 conv layers which I refer to as `smooth_m{4,3,2}`. As all layers have been initialized, what we need to do next is to assemble them in the `forward()`method according to the structure I showed you earlier in Figure 3. In addition to this, the paper also mentions that we don’t need to implement any nonlinearities within the FPN, which is the reason that all the convolution layers in the `FPN`class above are not followed with ReLU. Now in Codeblock 6 below I try to pass the _C_ tensors we obtained earlier through the FPN neck we just created. We can see in the output that the resulting tensors have different spatial resolutions. Later on, the `p2`tensor (the one that has 56×56 dimension) will be forwarded to a detection head to detect small objects, whereas `p5`(the 7×7 tensor) is going to be responsible for large objects. python ``` # Codeblock 6 fpn = FPN() out_fpn = fpn(c2, c3, c4, c5) p2, p3, p4, p5 = out_fpn print(p2.shape) print(p3.shape) print(p4.shape) print(p5.shape) ``` markup ``` # Codeblock 6 Outputtorch.Size([1, 256, 56, 56])torch.Size([1, 256, 28, 28])torch.Size([1, 256, 14, 14])torch.Size([1, 256, 7, 7]) ``` Here have already completed the FPN part. Remember that FPN is just the _neck_ of a detection model, which essentially means that at this point we still haven’t got the bounding box prediction just yet. In order to actually obtain the prediction result, we need to connect a specific _head_ to the FPN, and in this case I will use the RPN (Region Proposal Network) head. ··· ### RPN Head In case you’re still not yet familiar with RPN, this is essentially the head of an object detection model used for creating bounding box, which was first proposed in the Faster R-CNN paper. Note that while in this demonstration we refer to the RPN as a head, keep in mind that it is actually not a complete detection head since it has no capability in performing classification on the detected objects. We can see in the RPN architecture below that it utilizes the so-called _cls layer_ and _reg layer_ , which produce _objectness_ score and the bounding box coordinates, respectively. The objectness score tensor has the length of 2 _k_ , where _k_ is the number of predetermined anchor boxes and 2 is the probability of the corresponding anchor box being there. We can think of this like a binary classification treated with a one-hot representation (object/non-object). Meanwhile, the number 4 in the 4 _k_ length of the coordinates tensor simply correspond to the _xywh_ prediction. ![](../images/body-95ca0e48450b435d8fa7.png) Figure 8. The structure of RPN [4]. Going back to our code implementation, in the Codeblock 7 below we initialize the `intermediate`, `cls`, and `reg` layers within the `__init__()` method of the RPN class. Note that the intermediate layer is the only one that uses 3×3 convolution, whereas both the `cls` and `reg` layers use 1×1 convs. Regarding the number of channels, the `intermediate` layer maps the input tensor into 256 channels, while the `cls` and `reg` map it into 2 _k_ and 4 _k_ , respectively. Finally, we can simply connect these layers within the `forward()` method. python ``` # Codeblock 7 NUM_ANCHORS = 3 class RPN(nn.Module): def __init__(self): super().__init__() self.intermediate = nn.Conv2d(in_channels=256, out_channels=256, kernel_size=3, padding=1) self.cls = nn.Conv2d(in_channels=256, out_channels=NUM_ANCHORS*2, kernel_size=1) self.reg = nn.Conv2d(in_channels=256, out_channels=NUM_ANCHORS*4, kernel_size=1) def forward(self, x): x = self.intermediate(x) objectness_scores = self.cls(x) bbox_regressions = self.reg(x) return objectness_scores, bbox_regressions ``` Now let’s test if our RPN class works properly by running the Codeblock 8 below. Here I test it on the `p2` feature map we obtained from Codeblock 6. python ``` # Codeblock 8 rpn = RPN() p2_objectness, p2_bbox = rpn(p2) print(p2_objectness.shape) print(p2_bbox.shape) ``` Below is what the resulting output looks like. You can see that `p2_objectness` is a tensor having the size of 6×56×56, indicating that every single pixel in the 56×56 spatial dimension contains 6 prediction values, where the first 2 values are for the first anchor box, the second 2 values are for the second anchor box, and the last 2 values are for the third one. The similar thing also applies to the `p2_bbox` tensor, which in this case it contains the _xywh_ values. markup ``` # Codeblock 8 Outputtorch.Size([1, 6, 56, 56])torch.Size([1, 12, 56, 56]) ``` ··· ### The Entire Detection Model In Codeblock 9 below we are going to construct the entire detection model so that you can better understand how FPN works together with the other components. Here in the `__init__()` method I initialize the CNN bottleneck, FPN neck, and RPN head. In the `forward()` method, we first pass the image tensor into the CNN (`#(1)`). This backbone returns 4 tensors, which are ready to be connected to the FPN through lateral connections. Next, at line `#(2)` we feed all the _C_ tensors as the input of FPN, producing the _P_ tensors. Lastly, we use all the _P_ s as the input to the RPN (`#(3–4)`). Keep in mind that RPN shares its parameters across all detection heads, so we only need to initialize it once and use it for all feature map of different scales. python ``` # Codeblock 9 class DetectionModel(nn.Module): def __init__(self): super().__init__() self.cnn = CNN() self.fpn = FPN() self.rpn = RPN() def forward(self, x): c2, c3, c4, c5 = self.cnn(x) #(1) p2, p3, p4, p5 = self.fpn(c2, c3, c4, c5) #(2) p2_pred = self.rpn(p2) #(3) p3_pred = self.rpn(p3) p4_pred = self.rpn(p4) p5_pred = self.rpn(p5) #(4) return p2_pred, p3_pred, p4_pred, p5_pred ``` Now as the detection head is complete, we can test it with the Codeblock 10 below. Here I try to pass a dummy tensor of size 1×3×224×224, simulating a single RGB image of size 224×224 (`#(1)`). Next, we can just pass it through the `detection_model` (`#(2)`) and unpack the prediction results (`#(3–4)`). python ``` # Codeblock 10 detection_model = DetectionModel() x = torch.randn(1, 3, 224, 224) #(1) p2_pred, p3_pred, p4_pred, p5_pred = detection_model(x) #(2) p2_objectness, p2_bbox = p2_pred #(3) p3_objectness, p3_bbox = p3_pred p4_objectness, p4_bbox = p4_pred p5_objectness, p5_bbox = p5_pred #(4) print(p2_objectness.shape) print(p3_objectness.shape) print(p4_objectness.shape) print(p5_objectness.shape) print() print(p2_bbox.shape) print(p3_bbox.shape) print(p4_bbox.shape) print(p5_bbox.shape) ``` Below is what the output looks like. You can see here that the resulting tensor dimensions are as intended, where the `objectness` and `bbox` tensors contain 6 and 12 values for each grid cell, respectively. So, I believe this implementation is correct and thus ready to be trained for object detection task. markup ``` # Codeblock 10 Outputtorch.Size([1, 6, 56, 56])torch.Size([1, 6, 28, 28])torch.Size([1, 6, 14, 14])torch.Size([1, 6, 7, 7])torch.Size([1, 12, 56, 56])torch.Size([1, 12, 28, 28])torch.Size([1, 12, 14, 14])torch.Size([1, 12, 7, 7]) ``` ··· ## Ending I think that’s pretty much all about the underlying theory and the from-scratch implementation of FPN. Here I challenge you to try implementing FPN on the real ResNet instead of a dummy CNN model like I demonstrated above. I actually got a separate article about ResNet, which you can check as a reference [5]. Or, it is also possible to use other models if you want, such as VGG, ResNeXt, ConvNeXt, etc since FPN can basically work on any CNN-based backbone model. Not only that, it would also be better if you can implement YOLO-style head as a replacement of RPN, which the examples can be seen in my previous articles given at references number [6] for YOLOv1, [7] for YOLOv2, and [1] for YOLOv3. Please let me know if there are mistakes in my writing or in the code. Thanks for reading! By the way, you can find the code used in this article in my GitHub repo [8]. ··· ## References [1] [Muhammad Ardi](https://medium.com/u/9801a58700ac). YOLOv3 Paper Walkthrough: Even Better, but Not That Much. Medium. [Accessed June 1, 2026]. [2] Tsung-Yi Lin. Feature Pyramid Networks for Object Detection. Arxiv. [Accessed September 9, 2025]. [3] Image created originally by author. [4] Shaoqing Ren. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Arxiv. [Accessed September 9, 2025]. [5] Muhammad Ardi. Paper Walkthrough: Residual Network (ResNet). Python in Plain English. [Accessed September 9, 2025]. [6] [Muhammad Ardi](https://medium.com/u/9801a58700ac). YOLOv1 Paper Walkthrough: The Day YOLO First Saw the World. Medium. [Accessed June 1, 2026]. [7] [Muhammad Ardi](https://medium.com/u/9801a58700ac). YOLOv2 & YOLO9000 Paper Walkthrough: Better, Faster, Stronger. Medium. [Accessed June 1, 2026]. [8] MuhammadArdiPutra. FPN. GitHub. [Accessed September 9, 2025]. ## Register Now to Continue Reading ### Unlock Towards Data Science with a Free Account All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account * * * ![](../images/body-59e1c17daab1fcb582c5.jpeg) Written By ## [Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/) Deep learning storyteller [See all from Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/) [Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Deep Learning](https://towardsdatascience.com/tag/deep-learning/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)[Python](https://towardsdatascience.com/tag/python/) Share This Article * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Ffpn-paper-walkthrough-leveraging-the-internal-pyramid%2F&title=FPN%20Paper%20Walkthrough%3A%20Leveraging%20the%20Internal%C2%A0Pyramid) * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Ffpn-paper-walkthrough-leveraging-the-internal-pyramid%2F&title=FPN%20Paper%20Walkthrough%3A%20Leveraging%20the%20Internal%C2%A0Pyramid) * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Ffpn-paper-walkthrough-leveraging-the-internal-pyramid%2F&text=FPN%20Paper%20Walkthrough%3A%20Leveraging%20the%20Internal%C2%A0Pyramid) Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. [Write for TDS](https://towardsdatascience.com/submissions/) ## Related Articles * [![](../images/002.png)](https://towardsdatascience.com/dueling-double-deep-q-learning-using-tensorflow-2-x-7bbbcec06a2a/) ## [Dueling Double Deep Q Learning using Tensorflow 2.x](https://towardsdatascience.com/dueling-double-deep-q-learning-using-tensorflow-2-x-7bbbcec06a2a/) [Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/) Implementing dueling double deep q learning with TensorFlow 2.x [Abhishek Suran](https://towardsdatascience.com/author/abhisheksuran/) July 10, 20202 min read * [![](../images/003.jpeg)](https://towardsdatascience.com/semantic-textual-similarity-with-bert-fc800656e7a3/) ## [Semantic Textual Similarity with BERT](https://towardsdatascience.com/semantic-textual-similarity-with-bert-fc800656e7a3/) [Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/) How to use BERT to calculate the semantic similarity between two texts [Ruben Winastwan](https://towardsdatascience.com/author/marcellusruben/) February 15, 202311 min read * [![](../images/004.jpg)](https://towardsdatascience.com/stop-using-print-and-start-debugging-3cd11524c92a/) ## [Stop using Print, and start Debugging](https://towardsdatascience.com/stop-using-print-and-start-debugging-3cd11524c92a/) [Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/) Learn how to debug your Python program with Visual Studio Code [Marcello Politi](https://towardsdatascience.com/author/marcellopoliti/) December 5, 20227 min read * [![](../images/005.png)](https://towardsdatascience.com/getting-started-with-sentiment-analysis-731531ec880d/) ## [Getting Started With Sentiment Analysis](https://towardsdatascience.com/getting-started-with-sentiment-analysis-731531ec880d/) [Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/) Notes From Natural Lanaguage Processing Specialization Course1 Week1 [Kurtis Pykes](https://towardsdatascience.com/author/kurtispykes/) October 17, 20208 min read * [![](../images/006.jpg)](https://towardsdatascience.com/5-visualizations-with-python-to-show-simultaneous-changes-in-geospatial-data-ddc2eaab9d78/) ## [5 Visualizations with Python to Show Simultaneous Changes in Geospatial Data](https://towardsdatascience.com/5-visualizations-with-python-to-show-simultaneous-changes-in-geospatial-data-ddc2eaab9d78/) [Data Science](https://towardsdatascience.com/category/data-science/) Ideas to express simultaneous data value changes in multiple locations over time [Boriharn K](https://towardsdatascience.com/author/borih.k/) November 4, 20237 min read * [![](../images/007.png)](https://towardsdatascience.com/interpretable-machine-learning-using-shap-theory-and-applications-26c12f7a7f1a/) ## [Interpretable Machine Learning using SHAP - theory and applications](https://towardsdatascience.com/interpretable-machine-learning-using-shap-theory-and-applications-26c12f7a7f1a/) [Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/) Introduction [Khalil Zlaoui](https://towardsdatascience.com/author/khalilz/) October 1, 20218 min read * [![](../images/008.jpg)](https://towardsdatascience.com/tds-newsletter-how-to-build-robust-data-and-ai-systems/) ## [TDS Newsletter: How to Build Robust Data and AI Systems](https://towardsdatascience.com/tds-newsletter-how-to-build-robust-data-and-ai-systems/) [The Variable](https://towardsdatascience.com/category/the-variable/) Gain a high-level understanding of how the different pieces in your workflow come together [TDS Editors](https://towardsdatascience.com/author/towardsdatascience/) November 20, 20253 min read * [![](../images/009.jpeg)](https://towardsdatascience.com/build-a-simple-web-app-to-classify-us-50-dollar-100-dollar-bills-d4ef8298ae6/) ## [Build a Simple Web App to Classify US 50 Dollar & 100 Dollar Bills](https://towardsdatascience.com/build-a-simple-web-app-to-classify-us-50-dollar-100-dollar-bills-d4ef8298ae6/) [Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/) From model to production using Jupyter notebook [ASM Saiful Islam](https://towardsdatascience.com/author/a-islam/) January 25, 202112 min read * [![](../images/010.jpg)](https://towardsdatascience.com/uncertainty-in-deep-learning-experiments-with-bayesian-cnn-1ca37ddb6954/) ## [Uncertainty in Deep Learning: Experiments with Bayesian CNN](https://towardsdatascience.com/uncertainty-in-deep-learning-experiments-with-bayesian-cnn-1ca37ddb6954/) [Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/) How does KL-Weight affect the model? Writing a Bayesian CNN for chest X-Ray images. [Kaan Bıçakcı](https://towardsdatascience.com/author/frightera/) April 12, 20225 min read * [YouTube](https://www.youtube.com/c/TowardsDataScience) * [X](https://x.com/TDataScience) * [LinkedIn](https://www.linkedin.com/company/towards-data-science/) * [Threads](https://www.threads.net/@towardsdatascience) * [Bluesky](https://bsky.app/profile/towardsdatascience.com) [](https://towardsdatascience.com/) Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/) © Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network) #### Recommended Articles Close * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ "How Relevance Models Foreshadowed Transformers for NLP") * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ "Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1") * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ "The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs") * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ "Does Your Company Have a Data Strategy?")