404 errors happen when someone requests a page or file that doesn’t exist. Actually, it’s more accurate to say the webserver can’t find the file.
404 errors can happen when:
- Someone types the wrong address
- Someone links to the wrong address
- The file or page moves to a different address
- The page or file is deleted
- The page or file never existed
404 errors might sound scary, but they are mostly harmless. It is easy to monitor 404 errors and to fix them, thanks to Rank Math.
When you visit a website through a link, or by typing in an address, you aim to find and consume content. Instead, if you’re hit with a 404 error, not only will you not be able to consume the content, but your experience will also be less than satisfactory.
It doesn’t matter if it’s your fault, the website owner’s fault, or someone else’s. Maybe you clicked on a broken link caused by a typing mistake on another website. Maybe you cut-and-pasted a URL wrong. Or, you possibly made a spelling mistake when you typed a URL manually.
404 errors are so common that it is unlikely that you haven’t experienced them. And, as any Internet user will tell you, they are definitely an inconvenience.
When you operate your website, you make a lot of effort to give your users an excellent experience. You create amazing content, you use the best WordPress themes, you design your content, you make your website as fast as possible, and many other things. All that effort can go to waste if your user can’t even reach your content.
You might think 404 errors only affect a very small percentage of your audience, and you will eventually fix them—so there is no need to be on top of them all the time. But, you’d be wrong. What if a large publication found your content useful and decided to link to you, sending you thousands of visitors for years to come, but made a mistake while linking to you?
From your end, nothing would be wrong. You might see better traffic, but all those visitors will quickly bounce off your website—costing you thousands of subscribers, followers, and lifelong fans. Not to mention the wasted SEO value of the link you received.
Fortunately, Rank Math’s 404 Monitor can save you all sorts of SEO trouble. Let’s check it out. To access Rank Math’s 404 Monitor feature, at first, head over to the WordPress Dashboard > Rank Math > Dashboard > Modules and enable the 404 Monitor module from the list.
To reach the 404 Monitor, hover over Rank Math’s menu entry on the WordPress menu. In the sub-menu that appears, click the 404 Monitor entry.
You’ll reach the 404 error page. Notice that it says, “The 404 error log is empty”. Well, of course, it is, we keep our site clean.
An empty log is great, but it doesn’t show you much. So, for this example, we’ll manually introduce a few errors. After introducing the errors, here is how the 404 Monitor will look.
Let’s understand the data that Rank Math is sharing with us. If you notice, there are 3 fields in the 404 Monitor.
- Access Time
These fields help you navigate your 404 errors better. In the initial days, you might not get a lot of errors, but as your site grows, you will have to use some precision rather than eyeball it.
The URI (Uniform Resource Identifier) represents the URL that the user tried to access. The only purpose of this field is to see if the error is caused by an obvious typo. If it is, you can fix it and not think about it anymore.
Hits represent the number of times an error-causing URI was accessed. This is an important number as you would want to fix errors with the most number of hits first.
The access time field captures the time of the last hit on the error-causing URL. This field, combined with the hits field, is pretty useful in solving issues.
How to Use This Information
Using this information is quite straightforward. Using the URI and the number of hits, you can prioritize the URIs with the most errors and start fixing them by following the instructions in Rank Math’s guide on fixing 404 errors.
But, there is an important limitation with this data. This data does tell you about the URL with the error, but it does not tell you if the error was caused on your own website, or from an external website.
The distinction is important. You can fix errors on your own website quickly and easily, but if the cause of the error is on another website, then your potential solutions to the problem are limited.
But first, you have to find the source of the problem, for which, you can use the Advanced Mode in Rank Math’s 404 Tracker.
Advanced 404 Monitor
To activate the Advanced 404 Monitor, go to Rank Math’s General Settings > 404 Monitor and activate the Advanced 404 Monitor.
Remember to save your settings to activate the Advanced 404 Monitor.
Before getting into what the Advanced 404 Monitor, let us give you some basic guidelines on how and how not to use the 404 Monitor.
The Advanced 404 Monitor captures a lot more information about your 404 errors; which can definitely help you in solving your errors. But, it also means that it increases the size of the error log significantly.
In the Simple 404 error monitor, errors are grouped by their hit count. What that means is that a single error that occurs a hundred times will show up in the logs as a single error with a hit count of 100.
The Advanced monitor is different. Due to the nature of the information it captures, errors, even on the same URL, they are captured as individual errors. But there is a catch, and you can still isolate the grouped errors, and we’ll show you how to do that later in this article.
If you have tons of errors on your website, then that can become a problem. Our recommendation is that you only use the Advanced 404 Monitor to fix 404 errors that you aren’t able to fix with the simple 404 Monitor.
Now that you understand, let us see what the 404 Monitor actually does.
Once you come back to the 404 Monitor, here is what you’d see. Note the 2 new fields that were added to the logs.
Obviously, the 2 fields will be empty for all the captured errors as the advanced mode was not active. To explain the importance of those fields, let us introduce a few errors again so that the fields are populated.
Here is how the error logs look after we introduced the errors.
That’s more like it. The Referer field is empty for most logs, and we’ll explain that in a minute.
This field will capture the referral URL, which is linking the error-prone URL. This field’s information is crucial when a bad internal or external URL causes a 404 error. For example, if another website links to an important post on your website but with an error, this field will help you identify the source of the error. Using that data, you can contact the site linking to you and ask them to correct their link.
In some of the errors, we simply typed in an incorrect address in the address bar, causing the referer field to be empty. But, for the last example, we created a post and added a broken link. Once we clicked the link, Rank Math captured the URL of the post in the referer field, as you can see in the image.
It’s important to know that the referer link can be an internal link or an external link. That means that if you make a mistake in linking internally to your post, Rank Math will capture that for you as well.
The Hits column indicates the number of times, an unavailable URL on your website has been accessed. And, there is more to it — the 404 errors grouped in the Advanced 404 Monitor can be accessed by simply clicking the Hits count.
All the 404 errors resulting from the specific URI will be filtered. This makes it easy for you to analyze the referer and user-agent for an unavailable URI, especially when your 404 error log includes hundreds or even thousands of 404 errors.
Wikipedia defines User-Agent as, “In computing, a user agent is a software (a software agent) that is acting on behalf of a user, such as a web browser that retrieves, renders and facilitates end-user interaction with Web content. An email reader is a mail user agent.”
In this context, the User-Agent is the browser. That means that Rank Math will also capture the browser the user was using when they got a 404 error. This is extremely powerful information for many reasons, and here is an example.
If you have or know about affiliate sites, you’ll know that they spend a lot of time creating and managing links. Many affiliate networks offer scripts that can convert regular URLs to affiliate URLs. Imagine a scenario where a script has an issue with a particular browser which causes the URL to throw an error. Without the user agent data, you’ll keep on trying to reproduce the error but will fail. Only when you see the user agent data will you be able to understand the issue.
Note: The Advanced 404 Monitor collects only the details of the user-agent, such as the browser and operating system, and no personally identifiable information of the end-user (such as IP address) is stored.
Fixing the 404 Errors
Once you have the list of 404 errors, the next thing to do is to fix the errors. There are many actions you can take to fix the errors, and we’ve created a dedicated tutorial for it. Be sure to check it out to understand how to solve 404 errors.
Exporting 404 Errors PRO
The 404 errors that you’ve fixed are only for now. But new errors may be introduced in the future, and you may need to periodically review and fix them.
Though you can check the access time when the broken link was last accessed, the export option in Rank Math can help you generate a 404 error report easily for the chosen timeframe. This feature is available only in the Rank Math PRO version.
To export your 404 errors, click the Export option at the top of the 404 Monitor.
Then you will see a form underneath that lets you select the time period for the export. Choose the From date and To date from the date picker and then click the Export button, as shown below.
Now the CSV file will be downloaded to your computer, and you can open it with the spreadsheet editor of your choice. You can go ahead review them and decide the ones that need to be fixed.
That’s it. If you still have any questions on monitoring 404 errors with Rank Math, feel free to reach our support team, and we’re always here to help.