Fair warning; this is going to be a bit of a rant.

First, a bit of background. I have conversations every week about SEO – if not every day.

They tend to be about implementation and range from talking about designing and building out a keyword strategy to more tactical topics like how to scale link building.

With those conversations inevitably comes talk of keywords, and more specifically keyword research. If you’re looking for information on the tools I like to use, check out this post on keyword tools.

The fact of the matter is that it all goes down hill when I get asked questions like this:

Which is pretty much a red flag for me, as you see – 99% of people have no bearings on how to find keywords they actually should be targeting, let alone any process for prioritizing their term list into something actionable.

So when I dive a bit deeper and inquire about how they came up with these keywords, I more often than not, get a response like this:

If you just read that above snippet of conversation and thought “yeah, what’s wrong with that?”

I’m Going To Tell You

For starters, the conversation was about SEO (not PPC) so the keyword planner provides no useful information when it comes to competition for organic rankings – the competition scale is for advertisers on AdWords.

A keyword can have almost no competition from advertisers (which just means no one is willing to pay for the clicks) but can be extremely competitive in terms of organic search.

This tends to happen for informational, non-commercialized terms, where there’s no buying or shopping intent – but loads of bloggers and journalists have written on the topic.

For example, if we look at an informational keyword like “how to grow roses” in keyword planner:

you can see it’s reporting the “Competition” as Low, and that’s because there aren’t advertisers clambering to spend money to buy these clicks…

Whereas if we use a tool that actually correlates the difficulty to rank for this term using SEO, like Ahrefs keyword explorer – we see a much different picture:

you might see 10 and think “oh, well that’s still easy” – and to be honest, it should bebut you’d still be wrong.

Instead you need to look at the SERP and gauge who you’re actually competing with.

Here’s the URL’s you’d need to overtake:

To put this in perspective, pay specific attention to the domain rating (DR) of this SERP.

The average DR (or relative measure of authority and ability to rank) for this keyword’s search results is 64.1.

What that means is that unless your website’s baseline link profile and associated trust is equivalent to that of almanac.com… it’s probably going to be pretty difficult to crack this SERP, let alone rank in a position that would bring any form of meaningful traffic, i.e. in the top 5.

More So Still

Simply writing blog posts and creating pages where you shoehorn your target keywords into the page titles, URL’s, and header tags isn’t going to do much of anything.

Every blue moon you might get lucky and rank for a term that has zero competition, but that’s likely because it either a) has no meaningful intent or b) has no traffic.

Instead, use your keyword research data to create a map of all the content you have versus what you need, map your keywords to your content, build them into your site’s architecture and internal link structure – and then go promote those pages.

For those of you who don’t understand my subtlety – promote means build links.

How To Not Failboat Your Keywords

As I mentioned above – you need to take the existing SERP into consideration, more specifically – the URL’s and Domains that already have the rankings you’re after.

To do this you need to capture the baseline ranking heuristics that Google is likely using to calculate and score the relevancy of those pages.

These include (among other attributes):

  • Keyword usage in meta attribute
  • Word count
  • Page level / position in the site’s document architecture
  • Internal links to the page
  • External links to the domain and page
  • The domains trust score
  • Anchor text variation
  • The age of the domain

and a litany of other factors, but the above list is a good starting point, and then from here you need to go into data mining mode.

Get All The Data You Need To Analyze

This means scraping Google to get:

  • The URL and ranking position for all of page 1
  • The ranking page title and meta description
  • Any SERP features or additional UI elements affecting ranking positions and SERP CTR

Then you need to scrape each of the ranking URL’s for:

  • Keyword usage in meta attributes;
    • Page title
    • Meta description
    • Header tages
    • On-page content
  • Word count
  • Page creation date or last updated date

Lastly, you’ll want to grab some 3rd party data for each of the URL’s

  • Domain age
  • Links to domain
  • Links to page
  • Trust flow
  • Citation flow

Gauge The SERP’s Rank Potential

Welcome to the final 10%.

90% of people won’t get to this point, and even fewer will take all the data you’ve just collected and analyze it to find what’s actually important, which is based on all the ranking heuristics for this keyword’s SERP – can you (and more so YOUR WEBSITE) actually rank here – and if so, what’s going to be required.

To do this – it’s best to have all of this data loaded into a spreadsheet, and then to throw in some conditional formatting.

Here’s an example spreadsheet you can use
just remember to click File > Make a Copy

This allows you to look for rows with the most green to identify potential opportunities.

What’s more, the opportunities you’re looking for are to identify specific rank potential for each keyword.

What is Rank Potential

It’s where you (and your website) can realistically rank for a given keyword.

It doesn’t mean “create this page” and *POOF* you magically rank on page 1.

It means, based on your website in it’s current form – with all the above data considered, what needs to be DONE to rank in the position highlighted in the row with all the green boxes in it.

If you follow the process in this post for each and every keyword in your target universe, what you will ultimately be left with will look a lot like this:

From there you can filter down, sorting specific columns in ascending order to find opportunities that are a good fit for you and your website based on your strong points.

For example you can sort and find opportunities that:

  • Require the least links
  • Require the least content
  • Don’t require you to create new pages
  • Can be attained simply by creating a new page targeting the keyword

and so on.

Best of All

If you scraped ALL this data, you now have a comprehensive snapshot of your SEO market, including all your competitors, their keyword footprints, and your current keyword marketshare.

This would allow you to quickly spin up pivot tables to visualize your competitors market share by groups of rankings based on attributes like ranking position, intent, volume, etc.

Ready to Do Keyword Research the RIGHT Way?

I’m here to help – feel free to drop any questions in the comments.

If you want to really get started down the right path, use the Google Sheets template I linked to above and start filling in your data – then drop comments here for help as you hit roadblocks.

Nick Eubanks
Nick Eubanks

Nick is the Founder and CEO of From The Future. When he's not elbow deep in data, he's spending time with his wife, his dogs, or his cars.

COMMENTS (58)

Avatar
  • Avatar Sivagopi says:

    As usual awesome stuff Nick !

    Thank you

  • Hi Nick,

    Thanks for walking us through this.
    Back in the ‘days of yore’ – you were using Excel, SEMRush, Term Explorer etc…

    Are you able to divulge your current software workflow?

    • Hey Jason –

      You can definitely still do this with that tool stack, even swapping Ahrefs for SEMRush but at this point we’ve built a new version of Sentinel that grabs all this data for us (will be in public beta by October).

      • Avatar Dave says:

        Is Sentinel in public beta yet?

        • Hey Dave, no not yet. We’re just as excited but at this point the tech is all built out and stable but we’re building out a completely new front-end in react.js from the ground up. We’re going to be opening it up in private beta before we open it up to the public, if you’d like to help us test it sign up below (we’d love to have you). Thanks!

  • I love what you’re saying! What tools do you use to gather all this data? How do you get the data into the Google Sheet? Thanks, Whit

    • Thanks Whit! We used to use SEMRush, TermExplorer, ScreamingFrog, and Excel and then dump it all into Sentinel, but we’ve built a new version of Sentinel that gets all of it (not yet released, but coming soon!).

  • Avatar Alex says:

    Really great article, love the spreadsheet. 🙂

    What do you use for all that scraping? Any inhouse tool or something else?

    Cheers!

    • Thanks Alex 🙂

      You can get all of this data using Ahrefs or SEMRush (for the keyword data) combined with ScreamingFrog (for all the URL-level data, but takes a while), and then TermExplorer to grab a bunch of the on-page metrics and stitch it all together in Excel or a Google Sheet.

      That’s how we did it for *years* then we finally built our own tool to pull it all in. It’s called Sentinel and will be in public beta this fall.

  • Avatar Markus says:

    Great strategy, it’s 100% inline with my experience. Please let me know when you Sentinel is available and / or beta is open.

    • Thanks Markus – yeah this is how the people actually ranking sites have been doing it for a while now 🙂

      If you sign up for the IFTF mailing list we’ll be announcing a private Alpha in September.

  • Avatar Yasir says:

    A good refresher of the process and I can vouch that it works as I have done the same over the last 9 years using Market Samurai.

    Looking forward to Sentinel though as it seems to have more data points than MS.

    • I have also had some good experiences with market Samurai but at one point I found the keyword scoring to not be as directionally accurate as it once was.

  • Avatar Joe says:

    Great stuff!

    I agree with you in that although Ahref’s keyword difficulty score is the best in the biz, it doesn’t take into account on-page factors. Their reasoning is that it takes too much computational power across their massive data set – which is fine – but on an individual level it’s worth going deper like you did.

    Do you think it’s feasible to take it one step further and to calculate your own score based on all those factors you mentioned? I had started working on a template that was attempting to do that to make it more scalable, but it seems the factors need to be adjusted based on ever industry or vertical. Like some industries, seems that anchor text weighs more heavily, while word count in others (just two examples of the many factors of course).

    Do you think it’s a fools errand to try to create a new score based on on-page factors and then multiple that against the KD score, or worth pursuing?

    • Hey Joe – good to see you 🙂

      I DO think it’s a worthwhile endeavor to compute more vertically specific KD scores based on the nuances that are not only prevalent within specific niches (to your points) but based on patterns that I see between specific SERP’s and those correlated even to specific modifiers, like for example; reviews, prices, vs, near me, etc.

  • Avatar Kyle Wilson says:

    I haven’t read this post yet, but you prompted me in the email and I just want to say you have great hair 🙂

  • Avatar Vin says:

    Man, we really gotta meet up for that drink and chat.

    Let me ask you… assuming all variables other than age and authority are equal… how would you approach the SERP? I find that I can go after keywords with literally zero competition in terms of onpage metrics and backlinks, but have a hard time getting onto the first page due to the age and authority of the sites ranking in the top 10. What would you do here and how can you anticipate this? Is it a simple matter of just adding more links (which is how I’m compensating now)?

    • Yes indeed 🙂

      Outside of ramping up your link acquisition velocity; paying specific attention to acquiring new LRD’s each month and focusing on getting links that are as contextually relevant as possible, some times it’s simply a matter of patience (which sucks to hear) but IME if you’re going after a SERP where the average site is 10+ years old with a 1-2 year old domain, it’s going to just take time for G to build up a trust profile for your site.

      One way I’ve skirted this in the past is to float in 1-2 older domains per year via a 301 to the homepage, OR in the case you’re open to it.. get your hands on an older older domain, with an old and established link profile — and make the move.

  • Avatar Dee says:

    Hi Nick. Helpful post. Can you point me to what bare minimum tools I need to get rolling in on this process. Which ones would I need to buy $$?

    • Hey Dee – Funny story my Ecommerce site uses Scribe but we came **so close** to using jitterbit (may still switch 🙂 )

      Bare minimum to put this together would be SEMRush or Ahrefs, ScreamingFrog, and Excel.

  • Avatar Dee says:

    Thanks Nick. Not to sound sales-y but Scribe is no match to Jitterbit’s capabilities. Do send me a note whenever you think is the right time to evaluate an integration and API platform.

  • Hi Nick,
    This is great information! I knew there must be ways of getting this type of amazing stuff but I haven’t been doing this all that long so I didn’t really know how to get it. But this is so great!! It sounds like that tool you’re building will do everything so you don’t need to spend tons of money every month to get bits and pieces.

    Thanks again, this is great! How do I get on that list to sign up for Sentinal when it comes out?
    Barbara

  • Avatar Luke says:

    Solid post Nick, thanks for sharing! Looking forward to getting stuck into Sentinel in the Autumn (still can’t bring myself to typing ‘Fall’ and taking myself seriously!)

    We’ve used similar via Market Samurai, Majestic and Screaming Frog in Excel but will happily give your G-sheets a whirl while we await the Sentinel beta!

    Keep up the solid work.

    Luke

  • Avatar Ilan says:

    Hi Nick, thank you for the great post. In on of the columns, it says relevancy, where did you pull that metric from?

  • Great post Nick. Rare to see someone talk in depth about this stuff and at a 2017 level. I focus a lot on finding keyword nuggets like you, but I must say your Sentinal tool looks like the next level. Love the idea of automating and scraping!

    How do I get on your alpha list :> ?I saw you mention your email list, but can’t find it – oh hang on, the slide-in just kicked in at the bottom! got it :>

    keep rocking it
    ash

    • Thanks so much Ashley – Yes the goal of Sentinel is to automate opportunity discovery based on the current state of the domain you’re trying to rank and the barriers to entry at the SERP level. We’re pretty excited!

  • Avatar Chris M says:

    Cheers for covering this topic, Nick!

  • Avatar Crusoe says:

    Hi Nick
    thanks for the great post,

    How do you calculate the “KW Difficulty Score”?
    in you spreadsheet
    thanks

    • Thanks Crusoe –

      It’s a manually computed metric based mostly on a mix of authority and trust metrics of the ranking URL’s on the SERP as well as some on-page relevancy scoring.

  • Avatar Dan says:

    Well I would like to throw out alittle curveball here on the importance of keyword research. I have seen posts, such as in Semrush that keyword research is not going to be as relevant as it used to be due to improvements in artificial intelligence of Google. The trend is to move toward higher quality topic content rather than specific keywords. Any thoughts on this.

    • Avatar Jeremiah says:

      Ooo that’s a good question, I wonder if he’ll reply ?

    • This is a great question and a topic I’ve actually been exploring more in-depth recently.

      I do see the shift already taking place towards more topic-based modeling for content structure and rankings, however, the notion that keyword research will *ever* go away is sort of silly if you think about it… as long as search exists, knowing which words people are using to find specific products/services/answers is always going to remain relevant.

      I do think the advent of voice search, and the integration of AI into so much of user’s daily life is going to force a shift on the keyword landscape – but I see this being more to hyper-local search, and needed to be aware of the local slang that exists regionally versus internationally.

      Does that make sense?

    • On the topic of AI, I think it is important to point out that type of intelligence is artificial. Something a lot of people over look and tend to think that the software is getting smarter on it’s own, when it’s not. There is always a programmer on the other end giving a set of instructions – even if it is telling a piece of software to become smarter. It’s all speculative but it’s easy to imagine that Google might push further towards relevancy and creating some divide between out-dated content and new, popular or viral content that is based on user experience. With the phones and watches, search could become more individualized based on a users personal preferences, their likes and dislikes and things they tend to gravitate towards.

  • Great post Nick. By using some of the tips in your article I was able to identify a product in our niche that has relatively low competition and with a little bit of work we can rank for. Thanks again, I always enjoy reading your articles and appreciate all of the effort that goes into them!

  • Avatar Dominique says:

    I use ahrefs to evaluate keyword difficulty. What does Sentinel offer extra or am I good with ahrefs? First time I hear about Sentinel.

  • Avatar Julia says:

    Nick, thanks for the post!
    Actually, I’ve never used Sentinel before. But I am using SE Ranking tool for finding long-tail keywords, moreover it’s a lot cheaper than Ahrefs. It runs a deep search for the long-tail keywords. The tool also provides monthly search volume, seo competition score, suggestion bid, the number of search results and rankings for chosen keyword. https://seranking.com/keyword-suggestion-tool.html

  • it is the first time I have visited your agency’s website Nick and I must say it is very beautiful. Okay so I am planning to create an SEO blog, I was wondering what advice would you give to me in this competitive landscape and does having an SEO blog help you land clients?

    • Hey Shaurya – Yes I think having an SEO blog will absolutely help you generate leads (converting those leads to clients is a different story) but in order to stand out in such a competitive vertical where blogging is commonplace – you need to select topics that haven’t been covered ad infinitum, and then take it a step further and do something different; which is where the challenge really lies IMO.

  • Avatar Randy says:

    All I can tell you is don’t use TermExplorer – very bad customer service.

    Plus, Nick – you still sell your ebooks recommending them, old use processes (also with non-preferred vendors which you’ve told me you augment with another vendor)…you should at least add an “updates” page to both your Master Keyword Research in 7 days and your Keyword Strategy ebooks. I’ve probably spent about $200 on vendors you recommended – and now again in this post you’re recommending something improved, and a bit cheaper. I’m almost wishing I never bought the books honestly considering the old info and with posts like above – almost make whole parts of your ebook not even needed.

    Again, don’t use TermExplorer – long wait on service requests and now being ignored. Not bashing you but the organization and outdated info just kills me.

    • Hey Randy –

      First off, thanks for taking the time to leave a comment.

      Your points are heard and mostly valid; however there are a few caveats that I would like to address. I’m not trying to argue semantics, so if I’m not hitting the nail squarely on the head here please feel free to let me know.

      1. The processes are not old in the sense that they’re outdated, no longer work, or are no longer valuable. I would argue this and say at this moment in time it’s actually quite the opposite. The purpose of this post is to point out that a LOT of people responsible for building up a base of SEO traffic for their website(s) are still approaching keyword research inside an ill-informed bubble.

      2. Have the tools in the process become outdated? Yes, absolutely. The problem there lies is there’s simply not another tool available that pulls in most of the data you need to holistically analyze a SERP, so as of this moment you’re forced to rely on whatever tech exists in whole or in part to get you the data you need, unless of course you have the development resources to build your own (like we’ve done and hence are prepping to bring to market). But for most – this simply isn’t possible.

      3. I definitely take issue with you saying it makes whole parts of the books unneeded, as the books are meant to spoon-feed the process, step by step and click by click, for anyone with no knowledge of how to run through this process to complete the build out of what we call a keyword matrix. Master keyword research *is* now 3.5 years old, and it definitely has a shelf-life as I would argue any (if not all) training content in the tech space does. To that end I’m actually prepping to stop selling them all together and give them away for free.

      Lastly, with respect to TermExplorer: Yes the tool now spits out some faulty data, specifically search volumes and sometimes CPC’s, so if you wish to use it for all of the other valuable data it does provide, you need to augment your data sources to replace these portions. This is a huge PITA and their customer service has become abysmal… the analyzer runs will also sometimes outright just stall, for days – and on several occasions have delayed client deliverables; which is UNACCEPTABLE.

      For all of the above reasons we have built our own tech, and for all of the aforementioned dilemmas in getting all this data at this instantaneous point in time — they’re still a useful, although headache ridden, tool in my/our SEO toolbox.

      So in closing to wrap up what has become a pretty long comment: I would say the information is not outdated, in fact far from it as I think it’s remains extremely relevant now more than ever. The processes are not outdated, we still use the exact same process to analyze THIS data to find opportunities, the tools however have become lackluster, but remain a necessary evil.

  • Avatar nilesh singh says:

    Very nice post……yaa! that’s a good question……

  • […] That’s NOT How To Do Keyword Research — Creating a keyword matrix is the ideal way to do keyword research to find gaps in the market so your content development and link building can be smarter and more efficient. This is a great post on how to develop a keyword matrix and includes a Google doc template to help get you started. […]

  • Hello Nick,
    Wanted to say thanks for this sophisticated blog post. I have a question about “Link Strength.” How do you determine whether the link is strength or week? Also how you are making the calculation of this parameter?

    • Hey Template Monster – Thanks for dropping a comment 🙂

      Link strength is a logarithmic calculation based on how strong the links are for the 10 URL’s currently ranking on SERP 1 for that particular keyword.

  • […] If you use a tool that correlates the difficulty to rank for a term, like Ahrefs keyword explorer, you can quickly find out how likely it is to rank on page one. That said, you need to make sure you do keyword research correct. […]

  • Every time I start feeling like I’m getting a hang of the SEO game, you drop another post like this and I see I have lots more to learn, haha.

    Killer post, man. Really great and informative stuff!

  • Avatar M A HASAN says:

    So useful this post. thanks for your informative stuff! I will do targeted and low competition SEO keyword research

  • Avatar Henry says:

    Hey Nick, great post! I’m still a rookie at SEO, but having read this awesome post, feel like it could be a good thing! I want to learn the right way, and the “new” way. I take it the name has changed from “Sentinal” to “Semantic”, looking forward to seeing it!!! Makes a lot of sense.

  • Nick, I guess you brainstorm a lot. This post is fantastic and you covered almost everything. I have been using Aherfs for my niche project. I didn’t know about Sentinel, Term Explorer. It’s blessing to be here. Your strategy is 100% solid. Do you have the resource about converting leads into clients?

  • […] That’s NOT How To Do Keyword Research Ξ IFTF […]

  • Thanks for this great post and the explanation! I hope this article will help me building my new sites!

  • CONTINUE READING

    FREE VIDEO

    INCREASING CONVERSIONS WITH "KEYWORD INTENT"

    Our data driven approach to keyword research.