A couple of weeks ago, we ran across an article published in CNU’s Public Square entitled “Walkability indexes are flawed. Let's find a better method.” Naturally, as an urban data and predictive analytics software company - who produces a quality of place index that measures walkability among other “abilities”, ahem, the State of Place Index - this got our attention. While we ourselves have produced carefully researched, constructive criticism of indices like Walk Score and have produced a guide to walkability indices outlining each of their pros and cons, we wanted to point out a few ways in which 1) we address many of the stated “flaws” that Mr Steuteville outlines in the article and 2) why his solution alone - to focus on measuring walking - has key downsides that citymakers must consider. So let’s dig in before my datageeky heart bursts with joy (we love getting the opportunity to get geeky with it in pieces like these - so thanks so much, Mr. Steuteville for writing your piece!).

Point 1: “Walkability indexes…systematically overestimate the walkability of certain types of places, while underestimating walkability in other places.”

We totally agree! This is because most walkability indices - both Walk Score and the relatively new National Walkability Index (NWI) created by the EPA that Mr. Steuteville references in his piece - are based on the number (and sometimes quality) of non-residential locations within walking distance of a certain point. Most of them do not account for the “experience” of the walk, in terms of the “Hierarchy of Walking Needs”:

Accessibility (whether there is a way to walk to that destination - think, the presence of sidewalks and of course the presence of destinations)
Safety (whether people feel safe, crime-wise, to walk to that destination - think lighting, graffiti, litter, or what is called in the biz, “physical incivilities,” or physical features that impact people’s perception of safety)
Comfort (whether the street is designed for pedestrians/bicyclists - think the width of sidewalks, whether there are sidewalk buffers, benches, trees, etc.
Pleasurability (whether a street is interesting, enjoyable - or KonMari style, sparks joy! - think mixed-uses, outdoor dining, pleasing architecture, nice landscaping, etc.)

Most walkability indices miss the mark on all of the above. This is why there is both overestimation (mostly of car-oriented streets with lots of “destinations”) and underestimation (mostly of residential - but walkable - streets). But this is precisely why we measure nearly 300 different built environment features that capture all four of these important walkability characteristics - specifically broken down into ten urban design dimensions known to impact actual walking rates (more on that in a sec).

But don’t take my word for it, take a look at the State of Place Index for each of the blocks that Mr. Steuteville points to as evidence of the faults of most walkability indices in his article.

Let’s take the Virginia Beach example that Mr. Steuteville highlights first as evidence that walkability indeces like the NWI often overestimate the walkability of suburban commercial centers. Indeed, the NWI perplexingly categorizes this street as “most walkable!” We totally agree that this is an astoundingly embarrassing and frankly, dangerously wrong assessment. As you can see below, however, the State of Place Index quite accurately captures the dismal walkability - or complete lack thereof - of this nightmare of a street. And this is partly because we measure nine other urban design dimensions besides proximity - accounting for 290+ on-the-ground, “micro-scale” built environment features. And even more to the point, we don’t overly count the “proximity” afforded by shopping malls and strip commercial centers - as this street only scores 15/100 on proximity to non-residential locations. And as Mr. Steuteville astutely points out: “Despite the abundance of destinations in stores, restaurants, and other businesses along this thoroughfare, the distances are too great, and the walk would be dead-on boring. There are no active frontages and little of interest to see, except for cars rushing past. The photo tells the story—lots of cars and no people. Virginia Beach Boulevard, at this location, should get a Below Average Walkable rating at best.” Yup, totally. We quantify everything he points out - and more - and it’s reflected in this block’s State of Place Index and Profile.

Screen Shot 2019-01-22 at 5.50.57 PM.png

Mr. Steuteville offers a second example of the flawed nature of the NWI - Bristol Street in Irvine (which is actually in Costa Mesa, but I digress ;)). It also gets the “Most Walkable” rating by the NWI. And it’s just as baffling. But once again, the State of Place Index is a much more accurate assessment of the lack of place quality and poor walkability of this street. Just ask the City of Irvine - they used State of Place to conduct an assessment of their entire Irvine Business Center to help them properly inform their redevelopment plans - and give them the data they need to convince city council to get them done!

Point 2: “There is no easy way to fix these indexes, which are based on the faulty premise that proximity to destinations makes a place walkable—or at least is an accurate proxy for walkability.”

Well, see our response to Point 1 above. There IS a fix. We implemented it years ago. It just hasn’t been the easiest fix - because it requires a lot of data collection time, making it difficult to just punch in an address and out comes a score (for now…)…but let me walk you through our history, how and why we decided to integrate more “micro-scale” urban design features that captured way more than the proximity to destinations, and most importantly, where we are (very excitedly) going!

So, avid readers of this blog may know that the history behind the State of Place Index runs deep (literally, I had no grey hairs and wrinkle-less skin when this thing first came to life!). In 2001, I was working on a project evaluating* the Safe Routes to School program in California. If you think the lack of good walkability indices is dire now, no one had even heard of the word walkability back then! So suffice it to say, we had to cobble together a basic index (but dare I say already better version of Walk Score - as it contained more than just proximity measures), which included about 12 urban design features. This led us** to apply for - and be granted - funding from the Robert Wood Johnson Foundation’s Active Living Research initiative, which tasked us with creating an audit tool that researchers could use to test their hypotheses between the built environment and physical activity/walking. As a result, we created the then 162 item, Irvine Minnesota Inventory (IMI), published and validated in peer-reviewed articles in 2005.

The IMI was widely used in academia, however after countless practitioners and academics alike reached out to me, begging me to create a way to pull all this data together into some sort of index, I obliged. Having already devised a preliminary algorithm to combine part of those 162 items into a score (circa 2006), I built upon that and created a full version of the algorithm, that was then used in both the seminal Brookings research (that tied - what would become known as State of Place - to real estate value) and in the Healthy Places study that quantified the walkability of “Smart Growth” communities (circa 2010). Both the Irvine Minnesota Inventory and the corresponding algorithm have since been updated (as part of a study to evaluate the walkability in Shanghai, circa 2013) to reflect the current version (with 290+ micro-scale urban design features) that underlies today’s State of Place Index. Today, our database includes over 8,000 blocks and counting…and most importantly, we have already begun to automate our data collection process. The ability to extract our data instantly from images means that the last remaining, critical disadvantage of the State of Place Index will soon be overcome and accessing State of Place will be as easy as entering in an address and out comes your score, giving citymakers a treasure trove of accurate data they can affordably and quickly use to not just make the case for great places but also much more easily inform land acquisition, corridor studies, comprehensive plan updates, master plans, the RFP approval process, funding protocols, and more, not to mention properly benchmark themselves and measure and communicate success.

Point 3: “[Walkability indices - specifically the NWI - are] not fine-grained enough to accurately determine walkability [because it] only measures ‘census block groups,’ which…unfairly punishes some neighborhoods in small cities and towns that are highly walkable.”

This one is easy. State of Place collects data at the block level - and by block, we mean the area between two intersections, including both sides of the block. And the State of Place Index can be aggregated up into whatever geography our customers deem appropriate. And again, there’s those pesky 290 features we collect - for each of those blocks!

To prove his point here, Mr. Steuteville points to a lovely block in Ithaca which the NWI index has misleadingly categorized as “Below Average” in walkability. Again, this underestimation is just as astounding - and equally problematic - as the overestimations he points out. But once again, the State of Place index more accurately captures the walkability of this block. Let’s dig in because this one deserves a bit of an explanation from our end. We know what you’re going to say - wait, only a 46.3? This deserves like an 90 or at least an 80. And herein lies both the uniqueness - and dare we say advantage - of how the State of Place Index is constructed. First, and foremost, the State of Place Index measures performance - it is NOT a ranking. That means, it scores blocks on its existing assets and needs. And while you do not need all 290 urban design features to get a score of a 100 (because you don’t need all 290 to be an amazing block), each block is measured relative to the performance of the other blocks in our database (which was originally carefully constructed - statistically - to ensure a continuum of types of land uses, densities, levels of walkability, urban, suburban, exurban, rural places, etc.). So all this score means is that there are other blocks in our database that outperform this block - and as great as this block is - it can be improved.

Another point to consider is that because the State of Place Index is based on relative performance, most (70.3% to be exact) blocks in our database score between 40 and 60 and only .2% score above 80! It’s not easy to score super high on the State of Place Index (because there’s always ways to improve places!). So this block’s score is actually not that bad. Further, it’s important to dig deeper and analyze the reasons why this block scores a bit less than one would think. Well, it has no parks and public spaces - that brings down its score a LOT. This is why it’s best to analyze the State of Place Index of various blocks at one time, so that a more full picture can be told (that said, the next version of the State of Place Index will indeed allow customers to adjust a block’s score to take into account the assets and needs of nearby blocks without just averaging the State of Place Index of several blocks at once, but I digress). Finally, I know, I know, you guys want a ranking. Fine, this block ranks in the 86.5th percentile! See, I told you this is not a bad score! ;)

Point 4: “Fortunately, there is a way, at least theoretically, to create a far more accurate of measure of walkability. With data from smartphones, we could measure how much people actually walk.”

Screen Shot 2019-01-22 at 5.56.03 PM.png

Mr. Steuteville is absolutely right - in a way. Accessing “big data” can help us measure the amount of actual walking done in a location. And this is indeed an obviously important measure to track. But this will not - and should not - replace measurements of the built environment itself. Why? Because there are plenty of places were people do walk - a LOT of them - but because people have no other choice. Just because people are walking does not de facto mean that the built environment is conducive to walking - or that it is safe, equitable, comfortable, pleasurable, etc. By Mr. Steuteville’s premise, if you were to measure walkability based on walking across the three streets he highlights in his article, we would get a quite inaccurate understanding of the quality of the built environment of these places. Just look at this poor woman (in the image above) with her stroller crossing this nightmare of a street! Yes, she’s walking - but clearly because she lacks an alternative (who would want to cross this street - and with a baby no less!). Also, there was only one pedestrian spotted in the beautiful Ithaca street. Granted, this is only one moment in time, but the premise still holds that sometimes - and quite often in disenfranchised neighborhoods - people walk because they have to. Relying on walking rates - alone - to measure walkability is not only flawed but opens up all sorts of potential inequity issues whereby some places are deemed walkable - when they are far from it - simply because people are walking there. This is a critical point we must not overlook.

The second problem with relying on solely measures of walking as a measure of walkability is related to what I point out above. We cannot tell why people are - or are not - walking in a location. Meaning, if there are no people walking in an area, and we measure that, and provide that data to municipalities or developers - I bet you they already know that there are no - or few -people walking there. So now what? That’s the key question. Measuring the outcome - a behavior - that is influenced by the built environment without measuring the components of urban design that are facilitating that behavior - or not - misses a huge part of the equation. One cannot point to the “why” of an outcome if you’re not also measuring the inputs that influence that outcome. Specifically, if we are only measuring the number of people walking, we can’t tell citymakers what to change to get people to walk more or less - and we can’t point to what changes matter more or should be prioritized. In fact, herein lies the issue with most smart city predictive analytics. They are simply measuring - and/or predicting - problems (e.g., like too little walking or too much traffic) but cannot point to the solutions (i.e. changes to the built environment) that would address those problems (e.g., increase walking or reduce traffic). It’s like being able to predict whether you’ll get a terminal disease but stopping short of telling you what to do if you get the disease, or better yet, not telling you how to avoid the disease, period. So yes, let’s definitely measure walking. But unfortunately, that’s not a fail safe or replacement for measuring the built environment.

Point 5: “Our walkability indexes today only measure walkability indicators, and such indicators are often wrong. Computers don’t walk and they can’t truly recognize walkable places. When aggregating indicators yields bad results, computers don’t care.”

Oh, but us datageeks and placelovers here at State of Place DO care. In fact, measuring walkability - and doing so accurately - has kept me up (for better for worse) for over two decades, earning me the moniker of the methods girl! Anyway, point is, if we care, our computers care - or at least our computer models do! We are constantly working to refine our algorithms, truth-testing our results and validating it against - yes - walking data. :) From the start, we ensured that our models had 1) face validity - meaning that the score that a block or neighborhood got just “seemed right.” and 2) that a higher State of Place Index was tied to more people walking (in fact, our models show an R2 of .169 at p-values <.001 - email me if you want more info - don’t want to go overboard with the geekiness here). And not only do both of these still hold true, we’re now using machine learning and AI to further improve the accuracy of our models. And the more data we collect, the more precise and “smarter” or models get.

Plus, we’re always keen to learn from our customers how to improve not just our measures, but what we measure, and the metrics we offer via our software that will help citymakers measure what they care about and inform their critical planning, design, development, approval, funding, and policy decisions. Case in point, when I run the Ithaca street example through our model, I realize that some folks might still think it scores “too low.” Aside from the explanation I offered above as to why this is the case, we want our scores to mean something to our customers. So we can actually re-scale our scores (that’s the process that converts the scores into a number from 0 to 100) relative to the type of street it primarily is - residential, commercial, mixed-use - or even based on density - urban, suburban, rural, etc. For example, we did that for the residential street in Ithaca and it scored a 56.5 instead of a 46.3. We’ve even re-standardized our scores relative to a particular city, district, neighborhood, or project area, as was the case in the City of Tigard. All of their blocks scored so low on the State of Place Index that it was hard to custom-tailor recommendations based on each block’s assets and needs - because they were all so sorely lacking. But, our system allowed them to create a custom score that compared their blocks relative to each other, which then allowed them to dig deeper into which needed even more TLC than others (see image above). That’s the beauty of having a dynamic scoring system that isn’t just an index, but a powerful software that allows the customer to get data and predictive analytics that are useful and customized to them - while still accessing a national, standardized benchmark. Oh, and I haven’t even mentioned any of the advantages that come with our predictive analytics - outside of the indexing we do itself - like our automatic recommendations, identifying changes that will lead to the biggest bang for the buck, and calculating the ROI of proposed projects - but readers can read more about that here, lest we get too overly promotional for one blog post! ;)

Point 6: Inaccurate walkability ratings can disadvantage retrofitting suburbs by scaring away would-be developers and investors…

This is absolutely spot on, especially, as Mr. Steuteville points out, given the fact that the likes of one of our faves: “Smart Growth America uses the NWI to measure “smart growth potential,” for example, for the purpose of advising investors and municipalities on how to get more for their smart growth investments.” We AGREE Mr. Steuteville. Let’s TALK SGA! ;)

Point 7: On the flip side, walkability indices that inflate an area’s walkability may disincentivize those areas from investing in better places - and may even discourage efforts to boost walkability given that these indices won’t account for actual improvements.

Once again, we agree. Data must tell an accurate story or it’s not just useless but dangerous. Yes, citymakers’ relying simply on intuition and gut makes it hard for them to get approvals, funding, and buy-in for their projects (because pretty pictures alone don’t win these battles). And yes, data-driven, evidence-based approaches can help identify the best changes needed to make awesome places people love - and make the case for them. But indeed, it is important to understand that first, not all data (or indices) are created equally, but also that measuring outcomes (like walking rates) alone is not a suitable replacement - not if you want to actually inform citymaking decisions, funding, and policy.

So, we convince you the walkability index is not dead yet? Intrigue you to look under the “sole” and find out not just how well your community measures up - and most importantly, inspire you to figure out what to do about it? Well, we’d love to learn more and show you how we can not only show you what changes will lead to the biggest bang for the buck but also give you the data you need to make the case for those changes! Please choose a time for a quick chat/demo or try out State of Place for free today (and access those 8K+ blocks immediately)

*Shouts out to the formidable Tracy McMillan, whom first led me along this long walkability path. It was her amazing dissertation work on evaluating SR2S that paved the path toward me becoming “the methods girl! and to Marlon Boarnet (the original urban economics nerd), and Kris Day (under whom I learned everything I know about how the built environment impacts behavior and who is by far the best PhD advisor known to woman!), with whom I worked on both the SR2S projects and the IMI.

Featured