This is the third article in a five-part series examining key urban challenges in American cities: Suburban sprawl | Gentrification | Big Data | Climate change | Diversity

Data has been described as the "new oil" powering our economies. It is also increasingly powering our cities.

A handful of U.S. cities are starting to reap the dividends of using Big Data to help their cities flourish, replacing filing cabinets with complex data infrastructure. In the third article in this "urban challenges" series, let's look at some of the early pioneers and how we can expect data to influence the future of cities.

The Bloomberg effect

Former New York Mayor Michael Bloomberg has been one of the loudest cheerleaders for using data (and open data) to better manage cities. He, among many others, observed that the rise of cities in the early 21st century has coincided with a technological revolution making billions of data points available to city managers.

Bloomberg's What Works Cities initiative is designed to help 39 U.S. cities to better use those data points for decision-making and transparency, and ultimately to achieve urban services that are more effective, cost-efficient and result-driven.

Where is this data coming from? The possibilities for using data to shape urban services is expanding so quickly because of how our own interaction with the digital world is changing. This is because much of the data being is self-generated by people like you — through Twitter, Facebook, the transponder on your E-ZPass if you are a New Yorker or your Oyster card if you are a Londoner.

The CitiStat model

The excitement over data-driven city governance in the U.S. began in Baltimore. Since the city's CitiStat initiative — a form of urban dashboard to centralize data collected on the city — won the Innovation in American Government Award in 2004, cities have been looking to replicate the Baltimore model.

Martin O'Malley, a pennywhistle-playing advocate of stats-based governing (and erstwhile presidential contender) who was mayor of Baltimore from 1999-2007, oversaw the introduction of CitiStat. He pioneered the simple idea of regular data-driven meetings between the mayoral team and the heads of key agencies to analyze past performance and set objectives.

By keeping track of data in this way, city managers can improve the performance of cities. They can also look to the future. One of the greatest dividends is the ability to use predictive analytics to preempt future problems — such as emerging pockets of urban poverty, unemployment and instability — and take action earlier rather than later.

But data-driven governing is not all about a bunch of politicians and administrators sitting in a room to make efficiency gains. It is also about the relationship between the city and the citizen.

Responsive cities

One of the greatest drivers behind leveraging data in cities has been its participative possibilities.

Stephen Goldsmith — who worked with Bloomberg in New York's City Hall — talks of "responsive cities." Responsive cities are ones that leverage new digital possibilities such as data mining, apps, tech-savvy citizens and social media to create a city where the inhabitants hold leaders to account for failing schools, pot holes and anything else that affects their urban life.

Some urban planners are now also inviting feedback from citizens on future plans for a city through simple apps that serve as "Tinder for cities," allowing passers by to swipe their approval, or disapproval, for newly planned developments.

People like Goldsmith see these digital tools as an opportunity to face up to the crisis of local governments which, burdened with layers of bureaucracy, have become "tone deaf" to citizens' needs. But he recognizes that data and digital tools themselves are not enough.

Major changes in governance are also required to breakdown a business-as-usual bureacracy and, in Goldsmith's words, infuse these new digital tools with changes in the way that local government operates.

Smart cities

If CitiStat was where it all began, "smart cities" are where we are heading. Smart cities, building upon the concept of the Internet of Things (IoT), use vast networks of computer systems and embedded sensors strewn across a city to collect and analyze real-time data. This may be data about public transport use, parking availability, energy usage, air quality or waste management, and helps to manage cities more efficiently by allowing for minute-by-minute feedback.

After investing $15.7 million in tech innovations, Kansas City now calls itself a smart city. But cities like Masdar in the UAE (with plans for a trillion sensors) have gone furthest in making this futuristic vision a reality, although with mixed success.

The need for responsible data stewardship

Wherever government gains the power to start collecting data on this scale, there will be demands to keep citizens in the loop over their privacy rights — an issue for which many city government have come under fire. Schemes like China's alleged plans to create a "trustability score" for each citizen based on online data on whether they have been given traffic tickets or whether they have paid their taxes strike a worrying balance.

Closer to home, too, there have been minor scandals about sensors hidden in garbage bins and concerns about digital profiling. An "opt-out" provision has been suggested as one way of calming these fears, although that may not be as easy as it seems.

The most important ingredient in all of these moves is not technology but political will. What is needed are not cool gadgets, but "leveraging, understanding and learning from data" by city leaders. But as local governments move to become "digital default," cities that fear being left behind are increasingly looking to Big Data as a source of competitive advantage.