The Fracking Wall

In order for me to stay relevant in my chosen profession of software and database development, I must constantly self-educate in the evenings.  Since I’m already tired at the end of my “day job”, I try to find small projects of particular interest so that I may enjoy this process of self-education.  These small projects generally use open sources of information so that I can practice both the extraction, transformation, and loading (ETL) of data as well as its presentation to an end-user.  Enter into the picture: earthquakes.

Earthquakes provided an open source of information, e.g. the Advanced National Seismic System (ANSS), and an opportunity to plot their locations on a map using, e.g. Google Maps.  Mapping is one of my favorite programming activities and I enjoyed great success plotting earthquakes in a variety a ways using different modalities to do so.  Earthquakes were all well and fine, but I wished to blend them with other data.  Having read about earthquakes being associated with hydro-fracturing operations, I decided I would have a go at blending the data sets together: earthquakes coupled with the locations of oil or gas wells.

It was when I attempted to obtain well location information from governmental sources that I hit what I now call “the fracking wall.”  Stating it as such may seem less than objective, but as I’ve cited in the “Professional Background” page on this blog I have over three decades of experience with software, database, and end-user interface development and based upon this experience it is my objective opinion that a “fracking wall” does exist when it comes to obtaining data concerning hydro-fracturing wells, especially concerning well locations.

State after state provided little, if any, location-centric information.  Some, such as Oklahoma’s Corporation Commission, provided none.  Others, such as Texas, provided their location-centric information only in data sets available behind a “pay-wall.”  Still others, like Arkansas, provided location-centric information within Keyhole-Markup-Language (KML) files that needed special programming to extract the latitudes and longitudes from within.  Special programming I could do as I had the skill set, but not so for the more common non-programmer analyst.  State after state, organization after organization, frustration after frustration, led me to conclude that a “fracking wall” truly does exist.

Part of the purpose of this blog is to attempt to diminish the existence of this “fracking wall.”

2 thoughts on “The Fracking Wall

  1. Chad M

    I just wanted to thank you for your work. It’s admirable that you’re self-teaching, but to do so with real-world data that could be of use to those interested shows that you understand what it means to be a developer.

    1. frackinganalysis Post author

      Just to keep you informed of on-going developments:

      1) I have downloaded and transformed the fracking well chemical disclosure data (as extracted by into a database format suitable for use in an interactive program and for importation into the high-speed, open-source Apache Solr search engine.
      2) I have downloaded and transformed the chemical toxicity data available on GoodGuide’s Scorecard website into a database format suitable for use in an interactive program and for importation into the Apache Solr search engine.
      3) I have blended the aforementioned fracking well disclosure data with the chemical toxicity data and have authored a Java/JSF/PrimeFaces/MySQL web interface, which I’m calling “FracKhemAll”, that allows one to search for disclosed wells using a variety of criteria (e.g. state, county, operator, chemical ID number) and chemical category (e.g. recognized, suspected) and toxicity (e.g. carcinogen, neurological, reproductive, respiratory).
      4) I have deployed this interface (consider it a “beta” site) using Amazon Web Services (AWS) Elastic Compute Cloud (EC2) “spot instances” running Ubuntu Server 12.04 LTS, Apache Tomcat, Java 7, JSF/PrimeFaces, and MySQL 5.5.
      5) Now that I’ve proven that the aforementioned can be done by a single programmer working at night, I’ve started the process of working my way state-by-state through their websites to build a list of well information/location download links. When I’ve completed this list, I’ll post it here, and then my intention is to implement extract-transform-load (ETL) scripts using WGET, Java, and/or Selenium (where necessary) to pull down the down on a periodic basis (monthly works for me).
      6) I will then blend the well data with earthquake data obtained from the Advanced National Seismic Survey (ANSS), reverse geo-coded using a Java library from GeoTools so as to obtain the country, state, and county of each earthquake.
      7) At which point, I will explore various methods of doing time-series analyses (either maps or statistics or both) to see if a relationship between fracking and earthquakes can be discerned in a manner suitable for consumption by a reasonably intelligent citizen with an open mind.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s