Tidy Data Principles for your DataFrame

Data science requires a solid understanding of tidy data principles. A good data scientist can recognize the difference between tidy and untidy data. It takes a bit of practice, but following two principles can help keep it simple.

The Two Tidy Data Principles

  1. Each variable has a separate column.
  2. Each row represents a separate observation.

Shown below are two images. Each image represents a sample of weather observation data. One is tidy, the other is not. Follow the aforementioned tidy data principles to understand.

untidy data
This data sample is untidy.
tidy data principles
This data sample is tidy.

In the first image of untidy data, the single column ‘variable’ contains all the variables. Consequently, several rows will hold the same observation, but for each variable. The second image corrects the untidy data because there are separate column for each variable.

Use the first steps of data wrangling when presented with a new data set. This is the best way to detect if the data is already tidy or not.

Untidy Data Isn’t Always Bad

The tidy data principles were adopted from a paper by Hadley Wickham as standard way to organize data for analysis. But sometimes reshaping data away from this standard can make it present better for reports. It all depends on the data.

Think of a tidy data set as the standard starting point for analysis and visualization. It is fine to reshape data if needed for a certain purpose.

Melt Away The Tidy Data

Melting data is the process of turning columns into rows. In the above images, the tidy data can be melted with the Pandas pd.melt() method. This is how the tidy image reshapes to the untidy image. Assume the tidy dataframe is called airquality, and the untidy one will be called airquality_melt.

airquality_melt = pd.melt(airquality, id_vars=[‘Month’, ‘Day’])

Notice the parameters id_vars. This is a list of columns to not melt. The parameter value_vars specifies columns to melt. Every column not in id_vars will melt by default, if value_vars is not used.

Give Descriptive Names for ‘Variable’ and ‘Value’

Refer to the ‘variable’ and ‘value’ columns in the untidy data image above. Accomplish this with var_name and value_name parameters.

airquality_melt = pd.melt(airquality, id_vars=[‘Month’, ‘Day’], var_name=’measurement’, value_name=’reading’)

Pivot is Opposite of Melt

Use the pivot_table method to get the melted version of the dataframe back to its original state.

airquality_pivot = airquality_melt.pivot_table(index=[‘Month’, ‘Day’], columns=’measurement’, values=’reading’)

# Reset the index of airquality_pivot
airquality_pivot = airquality_pivot.reset_index()

Pivot will not work if there are duplicate rows of observations. Duplicates can be dealt with by providing an aggregate function. Use np.mean for the aggregate function to reduce duplicates with the mean value.

airquality_pivot = airquality_dup.pivot_table(index=[‘Month’, ‘Day’], columns=’measurement’, values=’reading’, aggfunc=np.mean)

# Reset the index of airquality_pivot
airquality_pivot = airquality_pivot.reset_index()

Data Wrangling Cheat Sheet for Python

A data wrangling cheat sheet for python data-scientists starts with an initial exploratory analysis. This is the crucial first step in data cleaning. Just like an experienced chess player’s first moves can be scripted, a data-scientist might have several scripted steps to get familiar with the data.

Data Wrangling Cheat Sheet First Steps

Some good commands to start the data analysis are to import pandas, load the data into a data frame, and get an overview of the data. These commands are as follows:

# Import pandas
import pandas as pd# Read the file into a DataFrame: df
df = pd.read_csv(‘file_name.csv’)# Print the head of df
print(df.head())

# Print the tail of df
print(df.tail())

# Print the shape of df
print(df.shape)

# Print the columns of df
print(df.columns)

The methods above will reveal if the data alredy meets the standard of tidy data. Note that .shape and .columns are not methods, but attributes. Thus, they don’t require the parenthesis.

Next, consider these steps:

# Print the info of df
print(df.info())# Print the value counts for ‘String_Column’
print(df[‘String_Column’].value_counts(dropna=False))# Print the description for ‘Numeric_Column’
print(df[‘Numeric_Column’].describe())

The .info() method will indicate if there is missing data. Moreover, it tells the number of rows, number of columns, and the data type for each column.

The .describe() method will calculate summary statistics on numeric data columns.

The .value_counts() method returns the frequency counts for each unique value in a column. This method also has an optional parameter called dropna which is True by default. Set dropna=False so it will provide the frequency counts of missing data as well.

Once a through exploration of the data is complete, then use visualization techniques such as scatter plots, histograms, etc.

Python Function Arguments | Args, Kwargs, Lambda

Python function arguments are variables that get passed during the function call.  Sometimes, the argument can have a default value. This is useful if no argument value is passed during the call.

This python function has a default value.

def shout_echo(word1, echo=1):
echo_word = word1 * echo
shout_word = echo_word + ‘!!!’
return shout_word

The function concatenates copies of a word and three
exclamation marks. It can be called with or without a value for echo. If no value for echo is passed, it assumes the default value of 1.

no_echo = shout_echo(‘Hey’)
with_echo = shout_echo(‘Hey’,5)

Functions can also have an unknown number of arguments.

def gibberish(*args):
hodgepodge = ”
for word in args:
hodgepodge += word
return hodgepodge

And it works, whether there are one or many arguments.

one_word = gibberish(‘luke’)
many_words = gibberish(“luke”, “leia”, “han”, “obi”, “darth”)

This same concept can apply to iterate through key-value pairs.

def report_status(**kwargs):
for key, value in kwargs.items():
print(key + “: ” + value)

Call the function with as many key-value pairs desired.

report_status(name=”luke”, affiliation=”jedi”, status=”missing”)

Finally, there are lambda functions. These are useful for writing anonymous functions on the fly. They are commonly used in the context of the map function.

spells = [“protego”, “accio”, “expecto patronum”, “legilimens”]
shout_spells = map(lambda item: item + ‘!!!’, spells)
shout_spells_list = list(shout_spells)