# Explaining P-value to a non technical audience

Wikipedia defines p-value as "the probability of obtaining test results at least as extreme as the results actually observed, under the assumption that the null hypothesis is correct". Well if we give this definition, say in a presentation to a product or a business team, you're most probably gonna receive piercing puzzled looks. One of the major … Continue reading Explaining P-value to a non technical audience

# Understanding Multicollinearity and Confounding Variables in Regression

Multicollinearity When two are more of the predictors are correlated, this phenomenon is called multicollinearity. This affects the resulting coefficients by masking the underlying individual weights of the correlated variables. This is why model weights are not equal to feature importance. Ways to deal with multicollinearity Looking at Variance Inflation Factor (VIf), which measures the … Continue reading Understanding Multicollinearity and Confounding Variables in Regression

# Unnest (explode) a column of list in Pandas

In python, when you have a list of lists and convert it directly to a pandas dataframe, you get columns of lists. This may seem overwhelming, but fear not! Pandas comes to our rescue once again - use pandas.DataFrame.explode() import pandas as pd df = pd.DataFrame({'col1': [[0, 1, 2], 'foo', [], [3, 4]], 'col2': 1, … Continue reading Unnest (explode) a column of list in Pandas

# RStudio in Docker – now share your R code effortlessly!

If you are a full time data science practitioner and have passed through the stages of starting out with the Titanic dataset and working through the various exercises in Kaggle , you would know by now that we wish real world data problems are that simple, but they are not! This post is about just one … Continue reading RStudio in Docker – now share your R code effortlessly!

# Setting up PySpark for Jupyter Notebook – with Docker

When you google “How to run PySpark on Jupyter”, you get so many tutorials that showcase so many different ways to configure iPython notebook to support PySpark, that it’s a little bit confusing. So when I finally figured out a way to do it, with the help of multiple websites, I thought I will post … Continue reading Setting up PySpark for Jupyter Notebook – with Docker