Unweaving Happiness

Gurudev Ilangovan

March 16, 2018


Introduction

The World Happiness Report 2018 by United Nations Sustainable Development Solutions Network (SDSN) based on data from the Gallup World Poll surveys was out a few days ago. I had come across such a report for the first time and it got me curious right away. How do you quantify something like happiness?

Basically, people in countries across the world are surveyed based on the following question:

The response to the question of respondents’ assessment of their current life based on an imaginary 11-point scale whereby 0 designates one’s worst possible life and 10 denotes the best possible life respondents can imagine for themselves. Based on the question “Please imagine a ladder with steps numbered from zero at the bottom to ten at the top. Suppose we say that the top of the ladder represents the best possible life for you, and the bottom of the ladder represents the worst possible life for you. On which step of the ladder would you say you personally feel you stand at this time, assuming that the higher the step the better you feel about your life, and the lower the step the worse you feel about it? Which step comes closest to the way you feel?”

It’s still not completely objective. How do you know how happy you are? Are you really as happy as you think you are? I mean what is happiness anyway? Without getting too philosophical, the question is a great way to measure happiness on a macro scale. There are however some questions like how representative the sample is but let’s assume that the sample indeed is representative. Luckily, they had made the summarized data publicly available.

A word of warning if you are reading from a mobile

Many of the plots in this article are iteractive and hence require a lot of screen space. Hence, if you are reading from a mobile, the experience will not be very… happy?

Let’s get exploring!

Setup

First I load all the packages I need and the data.

pacman::p_load(tidyverse,
               readxl,
               GGally,
               plotly,
               rebus, 
               janitor, 
               corrplot, 
               mice, 
               caret,
               naniar, 
               leaflet,
               visdat,
               knitr,
               kableExtra)


df_happiness_raw <- 
  read_excel("data/WHR2018Chapter2OnlineData.xls") %>% 
  clean_names()
df_subregions <- 
  read_excel("data/WHR2018Chapter2OnlineData.xls", sheet = 5) %>% 
  clean_names() %>% 
  mutate_if(is.character, str_to_lower)
countries_continents <- 
  read_csv("data/Countries-Continents.csv") %>% 
  clean_names() %>% 
  mutate_all(str_to_lower) 

make_table <- function(tbl) {
  # DT::datatable(tbl)
  knitr::kable(tbl, "html") %>%
        kable_styling(bootstrap_options =
                        c("striped", "hover", "responsive"),
                      full_width = FALSE,
                      position = "left") %>% 
    scroll_box(width = "1000px", height = "500px")
    
}

Data Munging

Missing

Are there any missing values?

vis_miss(df_happiness_raw)

The data has a few missing values. I will impute those values using the mice package

#missing too much data as is visible
df_happiness <- 
  df_happiness_raw %>% select(-gini_index_world_bank_estimate) 

dfs_imputed <- mice(df_happiness, m=1, maxit=20, meth='pmm', seed=1)
df_happiness <- complete(dfs_imputed, 1) %>% as_tibble()

Adding Continents

I want to cross-reference continent. However, in the country-continent file I had downloaded, there were a few countries missing/with different names. I create a new variable called country recoded that fixes this with the sole purpose of getting the continent. The countries recoded will not be used for anything else and hence, I didn’t worry too much about its validity.

df_happiness <- 
  df_happiness %>% 
  rename(healthy_life_expectancy = healthy_life_expectancy_at_birth,
         std_ladder = standard_deviation_of_ladder_by_country_year,
         std_mean_ladder = standard_deviation_mean_of_ladder_by_country_year,
         gini_index_world_bank_mean = gini_index_world_bank_estimate_average_2000_15,
         gini_of_household_income = gini_of_household_income_reported_in_gallup_by_wp5_year) %>% 
  mutate(country = country %>% str_to_lower(),
         country_recoded = case_when(
           country == "russia" ~ "russian federation",
           str_detect(country,"congo") ~ "congo",
           country == "myanmar" ~ "burma (myanmar)",
           country == "burkina faso" ~ "burkina",
           str_detect(country,"china") ~ "china",
           country == "kosovo" ~ "serbia",
           str_detect(country,"cyprus") ~ "cyprus",
           str_detect(country,"palest") ~ "israel",
           str_detect(country,"somal") ~ "somalia",
           country == "south korea" ~ "korea, south",
           TRUE ~ country
         )) %>% 
  left_join(countries_continents, by = c("country_recoded" = "country")) 
vis_miss(df_happiness)

No more missing values.

Exploration

Let’s do some EDA. Some maps are interactive (the legends can be used to filter!). So feel free to explore.

Happiness levels

Happiness levels have remained largely constant over the years except for South America…

plt_year_happiness <- 
  df_happiness %>% 
  filter(year > 2005) %>% 
  ggplot(aes(year, life_ladder, fill = continent, label = country)) +
  geom_jitter(alpha = .55, size = 4, width = .27, height = 0,  pch=21) +
  geom_smooth(method = "loess", se = F, aes(col = continent)) +
  scale_x_continuous(breaks = 2005:2017) +
  scale_y_continuous(breaks = 2:10) +
  scale_fill_brewer(palette = "Set1") +
  scale_color_brewer(palette = "Set1") +
  theme_light()

ggplotly(plt_year_happiness, tooltip = c("country", "continent"))

And the plot clearly shows the ranking of happiness levels. Although, the Nordic countries like Finland, Denmark and Norway are consistently among the top, Europe as a whole, is right there with North America and South America. Asia is about a step lower and Africa 2.

If we filter just one continent at a time, we can see that South America is the continent with the least variation in happiness levels. Asia has a huge amount of variation and so do Europe and North America. The Oceania countries are just Australia and New Zealand. Not too many countries to talk about there.

Standard Deviation of happiness

Admittedly, the mean is not the best reflector of happiness of a country. We need the variance too! We have 2 columns in the dataset named standard_deviation_of_ladder_by_country_year, standard_deviation_mean_of_ladder_by_country_year (standard error?). Not sure what the second means and assuming the first one is what I need, let’s see the countries with the most and least variation.

plt_year_std <- 
  df_happiness %>% 
  filter(year > 2005) %>% 
  ggplot(aes(year, std_ladder, fill = continent, label = country, size = life_ladder)) +
  geom_jitter(alpha = .55, width = .27, height = 0,  pch=21) +
  scale_x_continuous(breaks = 2005:2017) +
  scale_fill_brewer(palette = "Set1") +
  scale_color_brewer(palette = "Set1") +
  theme_light()

ggplotly(plt_year_std, tooltip = c("country", "continent","life_ladder"))

Wow! Many countries have a standard deviation around 3 and a happiness level around 4! Imagine the happiness-inequality if you will in these countries.

On the other side, the happiest countries are the ones where all of them are general very happy and not where some are extremely happy and some are just moderately happy getting averaging each other out. Powerful idea.

Let’s confirm this hypothesis by looking at a plot of standard deviation versus mean for 2017.

plt_std_happiness <- 
  df_happiness %>% 
  filter(year == 2017) %>% 
  ggplot(aes(life_ladder, std_ladder, fill = continent, label = country, size = life_ladder)) +
  geom_jitter(alpha = .65, width = .25, size = 3, height = 0,  pch=21) +
  geom_smooth(aes(group = 1), se = F) +
  scale_x_continuous(breaks = 2005:2017) +
  scale_fill_brewer(palette = "Set1") +
  scale_color_brewer(palette = "Set1") +
  theme_light()

ggplotly(plt_std_happiness, tooltip = c("country", "continent","life_ladder"))

Correlation Analysis

Let’s take a look at the correlation plot of all the variables in our dataset and explore relationships in detail.

cors <- cor(df_happiness %>% select_if(is.numeric), method = "spearman")
corrplot(cors)

highly_correlated_with_happiness <- cors["life_ladder",][order(-abs(cors["life_ladder",]))] %>% .[-1] %>% .[1:5]
highly_correlated_with_happiness
##      log_gdp_per_capita         std_mean_ladder healthy_life_expectancy 
##               0.7942000              -0.7739672               0.7590268 
##          social_support        delivery_quality 
##               0.7471541               0.6583947

We already analyzed the negative correlation standard deviation has with happiness.

GDP’s relationship

GDP per capita is the most highly correlated. No surprises here.

plt_gdp_happiness <- 
  df_happiness %>% filter(year > 2005) %>%  #we don't have neough values
  ggplot(aes(log_gdp_per_capita, life_ladder, col = continent, label = country)) +
  geom_point(alpha = .8, size = 2) +
  geom_smooth(method = "loess", aes(group = 1), se = F) +
  facet_wrap(~year) +
  scale_color_brewer(palette = "Set1") +
  theme_light()

ggplotly(plt_gdp_happiness, tooltip = "country")

Healthy Life Expectancy

We could take a look at healthy life expectancy vs happiness too. Note that it is that’s highly correlated with GDP. Also note that, it is not just life expectancy. It’s healthy life expectancy! It matters how healthy you are in your life!

I am not splitting by year in the next few plots.

df_happiness %>% 
  ggplot(aes(healthy_life_expectancy, life_ladder, col = continent)) +
  geom_point(alpha = .8, size = 2.5) +
  geom_smooth(method = "lm", aes(group = 1), se = F) +
  scale_color_brewer(palette = "Set1") +
  theme_light()

Social Support

So is social support. However, it is interesting to look at what social support is:

Social support is the national average of the binary responses (either 0 or 1) to the Gallup World Poll (GWP) question “If you were in trouble, do you have relatives or friends you can count on to help you whenever you need them, or not?”

A sense of security and connection are common among happy people.

df_happiness %>% 
  ggplot(aes(social_support, life_ladder, col = continent)) +
  geom_point(alpha = .8, size = 2.5) +
  geom_smooth(method = "loess", aes(group = 1), se = F) +
  scale_color_brewer(palette = "Set1") +
  theme_light()

Prediction

Setup

Okay, now to some machine learning. Let’s see if we can predict happiness!!

We split the data into 2 parts - year 2017 (which we’ll predict on) and all the other years (which we’ll build on using cross validation).

I am trying to model happiness using only qualities that can easily measured. For instance, variables like generosity, social support, perception of corruption etc are subjective. Also they are also collected in the same surveys as happiness and a model using them wouldn’t be too useful.

So the variables I am using:

  1. Continent
  2. log GDP
  3. Healthy life expectancy
  4. Democratic quality
  5. Gini of household_income
df_happiness_caret <- 
  df_happiness %>% 
  filter(year != 2017) %>% 
  dplyr::select(life_ladder, continent, log_gdp_per_capita, healthy_life_expectancy, gini_of_household_income) %>%
  as.data.frame()

df_test <- df_happiness %>% filter(year == 2017) %>% as.data.frame()

I am also centering, scaling and using box cox to remove skew as part of preprocessing.

pre_process_model <- preProcess(df_happiness_caret[,-1], method = c("center", "scale","BoxCox"))
df_happiness_caret <- predict(pre_process_model, df_happiness_caret)
df_test <- predict(pre_process_model, df_test)

Modeling

I am trying out many models - random forests, ranger(a different implementation for RF), Radial SVM, xgboost, knn, elasticNet and a liner regression. Like Stated above, we use 10 fold cross validation to build and validate our models and let caret take care of parameter tuning.

tr <- trainControl(method = "cv", number = 10)

model_rf <- train(life_ladder~., 
               data = df_happiness_caret, 
               trControl = tr,
               method = "rf",
               importance = T)

model_ranger <- train(life_ladder~., 
               data = df_happiness_caret, 
               trControl = tr,
               method = "ranger")

model_lm <- train(life_ladder~., 
               data = df_happiness_caret, 
               trControl = tr,
               method = "lm")


model_xgb <- train(life_ladder~., 
               data = df_happiness_caret, 
               trControl = tr,
               method = "xgbTree")

model_enet <- train(life_ladder~., 
               data = df_happiness_caret, 
               trControl = tr,
               method = "enet")

model_svm <- train(life_ladder~., 
               data = df_happiness_caret, 
               trControl = tr,
               method = "svmRadial")

model_knn <- train(life_ladder~., 
               data = df_happiness_caret, 
               trControl = tr,
               method = "knn")


results <- resamples(list(
  rf = model_rf,
  ranger = model_ranger,
  svm = model_svm,
  xgb = model_xgb,
  enet = model_enet,
  lm = model_lm,
  knn = model_knn
))

summary(results)
## 
## Call:
## summary.resamples(object = results)
## 
## Models: rf, ranger, svm, xgb, enet, lm, knn 
## Number of resamples: 10 
## 
## MAE 
##             Min.   1st Qu.    Median      Mean   3rd Qu.      Max. NA's
## rf     0.3085957 0.3209935 0.3410172 0.3374934 0.3507670 0.3624754    0
## ranger 0.2973487 0.3035172 0.3236273 0.3242100 0.3383682 0.3697525    0
## svm    0.3459667 0.3814176 0.3904982 0.3903180 0.4027713 0.4220382    0
## xgb    0.3458124 0.3664989 0.3764419 0.3820884 0.3920017 0.4315655    0
## enet   0.4504183 0.4683124 0.4831610 0.4876278 0.5073537 0.5270272    0
## lm     0.4430682 0.4781745 0.4900356 0.4870978 0.4942947 0.5425569    0
## knn    0.3372234 0.3711921 0.3813050 0.3763415 0.3925766 0.3992195    0
## 
## RMSE 
##             Min.   1st Qu.    Median      Mean   3rd Qu.      Max. NA's
## rf     0.4091449 0.4280066 0.4434299 0.4403527 0.4535164 0.4642917    0
## ranger 0.3862598 0.4077292 0.4115117 0.4230535 0.4458833 0.4773464    0
## svm    0.4708705 0.4835490 0.5017030 0.5020811 0.5135315 0.5594391    0
## xgb    0.4335335 0.4726310 0.4808734 0.4862154 0.4999162 0.5422476    0
## enet   0.5740713 0.5876342 0.6192317 0.6132220 0.6366709 0.6497864    0
## lm     0.5591706 0.5960663 0.6197742 0.6130768 0.6301045 0.6695080    0
## knn    0.4462196 0.4717578 0.4893428 0.4885298 0.5057886 0.5332419    0
## 
## Rsquared 
##             Min.   1st Qu.    Median      Mean   3rd Qu.      Max. NA's
## rf     0.8251137 0.8379160 0.8449772 0.8473371 0.8572410 0.8765737    0
## ranger 0.8314684 0.8411991 0.8630997 0.8589568 0.8711225 0.8841194    0
## svm    0.7694407 0.7892370 0.7987342 0.8014904 0.8232202 0.8326393    0
## xgb    0.7753951 0.7991493 0.8144542 0.8133090 0.8278964 0.8500155    0
## enet   0.6722763 0.6794478 0.7058135 0.7028195 0.7208017 0.7380072    0
## lm     0.6766263 0.6804220 0.6947861 0.7024158 0.7244885 0.7458092    0
## knn    0.7592300 0.7856492 0.8190506 0.8108781 0.8268775 0.8510122    0

Model Selection

Perfect! Let’s tidy up the results a bit and compare the models.

results_tidy <- 
  results$values %>% 
  gather(metric, value, 2:ncol(.)) %>% 
  spread(Resample, value) %>% 
  separate(metric, into = c("model", "metric"), sep = "~") %>% 
  gather(fold, value, starts_with("Fold")) %>% 
  mutate(fold = fold %>% str_extract(one_or_more(DGT)) %>% as.integer()) %>% 
  group_by(model, metric) %>% 
  mutate(mean = mean(value),
         median = median(value))

make_table(results_tidy)
model metric fold value mean median
enet MAE 1 0.4825111 0.4876278 0.4831610
enet RMSE 1 0.6245842 0.6132220 0.6192317
enet Rsquared 1 0.7060524 0.7028195 0.7058135
knn MAE 1 0.3880842 0.3763415 0.3813050
knn RMSE 1 0.4840062 0.4885298 0.4893428
knn Rsquared 1 0.8283164 0.8108781 0.8190506
lm MAE 1 0.5064197 0.4870978 0.4900356
lm RMSE 1 0.6345520 0.6130768 0.6197742
lm Rsquared 1 0.6790741 0.7024158 0.6947861
ranger MAE 1 0.3168925 0.3242100 0.3236273
ranger RMSE 1 0.4228394 0.4230535 0.4115117
ranger Rsquared 1 0.8595644 0.8589568 0.8630997
rf MAE 1 0.3390805 0.3374934 0.3410172
rf RMSE 1 0.4240145 0.4403527 0.4434299
rf Rsquared 1 0.8765737 0.8473371 0.8449772
svm MAE 1 0.3459667 0.3903180 0.3904982
svm RMSE 1 0.4724766 0.5020811 0.5017030
svm Rsquared 1 0.8286495 0.8014904 0.7987342
xgb MAE 1 0.3566903 0.3820884 0.3764419
xgb RMSE 1 0.4518704 0.4862154 0.4808734
xgb Rsquared 1 0.8500155 0.8133090 0.8144542
enet MAE 2 0.5105647 0.4876278 0.4831610
enet RMSE 2 0.6151434 0.6132220 0.6192317
enet Rsquared 2 0.7114941 0.7028195 0.7058135
knn MAE 2 0.3429505 0.3763415 0.3813050
knn RMSE 2 0.4477465 0.4885298 0.4893428
knn Rsquared 2 0.8510122 0.8108781 0.8190506
lm MAE 2 0.4541978 0.4870978 0.4900356
lm RMSE 2 0.5591706 0.6130768 0.6197742
lm Rsquared 2 0.7458092 0.7024158 0.6947861
ranger MAE 2 0.3328485 0.3242100 0.3236273
ranger RMSE 2 0.4119001 0.4230535 0.4115117
ranger Rsquared 2 0.8721119 0.8589568 0.8630997
rf MAE 2 0.3400141 0.3374934 0.3410172
rf RMSE 2 0.4408245 0.4403527 0.4434299
rf Rsquared 2 0.8602933 0.8473371 0.8449772
svm MAE 2 0.3943577 0.3903180 0.3904982
svm RMSE 2 0.4990411 0.5020811 0.5017030
svm Rsquared 2 0.7884983 0.8014904 0.7987342
xgb MAE 2 0.4197747 0.3820884 0.3764419
xgb RMSE 2 0.5422476 0.4862154 0.4808734
xgb Rsquared 2 0.7753951 0.8133090 0.8144542
enet MAE 3 0.4838110 0.4876278 0.4831610
enet RMSE 3 0.6233199 0.6132220 0.6192317
enet Rsquared 3 0.6854782 0.7028195 0.7058135
knn MAE 3 0.3706502 0.3763415 0.3813050
knn RMSE 3 0.4676750 0.4885298 0.4893428
knn Rsquared 3 0.8108291 0.8108781 0.8190506
lm MAE 3 0.4932059 0.4870978 0.4900356
lm RMSE 3 0.6176185 0.6130768 0.6197742
lm Rsquared 3 0.6844660 0.7024158 0.6947861
ranger MAE 3 0.3098746 0.3242100 0.3236273
ranger RMSE 3 0.4111233 0.4230535 0.4115117
ranger Rsquared 3 0.8666350 0.8589568 0.8630997
rf MAE 3 0.3085957 0.3374934 0.3410172
rf RMSE 3 0.4091449 0.4403527 0.4434299
rf Rsquared 3 0.8640769 0.8473371 0.8449772
svm MAE 3 0.4051697 0.3903180 0.3904982
svm RMSE 3 0.5149224 0.5020811 0.5017030
svm Rsquared 3 0.7914533 0.8014904 0.7987342
xgb MAE 3 0.3775190 0.3820884 0.3764419
xgb RMSE 3 0.4703394 0.4862154 0.4808734
xgb Rsquared 3 0.8177710 0.8133090 0.8144542
enet MAE 4 0.4504183 0.4876278 0.4831610
enet RMSE 4 0.5869277 0.6132220 0.6192317
enet Rsquared 4 0.7055747 0.7028195 0.7058135
knn MAE 4 0.3745257 0.3763415 0.3813050
knn RMSE 4 0.5227695 0.4885298 0.4893428
knn Rsquared 4 0.7592300 0.8108781 0.8190506
lm MAE 4 0.4788527 0.4870978 0.4900356
lm RMSE 4 0.6056407 0.6130768 0.6197742
lm Rsquared 4 0.6784126 0.7024158 0.6947861
ranger MAE 4 0.3697525 0.3242100 0.3236273
ranger RMSE 4 0.4773464 0.4230535 0.4115117
ranger Rsquared 4 0.8319152 0.8589568 0.8630997
rf MAE 4 0.3149646 0.3374934 0.3410172
rf RMSE 4 0.4460353 0.4403527 0.4434299
rf Rsquared 4 0.8480839 0.8473371 0.8449772
svm MAE 4 0.3772797 0.3903180 0.3904982
svm RMSE 4 0.4948083 0.5020811 0.5017030
svm Rsquared 4 0.8069320 0.8014904 0.7987342
xgb MAE 4 0.3458124 0.3820884 0.3764419
xgb RMSE 4 0.4335335 0.4862154 0.4808734
xgb Rsquared 4 0.8396079 0.8133090 0.8144542
enet MAE 5 0.5270272 0.4876278 0.4831610
enet RMSE 5 0.6466436 0.6132220 0.6192317
enet Rsquared 5 0.6722763 0.7028195 0.7058135
knn MAE 5 0.3955173 0.3763415 0.3813050
knn RMSE 5 0.5091001 0.4885298 0.4893428
knn Rsquared 5 0.7753238 0.8108781 0.8190506
lm MAE 5 0.4889867 0.4870978 0.4900356
lm RMSE 5 0.6219299 0.6130768 0.6197742
lm Rsquared 5 0.7046977 0.7024158 0.6947861
ranger MAE 5 0.3402081 0.3242100 0.3236273
ranger RMSE 5 0.4535646 0.4230535 0.4115117
ranger Rsquared 5 0.8314684 0.8589568 0.8630997
rf MAE 5 0.3484341 0.3374934 0.3410172
rf RMSE 5 0.4586532 0.4403527 0.4434299
rf Rsquared 5 0.8382168 0.8473371 0.8449772
svm MAE 5 0.3955761 0.3903180 0.3904982
svm RMSE 5 0.5093587 0.5020811 0.5017030
svm Rsquared 5 0.7967089 0.8014904 0.7987342
xgb MAE 5 0.4315655 0.3820884 0.3764419
xgb RMSE 5 0.5340736 0.4862154 0.4808734
xgb Rsquared 5 0.7798784 0.8133090 0.8144542
enet MAE 6 0.4768907 0.4876278 0.4831610
enet RMSE 6 0.5812898 0.6132220 0.6192317
enet Rsquared 6 0.7308714 0.7028195 0.7058135
knn MAE 6 0.3884868 0.3763415 0.3813050
knn RMSE 6 0.4926795 0.4885298 0.4893428
knn Rsquared 6 0.8167848 0.8108781 0.8190506
lm MAE 6 0.4779484 0.4870978 0.4900356
lm RMSE 6 0.5928748 0.6130768 0.6197742
lm Rsquared 6 0.7344141 0.7024158 0.6947861
ranger MAE 6 0.3303621 0.3242100 0.3236273
ranger RMSE 6 0.4081366 0.4230535 0.4115117
ranger Rsquared 6 0.8681542 0.8589568 0.8630997
rf MAE 6 0.3530204 0.3374934 0.3410172
rf RMSE 6 0.4481483 0.4403527 0.4434299
rf Rsquared 6 0.8332426 0.8473371 0.8449772
svm MAE 6 0.3852898 0.3903180 0.3904982
svm RMSE 6 0.4797959 0.5020811 0.5017030
svm Rsquared 6 0.8297439 0.8014904 0.7987342
xgb MAE 6 0.3947987 0.3820884 0.3764419
xgb RMSE 6 0.5054136 0.4862154 0.4808734
xgb Rsquared 6 0.7960545 0.8133090 0.8144542
enet MAE 7 0.4654529 0.4876278 0.4831610
enet RMSE 7 0.5740713 0.6132220 0.6192317
enet Rsquared 7 0.7380072 0.7028195 0.7058135
knn MAE 7 0.3939398 0.3763415 0.3813050
knn RMSE 7 0.4860061 0.4885298 0.4893428
knn Rsquared 7 0.8225608 0.8108781 0.8190506
lm MAE 7 0.5425569 0.4870978 0.4900356
lm RMSE 7 0.6695080 0.6130768 0.6197742
lm Rsquared 7 0.6958741 0.7024158 0.6947861
ranger MAE 7 0.3420317 0.3242100 0.3236273
ranger RMSE 7 0.4611669 0.4230535 0.4115117
ranger Rsquared 7 0.8366392 0.8589568 0.8630997
rf MAE 7 0.3420204 0.3374934 0.3410172
rf RMSE 7 0.4399831 0.4403527 0.4434299
rf Rsquared 7 0.8462708 0.8473371 0.8449772
svm MAE 7 0.4107364 0.3903180 0.3904982
svm RMSE 7 0.5157333 0.5020811 0.5017030
svm Rsquared 7 0.7694407 0.8014904 0.7987342
xgb MAE 7 0.3706242 0.3820884 0.3764419
xgb RMSE 7 0.4795059 0.4862154 0.4808734
xgb Rsquared 7 0.8264024 0.8133090 0.8144542
enet MAE 8 0.4977206 0.4876278 0.4831610
enet RMSE 8 0.6406998 0.6132220 0.6192317
enet Rsquared 8 0.6770990 0.7028195 0.7058135
knn MAE 8 0.3372234 0.3763415 0.3813050
knn RMSE 8 0.4462196 0.4885298 0.4893428
knn Rsquared 8 0.8461515 0.8108781 0.8190506
lm MAE 8 0.4430682 0.4870978 0.4900356
lm RMSE 8 0.5726946 0.6130768 0.6197742
lm Rsquared 8 0.7310855 0.7024158 0.6947861
ranger MAE 8 0.3013981 0.3242100 0.3236273
ranger RMSE 8 0.3862598 0.4230535 0.4115117
ranger Rsquared 8 0.8841194 0.8589568 0.8630997
rf MAE 8 0.3147840 0.3374934 0.3410172
rf RMSE 8 0.4171260 0.4403527 0.4434299
rf Rsquared 8 0.8436835 0.8473371 0.8449772
svm MAE 8 0.3866387 0.3903180 0.3904982
svm RMSE 8 0.5043649 0.5020811 0.5017030
svm Rsquared 8 0.8007595 0.8014904 0.7987342
xgb MAE 8 0.3753648 0.3820884 0.3764419
xgb RMSE 8 0.4813756 0.4862154 0.4808734
xgb Rsquared 8 0.8084336 0.8133090 0.8144542
enet MAE 9 0.4642721 0.4876278 0.4831610
enet RMSE 9 0.5897537 0.6132220 0.6192317
enet Rsquared 9 0.7239043 0.7028195 0.7058135
knn MAE 9 0.3728178 0.3763415 0.3813050
knn RMSE 9 0.4958540 0.4885298 0.4893428
knn Rsquared 9 0.8213164 0.8108781 0.8190506
lm MAE 9 0.4946577 0.4870978 0.4900356
lm RMSE 9 0.6249587 0.6130768 0.6197742
lm Rsquared 9 0.6766263 0.7024158 0.6947861
ranger MAE 9 0.3013837 0.3242100 0.3236273
ranger RMSE 9 0.4075934 0.4230535 0.4115117
ranger Rsquared 9 0.8548790 0.8589568 0.8630997
rf MAE 9 0.3624754 0.3374934 0.3410172
rf RMSE 9 0.4553058 0.4403527 0.4434299
rf Rsquared 9 0.8378157 0.8473371 0.8449772
svm MAE 9 0.4220382 0.3903180 0.3904982
svm RMSE 9 0.5594391 0.5020811 0.5017030
svm Rsquared 9 0.7700786 0.8014904 0.7987342
xgb MAE 9 0.3836106 0.3820884 0.3764419
xgb RMSE 9 0.4803712 0.4862154 0.4808734
xgb Rsquared 9 0.8111373 0.8133090 0.8144542
enet MAE 10 0.5176096 0.4876278 0.4831610
enet RMSE 10 0.6497864 0.6132220 0.6192317
enet Rsquared 10 0.6774376 0.7028195 0.7058135
knn MAE 10 0.3992195 0.3763415 0.3813050
knn RMSE 10 0.5332419 0.4885298 0.4893428
knn Rsquared 10 0.7772558 0.8108781 0.8190506
lm MAE 10 0.4910845 0.4870978 0.4900356
lm RMSE 10 0.6318198 0.6130768 0.6197742
lm Rsquared 10 0.6936982 0.7024158 0.6947861
ranger MAE 10 0.2973487 0.3242100 0.3236273
ranger RMSE 10 0.3906047 0.4230535 0.4115117
ranger Rsquared 10 0.8840810 0.8589568 0.8630997
rf MAE 10 0.3515446 0.3374934 0.3410172
rf RMSE 10 0.4642917 0.4403527 0.4434299
rf Rsquared 10 0.8251137 0.8473371 0.8449772
svm MAE 10 0.3801269 0.3903180 0.3904982
svm RMSE 10 0.4708705 0.5020811 0.5017030
svm Rsquared 10 0.8326393 0.8014904 0.7987342
xgb MAE 10 0.3651238 0.3820884 0.3764419
xgb RMSE 10 0.4834239 0.4862154 0.4808734
xgb Rsquared 10 0.8283943 0.8133090 0.8144542
results_tidy %>% 
  filter(metric == "RMSE") %>% 
  ggplot(aes(fold, value, col = model)) +
  geom_point() +
  geom_line() + 
  scale_color_brewer(palette = "Set1") +
  geom_hline(aes(yintercept = mean, col = model), alpha = 0.5, linetype = 2) +
  theme_light()

It’s a tight competition between ranger and random forest but ranger is the winner! It has the lowest average RMSE over the ten folds.

Testing our model/Diagnostics

Now let’s add predictions of our model to the test set and compare the actual with the predicted results.

df_test <- 
  df_test %>% 
  as_tibble() %>% 
  mutate(pred_life_ladder = predict(model_ranger, df_test)) %>% 
  select(country, life_ladder, pred_life_ladder, everything())

make_table(df_test)
country life_ladder pred_life_ladder year log_gdp_per_capita social_support healthy_life_expectancy freedom_to_make_life_choices generosity perceptions_of_corruption positive_affect negative_affect confidence_in_national_government democratic_quality delivery_quality std_ladder std_mean_ladder gini_index_world_bank_mean gini_of_household_income country_recoded continent
afghanistan 2.661718 4.017316 2017 -1.4229621 0.4908801 -1.2486931 0.4270109 -0.1063403 0.9543926 0.4963486 0.3713262 0.2611785 -0.4581462 -0.4174628 1.454051 0.5462828 0.3180000 -1.9222618 afghanistan asia
albania 4.639548 4.816166 2017 0.0768763 0.6376983 0.8852481 0.7496110 -0.0351404 0.8761346 0.6692409 0.3338841 0.4577375 -0.1100228 -0.2762941 2.682105 0.5780961 0.3032500 -0.1962243 albania europe
algeria 5.248912 5.560166 2017 0.2235255 0.8067539 0.4100662 0.4366705 -0.1946701 0.6997742 0.6419796 0.2887100 0.3585822 0.7669195 0.7532876 2.039765 0.3886071 0.2760000 0.9035290 algeria africa
argentina 6.039330 6.420210 2017 0.4972352 0.9066991 0.6678732 0.8319662 -0.1862999 0.8410525 0.8094226 0.2917173 0.3054303 -0.2388228 -0.9772229 2.409329 0.3989398 0.4760667 -0.3818571 argentina south america
armenia 4.287736 4.638295 2017 -0.2136873 0.6979249 0.3311460 0.6136971 -0.1321662 0.8646833 0.6250138 0.4371487 0.2469010 -0.3743018 -0.1511640 2.325379 0.5423325 0.3250667 0.4889734 armenia europe
australia 7.257038 7.307685 2017 1.3282599 0.9499578 1.4419869 0.9105502 0.3016933 0.4113465 0.7800789 0.2253610 0.4534070 1.3044155 1.8880664 1.745699 0.2405526 0.3427500 -0.0037446 australia oceania
austria 7.293728 7.294841 2017 1.3326481 0.9062178 1.3773105 0.8900306 0.1249967 0.5183038 0.7475687 0.1802687 0.4359079 1.3946700 1.8331888 1.561062 0.2140280 0.3038182 -1.3279177 austria europe
azerbaijan 5.152279 5.099732 2017 0.3251246 0.7870394 0.0546507 0.7310305 -0.2474807 0.6525390 0.5923592 0.1983187 0.7665834 0.7958571 0.8398616 1.673757 0.3248575 0.2288333 -1.4698256 azerbaijan europe
bahrain 6.227321 6.363974 2017 1.3032382 0.8757471 0.4689702 0.9058585 0.1207592 0.4229599 0.8135706 0.2897595 0.6191422 -1.2365553 0.2570486 2.215526 0.3557751 0.3630000 0.1827084 bahrain asia
bangladesh 4.309771 4.661218 2017 -0.9116217 0.7125528 -0.0207959 0.8962172 0.0175362 0.6350145 0.5688271 0.2135058 0.8766465 -0.8995469 0.0731710 1.865366 0.4328225 0.3290000 0.9448176 bangladesh asia
belarus 5.552915 5.215606 2017 0.3845694 0.9002558 0.5362607 0.6209792 -0.1305821 0.6541134 0.5409058 0.2327681 0.4479163 -0.0655526 1.0616212 1.662564 0.2994038 0.2818750 -1.9819299 belarus europe
belgium 6.928348 7.112010 2017 1.2743590 0.9216391 1.3443600 0.8568020 0.0454815 0.5430461 0.7863676 0.2335977 0.4497319 1.5007312 2.0093009 1.510013 0.2179471 0.2851818 -0.8800333 belgium europe
benin 4.853181 3.962099 2017 -1.3018425 0.4358790 -1.3083658 0.7268082 -0.0402694 0.7672346 0.6147223 0.4579203 0.6150893 -2.0174518 -1.5616895 2.998127 0.6177652 0.4326667 1.2770679 benin africa
bolivia 5.650553 5.695395 2017 -0.3818178 0.7786618 -0.2999585 0.8839049 -0.1159879 0.8192620 0.6981954 0.4339435 0.4276330 -0.6442907 -0.9633100 2.250484 0.3982768 0.5316429 -0.2476040 bolivia south america
bosnia and herzegovina 5.089902 5.073434 2017 0.0751259 0.7752953 0.7328509 0.5637987 0.0895895 0.9233431 0.5973417 0.2707455 0.1263797 0.7226796 0.4367597 2.281007 0.4481435 0.3272500 0.5547962 bosnia and herzegovina europe
botswana 3.504881 4.217853 2017 0.3518914 0.7682586 -0.5901079 0.8173080 -0.2582493 0.7314415 0.6563964 0.2762525 0.7708211 -0.3462968 -0.4135146 2.669426 0.7616308 0.6260000 1.6824290 botswana africa
brazil 6.332929 5.956961 2017 0.2301458 0.9046943 0.3762045 0.7647926 -0.1872647 0.7944574 0.7159451 0.3077169 0.1654903 0.0449757 -0.2094517 2.530506 0.3995791 0.5502143 -0.4000528 brazil south america
bulgaria 5.096902 5.117217 2017 0.4784112 0.9417546 0.5096091 0.6890470 -0.1600387 0.9107997 0.6142167 0.1886373 0.3260880 0.3959904 0.6298721 1.852364 0.3634294 0.3546667 -1.0328491 bulgaria europe
burkina faso 4.646891 4.027802 2017 -1.4407243 0.7847614 -1.2420885 0.6137747 -0.0512868 0.7274513 0.5851725 0.3538211 0.5406561 -0.6819565 -0.9642334 2.197592 0.4729166 0.3946667 1.1020029 burkina africa
cambodia 4.585842 4.562147 2017 -0.8815323 0.7650949 -0.5143242 0.9637747 0.0867194 0.8210229 0.7986166 0.4082838 0.7524425 -0.3530679 -1.1852673 2.732651 0.5958887 0.4971667 1.0454409 cambodia asia
cameroon 5.074051 4.801686 2017 -0.9379374 0.6945964 -1.4857685 0.7669450 -0.0310556 0.8435856 0.6323388 0.3774987 0.5787755 -0.4631247 -1.1766773 2.815224 0.5548277 0.4380000 1.1916923 cameroon africa
central african republic 3.475862 4.130709 2017 -1.8857113 0.3195891 -2.0336207 0.6452523 0.2374405 0.8895660 0.6138652 0.5993355 0.6502849 -1.7732567 -1.7015454 2.807074 0.8075907 0.4990000 2.1318140 central african republic africa
chad 4.558937 4.240524 2017 -1.4016320 0.6606156 -1.9060005 0.6148497 -0.0029291 0.7923899 0.6275032 0.5382454 0.4735444 -0.9781567 -1.3047615 2.880961 0.6319369 0.4155000 1.1643848 chad africa
chile 6.320119 6.431622 2017 0.6768699 0.8798408 0.9665295 0.7901165 -0.0352652 0.8359875 0.8384748 0.2910421 0.2682833 0.2225020 0.2862112 2.162420 0.3421486 0.4915714 1.2597736 chile south america
china 5.099062 5.233031 2017 0.3085426 0.7720332 0.9217964 0.8776176 -0.1946347 0.9563479 0.8210973 0.2140048 0.4660428 0.8578598 0.8119916 2.271664 0.4455063 0.4250000 1.6541159 china asia
colombia 6.157342 6.201875 2017 0.1797674 0.9092500 0.1965517 0.8375546 -0.1658781 0.8750181 0.8369270 0.2993090 0.2218826 -0.7372795 -0.0366263 2.544821 0.4132987 0.5525000 1.0172742 colombia south america
congo (brazzaville) 4.883991 4.322436 2017 -0.6026026 0.6554405 -0.9147252 0.7777835 -0.1687490 0.7627831 0.5989516 0.3816406 0.4698265 -0.9940138 -0.6955176 2.906801 0.5951692 0.4810000 0.4742696 congo africa
congo (kinshasa) 4.311033 3.746063 2017 -1.9704455 0.6696884 -1.4169748 0.7042395 0.0957925 0.8091819 0.5505259 0.4042621 0.4784123 -0.4332103 -0.7772939 2.642479 0.6129573 0.4215000 1.6484158 congo africa
costa rica 7.225182 7.010093 2017 0.3401547 0.9216971 1.0044342 0.9356185 -0.0782685 0.7423508 0.8743958 0.2754401 0.4074484 1.1489949 1.3093905 2.079984 0.2878798 0.4913125 0.4010203 costa rica north america
croatia 5.343166 5.489495 2017 0.6550328 0.7703097 0.5938236 0.7158223 -0.1094173 0.8915597 0.6553050 0.3164882 0.2110007 0.9402379 0.8239273 1.570821 0.2939870 0.3233333 -1.1025282 croatia europe
cyprus 6.062051 6.150609 2017 0.9843979 0.8186710 1.4388414 0.8116707 0.0362867 0.8512064 0.7841884 0.3005166 0.3126534 0.8268501 0.7457061 2.369525 0.3908784 0.3249091 -0.2001011 cyprus europe
czech republic 6.789568 6.665329 2017 1.0154101 0.9009690 1.2398533 0.8317855 -0.1845901 0.8665250 0.7387440 0.2266498 0.3438708 1.0083290 1.3179376 1.697134 0.2499621 0.2647273 -1.9874020 czech republic europe
denmark 7.593702 7.378589 2017 1.3653928 0.9521001 1.2716876 0.9554163 0.1453871 0.1811476 0.8236674 0.2057753 0.5723533 -0.2427972 1.5226122 1.668074 0.2196655 0.2670909 1.6380396 denmark europe
dominican republic 5.605203 5.501704 2017 0.2719291 0.8943681 0.1106589 0.8553590 -0.1270922 0.7604896 0.7389436 0.2747455 0.4599013 -0.8905090 -0.6851090 3.240837 0.5781838 0.4886875 0.5073997 dominican republic north america
ecuador 5.839518 5.906809 2017 -0.0534017 0.8489417 0.6399744 0.8791282 -0.1699555 0.7335888 0.8291420 0.3143435 0.6405129 -0.3229064 -0.8901181 2.693175 0.4611982 0.5059286 -0.1648151 ecuador south america
egypt 3.929344 4.751855 2017 -0.0215919 0.6382264 -0.1422215 0.5925048 -0.1619447 0.9568846 0.5393226 0.4144938 0.2640005 -0.7611743 -0.2743320 2.289861 0.5827590 0.3122000 -0.5415071 egypt africa
el salvador 6.339318 5.837183 2017 -0.2418477 0.8289526 0.2202757 0.7578273 -0.1802910 0.7777486 0.8488506 0.2684484 0.2702351 0.1473476 -0.2128177 2.477156 0.3907606 0.4611875 -0.2893726 el salvador north america
estonia 5.938396 5.908524 2017 0.9117952 0.9356863 0.6318066 0.8617492 -0.1095883 0.6684023 0.8052180 0.1601641 0.4059463 0.7236322 1.3859609 1.851840 0.3118418 0.3330000 -0.3536482 estonia europe
ethiopia 4.180315 3.920832 2017 -1.4374343 0.7335398 -0.7451926 0.7171012 0.0118712 0.7568989 0.6085148 0.3044362 0.7683543 -0.3762134 -1.0779244 1.829916 0.4377459 0.3150000 -0.0894701 ethiopia africa
finland 7.788252 7.155791 2017 1.2294812 0.9638264 1.2768844 0.9621990 -0.0121737 0.1924128 0.7871371 0.1760658 0.5975386 0.6633090 1.6066872 1.416398 0.1818634 0.2759091 -0.0324583 finland europe
france 6.635222 6.680901 2017 1.1786479 0.9314946 1.4122599 0.8338901 -0.1335850 0.6014860 0.7620978 0.2419839 0.3747935 1.4020228 1.4408134 1.631064 0.2458191 0.3203000 -0.6401198 france europe
gabon 4.782383 4.221481 2017 0.3807705 0.8069412 -0.6975400 0.6523595 -0.2508322 0.8683055 0.6340472 0.4461243 0.2869892 -0.9374192 -0.9370857 2.457131 0.5137880 0.4220000 1.3624710 gabon africa
georgia 4.450775 4.548145 2017 -0.0997843 0.5904952 0.2251585 0.8209088 -0.2314226 0.5896320 0.5811277 0.2096396 0.2963962 0.5489326 0.9032170 1.775080 0.3988250 0.4034375 0.2580324 georgia europe
germany 7.074325 7.350655 2017 1.3276178 0.8921661 1.1840929 0.8407279 0.1353075 0.4140212 0.7365657 0.1964348 0.6229357 0.7124424 1.7486047 1.441730 0.2037976 0.3115714 -1.3788970 germany europe
ghana 5.481311 4.378613 2017 -0.7857988 0.6691111 -0.9638039 0.7830464 0.0849905 0.8386100 0.7025117 0.2475189 0.6702886 -1.4030557 -0.9966525 2.551120 0.4654214 0.4250000 0.6855881 ghana africa
greece 5.148242 5.330590 2017 0.7518866 0.7528995 1.2894683 0.4383000 -0.2967350 0.8722395 0.6029390 0.3328308 0.1398809 -0.2388228 -0.2222434 2.411631 0.4684378 0.3469091 -0.9843989 greece europe
guatemala 6.325119 5.964455 2017 -0.3104897 0.8264921 0.0954421 0.9145217 -0.0617286 0.7997479 0.8458658 0.3080862 0.4586239 0.1424020 -0.6119473 2.858946 0.4519988 0.5267500 0.6348980 guatemala north america
guinea 4.873723 3.721186 2017 -1.3759748 0.6340256 -1.3779870 0.7382128 0.0543076 0.7500262 0.7044771 0.4224609 0.6399808 -1.6155609 -1.3743873 2.969935 0.6093772 0.3870000 1.5580059 guinea africa
haiti 3.823866 3.887599 2017 -1.4623858 0.6469850 -1.1366927 0.4844292 0.3845103 0.6471916 0.5733667 0.3216928 0.4350770 -1.2895987 -0.8154979 2.156945 0.5640745 0.4090000 1.2585280 haiti north america
honduras 6.019986 5.354558 2017 -0.7254962 0.8433549 0.1450148 0.8983774 0.0800741 0.7834294 0.8422009 0.2483833 0.4955850 -0.5374142 -1.2152498 2.963586 0.4922912 0.5556000 1.1646781 honduras north america
hong kong s.a.r. of china 5.362475 5.750028 2017 1.5350473 0.8310663 2.0314568 0.8306572 0.1229347 0.4158102 0.6395335 0.2005934 0.5450407 0.7593640 1.6203333 1.784560 0.3327865 0.3115714 0.4193198 china asia
hungary 6.065039 5.733893 2017 0.8234938 0.8767475 0.6189317 0.6611660 -0.1507930 0.8863613 0.7351843 0.1809213 0.3777705 0.6184236 0.6810377 1.792409 0.2955314 0.2968182 -1.2114690 hungary europe
iceland 7.476214 7.458142 2017 1.3768300 0.9667528 1.4377986 0.9387833 0.2354787 0.7268454 0.8952546 0.1481600 0.3650419 1.4794390 1.6856226 1.630091 0.2180369 0.2800909 -1.1868230 iceland europe
india 4.046111 4.737448 2017 -0.4343909 0.6067675 -0.4059988 0.8858504 -0.0545028 0.7808028 0.6820910 0.3179368 0.8392784 -1.0195436 -0.5396550 1.659828 0.4102279 0.3520000 -0.5501045 india asia
indonesia 5.098401 5.080524 2017 0.0320021 0.7955892 -0.2693514 0.8650263 0.4729739 0.9004164 0.8625842 0.3191716 0.8257654 -0.4637251 -1.0022614 2.325179 0.4560604 0.3950000 0.2133539 indonesia asia
iran 4.716783 5.214825 2017 0.5063416 0.7142329 0.4468176 0.7306352 0.1827351 0.7149413 0.6936660 0.4385339 0.4647121 1.2286601 1.6173073 2.497179 0.5294242 0.4132000 2.7878599 iran asia
iraq 4.462399 4.774311 2017 0.2971129 0.6951094 -0.2103162 0.6277220 -0.0384290 0.7571088 0.5052885 0.5905387 0.4240508 -0.0828087 0.1875076 2.899485 0.6497592 0.2905000 1.0810523 iraq asia
ireland 7.060155 7.279974 2017 1.6878865 0.9434820 1.2788190 0.9053411 0.2068017 0.3370847 0.8333889 0.2127841 0.6037996 1.4930528 1.9775437 1.761334 0.2494753 0.3255455 -0.7086066 ireland europe
israel 7.331036 7.134320 2017 1.0276825 0.9164408 1.3022722 0.7680763 0.1371585 0.7926522 0.6735913 0.2764432 0.3893681 1.4831723 1.4090226 1.522683 0.2077036 0.4110000 -2.7230207 israel asia
italy 6.198870 6.185548 2017 1.0921747 0.9197912 1.6508399 0.6328433 -0.0433807 0.8666679 0.6611807 0.3228461 0.2301963 0.8591065 0.6883198 1.927247 0.3109029 0.3412727 -0.9843652 italy europe
ivory coast 5.037735 4.698664 2017 -0.8980375 0.6613753 -1.8076287 0.7320979 -0.0953538 0.7709402 0.6977355 0.3574558 0.5166543 -0.3631718 -0.3678862 2.612090 0.5185049 0.4206667 1.1379769 ivory coast africa
jamaica 5.889759 5.777196 2017 -0.2211261 0.9130298 0.4266125 0.8606763 -0.1308327 0.8827962 0.7692822 0.2433996 0.3482018 -0.6548837 0.0987899 2.399338 0.4073745 0.4690000 1.1865544 jamaica north america
japan 5.910677 5.846263 2017 1.1859226 0.8819613 1.8367337 0.8493966 -0.2232968 0.6591986 0.7403876 0.1755122 0.4124900 1.2632596 1.7049079 1.950257 0.3299550 0.3210000 -0.0267603 japan asia
jordan 4.808083 4.797463 2017 -0.2215726 0.8146645 0.2346974 0.7662625 -0.1515229 0.9252855 0.6277984 0.3915052 0.7084055 -0.3132641 -0.2792303 2.523892 0.5249270 0.3430000 -0.7582029 jordan asia
kazakhstan 5.882351 5.617226 2017 0.7089777 0.9140932 0.2052205 0.7452440 -0.0491785 0.7552507 0.7567622 0.1714864 0.7557058 -0.7724240 -0.1746587 1.745324 0.2967051 0.2950667 -1.1174124 kazakhstan asia
kenya 4.475654 4.455347 2017 -1.0267139 0.7146043 -0.5070477 0.8533944 0.2501793 0.8540004 0.7884522 0.2302103 0.6378626 -1.0506376 -0.4738760 2.532363 0.5658085 0.4850000 0.8533562 kenya africa
kosovo 6.149200 5.135297 2017 0.5063416 0.7920873 -0.0354673 0.8576767 -0.1345799 0.9251918 0.7384356 0.1858791 0.2981961 0.5334914 0.0885412 2.432453 0.3955723 0.4373333 -0.2911018 serbia europe
kuwait 6.093905 6.189327 2017 1.7346699 0.8534913 0.3659139 0.8841816 -0.0485043 0.8590730 0.6920719 0.3073208 0.5306982 0.9522760 1.3178231 2.355753 0.3865753 0.5055000 1.3806490 kuwait asia
kyrgyzstan 5.629537 5.041148 2017 -0.9503076 0.8825867 0.0313731 0.8593898 0.1666674 0.8744945 0.7551252 0.1604383 0.5616727 -1.1866615 -0.8154979 1.901391 0.3377527 0.3063750 -0.5252173 kyrgyzstan asia
laos 4.623141 4.803245 2017 -0.4831245 0.7073358 -0.6076166 0.8910007 0.0764826 0.5916168 0.8727918 0.3442259 0.8399350 -0.2607035 -0.4992208 1.881974 0.4070769 0.3480000 1.3336328 laos asia
latvia 5.977817 5.654418 2017 0.7665156 0.8950987 0.3288958 0.6995201 -0.1638314 0.7983781 0.6233128 0.2317531 0.2640005 0.2704116 0.0419507 1.814803 0.3035896 0.3620000 -0.7766907 latvia europe
lebanon 5.153990 5.233359 2017 0.1804021 0.7765830 0.8580088 0.6045542 -0.0750281 0.9107273 0.5154439 0.2435492 0.2177181 -0.2502708 -0.0888470 1.929768 0.3744221 0.3180000 -1.2722744 lebanon asia
liberia 4.424491 3.467717 2017 -1.9679318 0.6848668 -1.1992283 0.7333899 0.0405768 0.8668064 0.6679465 0.3913305 0.5549915 -0.8469424 -1.7871207 3.393740 0.7670351 0.3485000 1.3736731 liberia africa
libya 5.646852 4.937271 2017 -0.0061642 0.8227588 -0.1537528 0.7786959 -0.0320096 0.6730655 0.6970493 0.3793742 0.6329269 -1.1504506 -0.7463109 2.507788 0.4441037 0.4971667 1.3404942 libya africa
lithuania 6.272941 5.862871 2017 0.9087429 0.9263166 0.5975158 0.7493073 -0.1822766 0.6815087 0.6076131 0.1951188 0.3212473 -0.9348004 -0.2227103 2.175804 0.3468556 0.3530909 -0.0020331 lithuania europe
luxembourg 7.061381 6.878814 2017 2.1062015 0.9054355 1.3532240 0.9028217 0.0297714 0.3301736 0.7658169 0.1844670 0.7389221 1.2337433 2.0125904 1.424476 0.2017277 0.3153636 -1.8218759 luxembourg europe
macedonia 5.233867 5.030792 2017 0.1910658 0.7999551 0.4345063 0.7521066 -0.0627284 0.8556973 0.5024599 0.2993906 0.4429165 -1.7502819 -0.8227038 2.111887 0.4035042 0.3901667 -0.6609156 macedonia europe
madagascar 4.078620 3.832729 2017 -1.5673952 0.6263320 -0.7133473 0.5703479 -0.0255079 0.8472608 0.7516981 0.3748380 0.5419161 -0.9657959 -0.9554395 2.328915 0.5710055 0.4310000 1.2859314 madagascar africa
malawi 3.416863 3.870356 2017 -1.7309996 0.5554227 -1.0362459 0.8479208 0.0215601 0.7346366 0.6086665 0.3120883 0.6203325 -0.2666336 -0.3472236 2.760293 0.8078443 0.4300000 1.2555401 malawi africa
mali 4.741850 4.235569 2017 -1.3188334 0.7413594 -1.5862200 0.7532133 -0.0645473 0.8626552 0.7418358 0.3927841 0.5048201 -0.4001058 -0.8142959 2.511668 0.5296810 0.3726667 1.0708788 mali africa
malta 6.675666 6.628908 2017 1.1216935 0.9373318 1.2968961 0.9236429 0.2417625 0.6904945 0.7207534 0.3024431 0.7520052 0.8970146 1.4583614 1.994550 0.2987792 0.3412727 -0.0962482 malta europe
mauritania 4.678160 4.420782 2017 -0.8928821 0.7792252 -1.1380021 0.5274468 -0.1347656 0.7773141 0.6371071 0.2723219 0.2921097 -1.0931803 -0.9397162 2.130550 0.4554248 0.3682500 0.0543634 mauritania africa
mauritius 6.174118 5.415187 2017 0.5626760 0.9101422 0.3948711 0.9123075 0.0739294 0.8181799 0.7477602 0.1687212 0.6146708 1.0568773 0.6393213 2.247791 0.3640667 0.3575000 -0.1397209 mauritius africa
mexico 6.410299 6.839648 2017 0.4046311 0.7998394 0.7477022 0.8614051 -0.2066392 0.8008931 0.8426423 0.2309909 0.2569532 0.5724148 1.1001015 2.236207 0.3488460 0.4847500 0.1039464 mexico north america
moldova 5.325531 4.801347 2017 -0.6138957 0.8307679 0.1381203 0.5528252 -0.0031789 0.9263338 0.5814888 0.2594778 0.1327296 -0.8776916 -0.6657077 1.885169 0.3539871 0.3299375 0.0465339 moldova europe
mongolia 5.333850 5.011784 2017 0.0288609 0.9242508 -0.0459439 0.6746274 0.1064513 0.8649522 0.6751653 0.2135995 0.2750584 0.0708450 0.3260131 2.095162 0.3928047 0.3358333 -0.4386800 mongolia asia
montenegro 5.614799 5.016963 2017 0.3616705 0.8811998 0.6117000 0.6259063 -0.0848908 0.7556800 0.5191281 0.3497851 0.3515499 0.2316854 -0.0351703 2.207512 0.3931596 0.3093000 -0.9060958 montenegro europe
morocco 5.312483 5.144761 2017 -0.3136984 0.6411931 0.3604002 0.8142580 -0.2279194 0.8405024 0.5592882 0.3227159 0.4072287 -0.6954367 -0.3511469 2.287168 0.4305272 0.4065000 0.9068164 morocco africa
mozambique 4.279863 3.725567 2017 -1.7022998 0.6784637 -1.5049819 0.8226705 -0.0215026 0.6821089 0.6483814 0.3531765 0.7592344 -0.1104992 -0.9842538 2.985020 0.6974568 0.4630000 1.4955083 mozambique africa
myanmar 4.154342 4.544248 2017 -0.5429304 0.7951838 -0.6353624 0.8860118 0.6287057 0.6188215 0.7456475 0.2822855 0.8241765 0.3866161 0.0804656 2.233866 0.5377184 0.3810000 -0.0478506 burma (myanmar) asia
namibia 4.441306 4.407504 2017 -0.0704735 0.8283390 -0.8907235 0.8104019 -0.2017659 0.8313029 0.7206782 0.2772522 0.7458536 -1.1100354 -1.3035891 2.739773 0.6168846 0.6215000 1.4546764 namibia africa
nepal 4.736692 4.072224 2017 -1.1805377 0.8163833 -0.1819974 0.8451476 0.1465111 0.7701771 0.5705770 0.3759779 0.5595785 -1.3907533 -1.3352239 2.468454 0.5211344 0.3830000 0.3488191 nepal asia
netherlands 7.458965 7.312706 2017 1.4083436 0.9365013 1.2652817 0.9203197 0.2374405 0.3631336 0.8521849 0.1845198 0.6695536 1.4218887 2.0206099 1.318480 0.1767644 0.2869091 0.1359619 netherlands europe
new zealand 7.327183 7.278460 2017 1.0953820 0.9549206 1.2644242 0.9422795 0.2850567 0.2218875 0.8174313 0.1717166 0.6090155 1.1398493 1.8711811 1.650336 0.2252347 0.3358333 0.1810478 new zealand oceania
nicaragua 6.476356 6.043261 2017 -0.5903558 0.8380437 0.4867643 0.9221628 0.0103267 0.6729633 0.8497642 0.3084479 0.5904252 -0.3295509 -0.6400592 2.657253 0.4103006 0.4827500 0.1447876 nicaragua north america
niger 4.615673 3.944304 2017 -1.8414465 0.5821096 -1.3592173 0.6835576 -0.0122363 0.7776600 0.7311614 0.4265224 0.6011173 -1.1617272 -0.7260407 2.780398 0.6023818 0.3680000 1.8135260 niger africa
nigeria 5.321928 4.950811 2017 -0.5821239 0.7334685 -1.9167164 0.8259056 0.1133015 0.8348920 0.7249905 0.2359688 0.5587907 -1.1866615 -1.3047615 2.947903 0.5539164 0.4155000 0.3944104 nigeria africa
norway 7.578745 7.492901 2017 1.7036661 0.9501277 1.1852121 0.9530168 0.2101040 0.2497114 0.8491000 0.2029138 0.7171597 0.8633111 1.9581583 1.621726 0.2139835 0.2717273 -1.2504214 norway europe
pakistan 5.830871 4.559519 2017 -0.6351751 0.6902636 -0.6569407 0.7126571 0.0303281 0.7139282 0.5861670 0.3083411 0.5574352 -0.2498085 -0.8233779 1.778068 0.3049404 0.3125714 -0.1834219 pakistan asia
palestinian territories 4.628133 5.403525 2017 -0.0707620 0.8243451 0.0542771 0.6316113 0.1129390 0.8306463 0.5967656 0.4160720 0.4144009 0.4825428 0.7380756 2.521674 0.5448578 0.3358333 0.4888666 israel asia
panama 6.567659 6.583463 2017 0.6443385 0.9119048 0.7401383 0.8995735 -0.1640805 0.8407771 0.8326888 0.2423193 0.3909961 -0.0468420 0.4915157 2.677188 0.4076320 0.5367500 0.4475849 panama north america
peru 5.710936 5.849397 2017 0.1115427 0.8301234 0.3698047 0.8265522 -0.1667474 0.8953841 0.7893909 0.3938737 0.2545953 -0.7507532 -1.1852673 2.594314 0.4542712 0.4891875 0.4943494 peru south america
philippines 5.594270 5.169175 2017 -0.2938108 0.8510286 -0.3116673 0.9257031 -0.1478517 0.7111655 0.7689815 0.3406215 0.8377301 0.3001381 0.1805593 2.597185 0.4642581 0.4188333 0.7043263 philippines asia
poland 6.201268 6.077228 2017 0.8405007 0.8818541 0.8918081 0.8308426 -0.1331340 0.6544431 0.6774355 0.2033878 0.5024804 -0.8595597 -0.2268903 1.627443 0.2624371 0.3347333 -2.4214082 poland europe
portugal 5.711499 5.647492 2017 0.8648439 0.8999848 1.3467606 0.9050657 -0.1825984 0.8809705 0.6491511 0.2942726 0.5010881 0.7859262 0.3885097 2.205087 0.3860784 0.3670000 0.1719510 portugal europe
romania 6.089905 5.663697 2017 0.6687536 0.8112401 0.5715147 0.8385867 -0.1644207 0.9256580 0.7337304 0.2308362 0.2983413 0.8268501 0.8587797 2.141298 0.3516144 0.2909286 -0.1782069 romania europe
russia 5.578743 5.852995 2017 0.7388619 0.8961513 0.0418513 0.7308742 -0.1585678 0.8615902 0.7102302 0.1945605 0.5565972 -0.6396016 -0.5006492 2.124689 0.3808544 0.3992500 -0.5800258 russian federation asia
saudi arabia 6.294282 6.419571 2017 1.4286363 0.8400863 0.1895330 0.8141422 -0.1569054 0.8908113 0.7748757 0.3058419 0.4987864 -0.7693577 -0.0955827 2.508923 0.3986035 0.4250000 0.2347601 saudi arabia asia
senegal 4.683025 4.393220 2017 -1.1727764 0.7437593 -0.5940534 0.6869370 -0.0290661 0.8252419 0.7457321 0.2908361 0.6107228 -0.8630440 -0.7772939 2.007637 0.4287053 0.4023333 0.0827725 senegal africa
serbia 5.122031 4.987490 2017 0.2424637 0.8837704 0.4080882 0.6848463 -0.0814156 0.8514578 0.5097212 0.3264071 0.5111980 -0.6374220 -0.8055180 2.394510 0.4674922 0.3059000 -0.7256005 serbia europe
sierra leone 4.089562 4.480276 2017 -1.5648529 0.6522871 -2.0578596 0.7106135 0.0892257 0.8483983 0.6003682 0.4950400 0.7375169 -0.5993640 -0.6387442 2.595259 0.6346056 0.3710000 1.3231203 sierra leone africa
singapore 6.378438 6.348405 2017 1.9490187 0.8973499 1.9154409 0.9261279 0.1204116 0.1617907 0.8001144 0.1793246 0.9269980 0.8493201 2.1213119 1.418072 0.2223228 0.2717273 1.1892952 singapore asia
slovakia 6.365509 6.327187 2017 0.9387440 0.9133866 0.8449816 0.7142247 -0.0720768 0.9204227 0.7880204 0.2127225 0.3416801 0.1093233 0.1283761 1.897522 0.2980943 0.2674545 -1.8177706 slovakia europe
slovenia 6.166838 6.354189 2017 0.9768260 0.9281878 1.1638041 0.9208627 -0.0334751 0.8287947 0.6151097 0.2856014 0.2397797 1.3127520 1.4257897 1.982301 0.3214453 0.2490909 -0.8152576 slovenia europe
south africa 4.513655 4.628088 2017 0.1044049 0.8703133 -0.9217178 0.7874279 -0.1392562 0.8647819 0.7848007 0.2681752 0.4232742 0.9450007 0.0630881 2.419570 0.5360554 0.6225000 1.1902312 south africa africa
south korea 5.873887 5.996473 2017 1.1090211 0.8069299 1.6363289 0.5381137 0.0033546 0.8506904 0.6233776 0.2348256 0.3585822 0.3418537 0.8603101 2.075730 0.3533826 0.3190000 -0.3884483 korea, south asia
south sudan 2.816622 3.950575 2017 -1.5849110 0.5568227 -1.4984168 0.4560111 0.2877157 0.7612696 0.5856022 0.5173638 0.4615505 -0.0477808 -0.0572206 2.736838 0.9716735 0.4630000 2.0644874 south sudan africa
spain 6.230173 6.151628 2017 1.0656339 0.9031582 1.7076579 0.7555608 -0.0421563 0.7912688 0.6250556 0.3023877 0.2695860 -0.1750235 0.6287525 1.805722 0.2898349 0.3445455 -0.5643747 spain europe
sri lanka 4.330946 5.413085 2017 0.0805749 0.8227706 0.3540347 0.8270774 0.0845772 0.8442100 0.7949516 0.2697275 0.5175899 -0.0468420 -0.4863982 2.230376 0.5149859 0.3922500 -0.0147107 sri lanka asia
sweden 7.286805 7.404961 2017 1.3868512 0.9140168 1.4762161 0.9345821 0.1554868 0.2393669 0.8135478 0.1750668 0.5594620 1.5300021 2.0004740 1.777177 0.2438897 0.2674545 -0.9823262 sweden europe
switzerland 7.473593 7.505019 2017 1.5741350 0.9496614 1.5019273 0.9249969 0.1678747 0.3161835 0.7739971 0.1958715 0.8197067 1.0338043 1.5931270 1.544616 0.2066764 0.3275714 -1.7625697 switzerland europe
taiwan province of china 6.359451 6.326316 2017 1.0599010 0.8911191 1.2036843 0.7596548 0.2316328 0.7427801 0.8372773 0.1141232 0.3228224 0.7735811 0.9065953 1.869602 0.2939880 0.3530909 -0.5310481 china asia
tajikistan 5.829234 5.051929 2017 -1.0666096 0.6626933 0.0408532 0.8320025 0.1249362 0.7183369 0.6026677 0.2777253 0.9297931 -1.8774925 -0.8575408 2.212851 0.3796126 0.3187500 -0.7022099 tajikistan asia
tanzania 3.347121 4.242359 2017 -1.1111003 0.7050103 -0.7526810 0.8004955 0.1065989 0.6536061 0.7146455 0.2553357 0.9133390 -1.3225747 -0.6387442 2.683667 0.8017837 0.3846667 1.0735261 tanzania africa
thailand 5.938895 5.382503 2017 0.3562018 0.8772687 0.5100707 0.9228968 0.2024524 0.8838168 0.8163218 0.2315976 0.6050788 0.4315332 0.3551119 2.243722 0.3778013 0.4026364 0.7879228 thailand asia
togo 4.360805 3.961140 2017 -1.5657636 0.5078052 -1.2577428 0.7166944 -0.0377045 0.7255196 0.6141890 0.4258243 0.4849524 -0.5551566 -1.2705966 2.494831 0.5721032 0.4373333 1.0948111 togo africa
trinidad and tobago 6.191860 6.126328 2017 0.9482791 0.9160290 -0.1208818 0.8591405 -0.0146044 0.9113364 0.8464671 0.2480988 0.2725409 0.3126907 1.1192414 2.276677 0.3676887 0.4611875 -0.1414822 trinidad and tobago north america
tunisia 4.124343 4.894247 2017 0.0078371 0.7173816 0.4351693 0.4779567 -0.2335312 0.8688267 0.4209619 0.3771968 0.2508273 -1.0253228 -0.4737499 2.214818 0.5370110 0.3810000 -0.1518759 tunisia africa
turkey 5.607262 5.721013 2017 0.7536353 0.8764678 0.4232812 0.6444342 -0.2469272 0.6709105 0.4498009 0.3128461 0.5861613 -0.0662526 -0.0352495 1.626329 0.2900397 0.4030769 -1.2684928 turkey asia
turkmenistan 5.229149 5.423382 2017 0.3690251 0.9084549 -0.2937486 0.7203992 0.0403375 0.8506471 0.5208851 0.3496277 0.4691116 -0.4449278 -0.2622029 1.432178 0.2738836 0.3180000 -2.3088805 turkmenistan asia
uganda 4.000517 3.959368 2017 -1.4348918 0.7399561 -1.3117724 0.7723441 0.0717200 0.8157700 0.7033756 0.4000257 0.6551091 -0.7574137 -1.2584507 2.673379 0.6682585 0.4332500 1.6997086 uganda africa
ukraine 4.311067 4.392610 2017 -0.2697226 0.8583249 0.0676698 0.5988755 0.0161250 0.9367640 0.5971122 0.2347638 0.1109365 -0.2607035 -0.3211868 2.255384 0.5231613 0.2660000 -0.0800391 ukraine europe
united arab emirates 7.039420 6.599910 2017 1.7450380 0.8355274 0.8171893 0.9620166 0.1911131 0.3015630 0.7950349 0.2075984 0.5384948 0.9023601 2.0093009 2.030041 0.2883818 0.3430000 1.4408398 united arab emirates asia
united kingdom 7.103273 7.117429 2017 1.2020815 0.9374953 1.3311943 0.8127333 0.2810460 0.4186113 0.7585718 0.2095724 0.4401208 1.1509768 1.5931270 1.615450 0.2274234 0.3420000 -0.6479590 united kingdom europe
united states 6.991759 6.966566 2017 1.5174761 0.9210029 0.9902774 0.8684967 0.1816574 0.6811913 0.8265553 0.2682691 0.3865350 1.0957146 1.4408134 1.880688 0.2689864 0.4068000 1.5084886 united states north america
uruguay 6.336010 6.487475 2017 0.5655538 0.9138017 0.7882050 0.8978516 -0.1047622 0.6265820 0.8358613 0.2803234 0.4130321 0.5053725 0.7161235 2.288599 0.3612050 0.4426000 -0.3155511 uruguay south america
uzbekistan 6.421448 4.922974 2017 -0.4390257 0.9421311 0.0667168 0.9851778 0.1140295 0.4646416 0.8389891 0.2027375 0.9646904 -0.6854888 -0.4806659 2.367689 0.3687157 0.3480000 0.1765125 uzbekistan asia
vietnam 5.175279 4.972014 2017 -0.4554275 0.7937337 0.4859254 0.8228476 -0.0450106 0.8000467 0.7259707 0.2754401 0.3367194 -0.9348004 -0.9645731 1.750836 0.3383076 0.3642857 0.3741068 vietnam asia
yemen 3.253560 4.117635 2017 -1.3963986 0.7895550 -0.9504997 0.5951908 -0.1247060 0.8683055 0.4551819 0.2950636 0.2477870 -0.6660423 -0.7591940 2.178544 0.6695878 0.3570000 -0.6192464 yemen asia
zambia 3.932777 4.514745 2017 -0.8745857 0.7437539 -1.0852099 0.8231686 0.1298215 0.7395406 0.6846233 0.3871889 0.7170041 0.2879881 -0.1937660 2.650283 0.6738960 0.5274000 1.5216509 zambia africa
zimbabwe 3.638300 3.935189 2017 -1.3684624 0.7541471 -1.2116933 0.7528261 -0.0660047 0.7512080 0.8064284 0.2240513 0.6826467 -0.3287278 -1.1423872 2.656848 0.7302443 0.4320000 1.3083362 zimbabwe africa
plt_predicted_actual <- 
  df_test %>% 
  ggplot(aes(life_ladder, pred_life_ladder, col = continent, label = country)) +
  geom_abline(intercept = 0, slope = 1, alpha = 0.5) +
  geom_point(size = 3, alpha = 0.8) +
  scale_x_continuous(limits = c(2,8)) +
  scale_y_continuous(limits = c(2,8)) +
  theme_light()

ggplotly(plt_predicted_actual)

Our model does a pretty neat job predicting happiness. Except in the lower tail where it’s a little off, the overall predictions are seem decent. Some countries actually have spot on predictions! On an average it is about 0.4676796 steps off.

Best and Worst Predictions

Let’s take a look at our best and worst predictions.

test_acc <- 
  df_test %>% 
  select(country, continent, life_ladder, pred_life_ladder) %>% 
  mutate(resid = life_ladder - pred_life_ladder,
         resid_abs = abs(resid),
         closeness = ifelse(resid_abs < mean(resid_abs), "close", "not_close"))

make_table(test_acc)
country continent life_ladder pred_life_ladder resid resid_abs closeness
afghanistan asia 2.661718 4.017316 -1.3555975 1.3555975 not_close
albania europe 4.639548 4.816166 -0.1766177 0.1766177 close
algeria africa 5.248912 5.560166 -0.3112540 0.3112540 close
argentina south america 6.039330 6.420210 -0.3808804 0.3808804 not_close
armenia europe 4.287736 4.638295 -0.3505584 0.3505584 not_close
australia oceania 7.257038 7.307685 -0.0506472 0.0506472 close
austria europe 7.293728 7.294841 -0.0011131 0.0011131 close
azerbaijan europe 5.152279 5.099732 0.0525473 0.0525473 close
bahrain asia 6.227321 6.363974 -0.1366537 0.1366537 close
bangladesh asia 4.309771 4.661218 -0.3514473 0.3514473 not_close
belarus europe 5.552915 5.215606 0.3373091 0.3373091 close
belgium europe 6.928348 7.112010 -0.1836621 0.1836621 close
benin africa 4.853181 3.962099 0.8910820 0.8910820 not_close
bolivia south america 5.650553 5.695395 -0.0448425 0.0448425 close
bosnia and herzegovina europe 5.089902 5.073434 0.0164681 0.0164681 close
botswana africa 3.504881 4.217853 -0.7129713 0.7129713 not_close
brazil south america 6.332929 5.956961 0.3759686 0.3759686 not_close
bulgaria europe 5.096902 5.117217 -0.0203148 0.0203148 close
burkina faso africa 4.646891 4.027802 0.6190887 0.6190887 not_close
cambodia asia 4.585842 4.562147 0.0236950 0.0236950 close
cameroon africa 5.074051 4.801686 0.2723651 0.2723651 close
central african republic africa 3.475862 4.130709 -0.6548474 0.6548474 not_close
chad africa 4.558937 4.240524 0.3184136 0.3184136 close
chile south america 6.320119 6.431622 -0.1115026 0.1115026 close
china asia 5.099062 5.233031 -0.1339693 0.1339693 close
colombia south america 6.157342 6.201875 -0.0445326 0.0445326 close
congo (brazzaville) africa 4.883991 4.322436 0.5615553 0.5615553 not_close
congo (kinshasa) africa 4.311033 3.746063 0.5649701 0.5649701 not_close
costa rica north america 7.225182 7.010093 0.2150888 0.2150888 close
croatia europe 5.343166 5.489495 -0.1463291 0.1463291 close
cyprus europe 6.062051 6.150609 -0.0885574 0.0885574 close
czech republic europe 6.789568 6.665329 0.1242390 0.1242390 close
denmark europe 7.593702 7.378589 0.2151134 0.2151134 close
dominican republic north america 5.605203 5.501704 0.1034984 0.1034984 close
ecuador south america 5.839518 5.906809 -0.0672909 0.0672909 close
egypt africa 3.929344 4.751855 -0.8225111 0.8225111 not_close
el salvador north america 6.339318 5.837183 0.5021356 0.5021356 not_close
estonia europe 5.938396 5.908524 0.0298718 0.0298718 close
ethiopia africa 4.180315 3.920832 0.2594833 0.2594833 close
finland europe 7.788252 7.155791 0.6324606 0.6324606 not_close
france europe 6.635222 6.680901 -0.0456784 0.0456784 close
gabon africa 4.782383 4.221481 0.5609021 0.5609021 not_close
georgia europe 4.450775 4.548145 -0.0973707 0.0973707 close
germany europe 7.074325 7.350655 -0.2763306 0.2763306 close
ghana africa 5.481311 4.378613 1.1026974 1.1026974 not_close
greece europe 5.148242 5.330590 -0.1823481 0.1823481 close
guatemala north america 6.325119 5.964455 0.3606641 0.3606641 not_close
guinea africa 4.873723 3.721186 1.1525365 1.1525365 not_close
haiti north america 3.823866 3.887599 -0.0637334 0.0637334 close
honduras north america 6.019986 5.354558 0.6654281 0.6654281 not_close
hong kong s.a.r. of china asia 5.362475 5.750028 -0.3875536 0.3875536 not_close
hungary europe 6.065039 5.733893 0.3311457 0.3311457 close
iceland europe 7.476214 7.458142 0.0180722 0.0180722 close
india asia 4.046111 4.737448 -0.6913367 0.6913367 not_close
indonesia asia 5.098401 5.080524 0.0178774 0.0178774 close
iran asia 4.716783 5.214825 -0.4980421 0.4980421 not_close
iraq asia 4.462399 4.774311 -0.3119115 0.3119115 close
ireland europe 7.060155 7.279974 -0.2198186 0.2198186 close
israel asia 7.331036 7.134320 0.1967159 0.1967159 close
italy europe 6.198870 6.185548 0.0133220 0.0133220 close
ivory coast africa 5.037735 4.698664 0.3390706 0.3390706 close
jamaica north america 5.889759 5.777196 0.1125635 0.1125635 close
japan asia 5.910677 5.846263 0.0644133 0.0644133 close
jordan asia 4.808083 4.797463 0.0106193 0.0106193 close
kazakhstan asia 5.882351 5.617226 0.2651251 0.2651251 close
kenya africa 4.475654 4.455347 0.0203073 0.0203073 close
kosovo europe 6.149200 5.135297 1.0139031 1.0139031 not_close
kuwait asia 6.093905 6.189327 -0.0954216 0.0954216 close
kyrgyzstan asia 5.629537 5.041148 0.5883884 0.5883884 not_close
laos asia 4.623141 4.803245 -0.1801046 0.1801046 close
latvia europe 5.977817 5.654418 0.3233993 0.3233993 close
lebanon asia 5.153990 5.233359 -0.0793698 0.0793698 close
liberia africa 4.424491 3.467717 0.9567742 0.9567742 not_close
libya africa 5.646852 4.937271 0.7095819 0.7095819 not_close
lithuania europe 6.272941 5.862871 0.4100692 0.4100692 not_close
luxembourg europe 7.061381 6.878814 0.1825667 0.1825667 close
macedonia europe 5.233867 5.030792 0.2030745 0.2030745 close
madagascar africa 4.078620 3.832729 0.2458913 0.2458913 close
malawi africa 3.416863 3.870356 -0.4534932 0.4534932 not_close
mali africa 4.741850 4.235569 0.5062809 0.5062809 not_close
malta europe 6.675666 6.628908 0.0467581 0.0467581 close
mauritania africa 4.678160 4.420782 0.2573777 0.2573777 close
mauritius africa 6.174118 5.415187 0.7589305 0.7589305 not_close
mexico north america 6.410299 6.839648 -0.4293488 0.4293488 not_close
moldova europe 5.325531 4.801347 0.5241840 0.5241840 not_close
mongolia asia 5.333850 5.011784 0.3220667 0.3220667 close
montenegro europe 5.614799 5.016963 0.5978360 0.5978360 not_close
morocco africa 5.312483 5.144761 0.1677220 0.1677220 close
mozambique africa 4.279863 3.725567 0.5542967 0.5542967 not_close
myanmar asia 4.154342 4.544248 -0.3899066 0.3899066 not_close
namibia africa 4.441306 4.407504 0.0338020 0.0338020 close
nepal asia 4.736692 4.072224 0.6644681 0.6644681 not_close
netherlands europe 7.458965 7.312706 0.1462593 0.1462593 close
new zealand oceania 7.327183 7.278460 0.0487224 0.0487224 close
nicaragua north america 6.476356 6.043261 0.4330956 0.4330956 not_close
niger africa 4.615673 3.944304 0.6713699 0.6713699 not_close
nigeria africa 5.321928 4.950811 0.3711169 0.3711169 not_close
norway europe 7.578745 7.492901 0.0858441 0.0858441 close
pakistan asia 5.830871 4.559519 1.2713520 1.2713520 not_close
palestinian territories asia 4.628133 5.403525 -0.7753918 0.7753918 not_close
panama north america 6.567659 6.583463 -0.0158041 0.0158041 close
peru south america 5.710936 5.849397 -0.1384600 0.1384600 close
philippines asia 5.594270 5.169175 0.4250953 0.4250953 not_close
poland europe 6.201268 6.077228 0.1240405 0.1240405 close
portugal europe 5.711499 5.647492 0.0640076 0.0640076 close
romania europe 6.089905 5.663697 0.4262080 0.4262080 not_close
russia asia 5.578743 5.852995 -0.2742518 0.2742518 close
saudi arabia asia 6.294282 6.419571 -0.1252891 0.1252891 close
senegal africa 4.683025 4.393220 0.2898051 0.2898051 close
serbia europe 5.122031 4.987490 0.1345415 0.1345415 close
sierra leone africa 4.089562 4.480276 -0.3907138 0.3907138 not_close
singapore asia 6.378438 6.348405 0.0300335 0.0300335 close
slovakia europe 6.365509 6.327187 0.0383222 0.0383222 close
slovenia europe 6.166838 6.354189 -0.1873517 0.1873517 close
south africa africa 4.513655 4.628088 -0.1144332 0.1144332 close
south korea asia 5.873887 5.996473 -0.1225857 0.1225857 close
south sudan africa 2.816622 3.950575 -1.1339524 1.1339524 not_close
spain europe 6.230173 6.151628 0.0785451 0.0785451 close
sri lanka asia 4.330946 5.413085 -1.0821395 1.0821395 not_close
sweden europe 7.286805 7.404961 -0.1181565 0.1181565 close
switzerland europe 7.473593 7.505019 -0.0314260 0.0314260 close
taiwan province of china asia 6.359451 6.326316 0.0331348 0.0331348 close
tajikistan asia 5.829234 5.051929 0.7773046 0.7773046 not_close
tanzania africa 3.347121 4.242359 -0.8952379 0.8952379 not_close
thailand asia 5.938895 5.382503 0.5563925 0.5563925 not_close
togo africa 4.360805 3.961140 0.3996649 0.3996649 not_close
trinidad and tobago north america 6.191860 6.126328 0.0655313 0.0655313 close
tunisia africa 4.124343 4.894247 -0.7699040 0.7699040 not_close
turkey asia 5.607262 5.721013 -0.1137507 0.1137507 close
turkmenistan asia 5.229149 5.423382 -0.1942330 0.1942330 close
uganda africa 4.000517 3.959368 0.0411492 0.0411492 close
ukraine europe 4.311067 4.392610 -0.0815425 0.0815425 close
united arab emirates asia 7.039420 6.599910 0.4395093 0.4395093 not_close
united kingdom europe 7.103273 7.117429 -0.0141557 0.0141557 close
united states north america 6.991759 6.966566 0.0251936 0.0251936 close
uruguay south america 6.336010 6.487475 -0.1514648 0.1514648 close
uzbekistan asia 6.421448 4.922974 1.4984743 1.4984743 not_close
vietnam asia 5.175279 4.972014 0.2032646 0.2032646 close
yemen asia 3.253560 4.117635 -0.8640748 0.8640748 not_close
zambia africa 3.932777 4.514745 -0.5819678 0.5819678 not_close
zimbabwe africa 3.638300 3.935189 -0.2968886 0.2968886 close

The countries which our model is off by a step.

df_test %>% 
  filter(!near(pred_life_ladder, life_ladder, 1)) %>% 
  make_table()
country life_ladder pred_life_ladder year log_gdp_per_capita social_support healthy_life_expectancy freedom_to_make_life_choices generosity perceptions_of_corruption positive_affect negative_affect confidence_in_national_government democratic_quality delivery_quality std_ladder std_mean_ladder gini_index_world_bank_mean gini_of_household_income country_recoded continent
afghanistan 2.661718 4.017316 2017 -1.4229621 0.4908801 -1.2486931 0.4270109 -0.1063403 0.9543926 0.4963486 0.3713262 0.2611785 -0.4581462 -0.4174628 1.454051 0.5462828 0.3180000 -1.9222618 afghanistan asia
ghana 5.481311 4.378613 2017 -0.7857988 0.6691111 -0.9638039 0.7830464 0.0849905 0.8386100 0.7025117 0.2475189 0.6702886 -1.4030557 -0.9966525 2.551120 0.4654214 0.4250000 0.6855881 ghana africa
guinea 4.873723 3.721186 2017 -1.3759748 0.6340256 -1.3779870 0.7382128 0.0543076 0.7500262 0.7044771 0.4224609 0.6399808 -1.6155609 -1.3743873 2.969935 0.6093772 0.3870000 1.5580059 guinea africa
kosovo 6.149200 5.135297 2017 0.5063416 0.7920873 -0.0354673 0.8576767 -0.1345799 0.9251918 0.7384356 0.1858791 0.2981961 0.5334914 0.0885412 2.432453 0.3955723 0.4373333 -0.2911018 serbia europe
pakistan 5.830871 4.559519 2017 -0.6351751 0.6902636 -0.6569407 0.7126571 0.0303281 0.7139282 0.5861670 0.3083411 0.5574352 -0.2498085 -0.8233779 1.778068 0.3049404 0.3125714 -0.1834219 pakistan asia
south sudan 2.816622 3.950575 2017 -1.5849110 0.5568227 -1.4984168 0.4560111 0.2877157 0.7612696 0.5856022 0.5173638 0.4615505 -0.0477808 -0.0572206 2.736838 0.9716735 0.4630000 2.0644874 south sudan africa
sri lanka 4.330946 5.413085 2017 0.0805749 0.8227706 0.3540347 0.8270774 0.0845772 0.8442100 0.7949516 0.2697275 0.5175899 -0.0468420 -0.4863982 2.230376 0.5149859 0.3922500 -0.0147107 sri lanka asia
uzbekistan 6.421448 4.922974 2017 -0.4390257 0.9421311 0.0667168 0.9851778 0.1140295 0.4646416 0.8389891 0.2027375 0.9646904 -0.6854888 -0.4806659 2.367689 0.3687157 0.3480000 0.1765125 uzbekistan asia

The countries that are predicted extremely well.

df_test %>% 
  filter(near(pred_life_ladder, life_ladder, .05)) %>% 
  make_table()
country life_ladder pred_life_ladder year log_gdp_per_capita social_support healthy_life_expectancy freedom_to_make_life_choices generosity perceptions_of_corruption positive_affect negative_affect confidence_in_national_government democratic_quality delivery_quality std_ladder std_mean_ladder gini_index_world_bank_mean gini_of_household_income country_recoded continent
austria 7.293728 7.294841 2017 1.3326481 0.9062178 1.3773105 0.8900306 0.1249967 0.5183038 0.7475687 0.1802687 0.4359079 1.3946700 1.8331888 1.561062 0.2140280 0.3038182 -1.3279177 austria europe
bolivia 5.650553 5.695395 2017 -0.3818178 0.7786618 -0.2999585 0.8839049 -0.1159879 0.8192620 0.6981954 0.4339435 0.4276330 -0.6442907 -0.9633100 2.250484 0.3982768 0.5316429 -0.2476040 bolivia south america
bosnia and herzegovina 5.089902 5.073434 2017 0.0751259 0.7752953 0.7328509 0.5637987 0.0895895 0.9233431 0.5973417 0.2707455 0.1263797 0.7226796 0.4367597 2.281007 0.4481435 0.3272500 0.5547962 bosnia and herzegovina europe
bulgaria 5.096902 5.117217 2017 0.4784112 0.9417546 0.5096091 0.6890470 -0.1600387 0.9107997 0.6142167 0.1886373 0.3260880 0.3959904 0.6298721 1.852364 0.3634294 0.3546667 -1.0328491 bulgaria europe
cambodia 4.585842 4.562147 2017 -0.8815323 0.7650949 -0.5143242 0.9637747 0.0867194 0.8210229 0.7986166 0.4082838 0.7524425 -0.3530679 -1.1852673 2.732651 0.5958887 0.4971667 1.0454409 cambodia asia
colombia 6.157342 6.201875 2017 0.1797674 0.9092500 0.1965517 0.8375546 -0.1658781 0.8750181 0.8369270 0.2993090 0.2218826 -0.7372795 -0.0366263 2.544821 0.4132987 0.5525000 1.0172742 colombia south america
estonia 5.938396 5.908524 2017 0.9117952 0.9356863 0.6318066 0.8617492 -0.1095883 0.6684023 0.8052180 0.1601641 0.4059463 0.7236322 1.3859609 1.851840 0.3118418 0.3330000 -0.3536482 estonia europe
france 6.635222 6.680901 2017 1.1786479 0.9314946 1.4122599 0.8338901 -0.1335850 0.6014860 0.7620978 0.2419839 0.3747935 1.4020228 1.4408134 1.631064 0.2458191 0.3203000 -0.6401198 france europe
iceland 7.476214 7.458142 2017 1.3768300 0.9667528 1.4377986 0.9387833 0.2354787 0.7268454 0.8952546 0.1481600 0.3650419 1.4794390 1.6856226 1.630091 0.2180369 0.2800909 -1.1868230 iceland europe
indonesia 5.098401 5.080524 2017 0.0320021 0.7955892 -0.2693514 0.8650263 0.4729739 0.9004164 0.8625842 0.3191716 0.8257654 -0.4637251 -1.0022614 2.325179 0.4560604 0.3950000 0.2133539 indonesia asia
italy 6.198870 6.185548 2017 1.0921747 0.9197912 1.6508399 0.6328433 -0.0433807 0.8666679 0.6611807 0.3228461 0.2301963 0.8591065 0.6883198 1.927247 0.3109029 0.3412727 -0.9843652 italy europe
jordan 4.808083 4.797463 2017 -0.2215726 0.8146645 0.2346974 0.7662625 -0.1515229 0.9252855 0.6277984 0.3915052 0.7084055 -0.3132641 -0.2792303 2.523892 0.5249270 0.3430000 -0.7582029 jordan asia
kenya 4.475654 4.455347 2017 -1.0267139 0.7146043 -0.5070477 0.8533944 0.2501793 0.8540004 0.7884522 0.2302103 0.6378626 -1.0506376 -0.4738760 2.532363 0.5658085 0.4850000 0.8533562 kenya africa
malta 6.675666 6.628908 2017 1.1216935 0.9373318 1.2968961 0.9236429 0.2417625 0.6904945 0.7207534 0.3024431 0.7520052 0.8970146 1.4583614 1.994550 0.2987792 0.3412727 -0.0962482 malta europe
namibia 4.441306 4.407504 2017 -0.0704735 0.8283390 -0.8907235 0.8104019 -0.2017659 0.8313029 0.7206782 0.2772522 0.7458536 -1.1100354 -1.3035891 2.739773 0.6168846 0.6215000 1.4546764 namibia africa
new zealand 7.327183 7.278460 2017 1.0953820 0.9549206 1.2644242 0.9422795 0.2850567 0.2218875 0.8174313 0.1717166 0.6090155 1.1398493 1.8711811 1.650336 0.2252347 0.3358333 0.1810478 new zealand oceania
panama 6.567659 6.583463 2017 0.6443385 0.9119048 0.7401383 0.8995735 -0.1640805 0.8407771 0.8326888 0.2423193 0.3909961 -0.0468420 0.4915157 2.677188 0.4076320 0.5367500 0.4475849 panama north america
singapore 6.378438 6.348405 2017 1.9490187 0.8973499 1.9154409 0.9261279 0.1204116 0.1617907 0.8001144 0.1793246 0.9269980 0.8493201 2.1213119 1.418072 0.2223228 0.2717273 1.1892952 singapore asia
slovakia 6.365509 6.327187 2017 0.9387440 0.9133866 0.8449816 0.7142247 -0.0720768 0.9204227 0.7880204 0.2127225 0.3416801 0.1093233 0.1283761 1.897522 0.2980943 0.2674545 -1.8177706 slovakia europe
switzerland 7.473593 7.505019 2017 1.5741350 0.9496614 1.5019273 0.9249969 0.1678747 0.3161835 0.7739971 0.1958715 0.8197067 1.0338043 1.5931270 1.544616 0.2066764 0.3275714 -1.7625697 switzerland europe
taiwan province of china 6.359451 6.326316 2017 1.0599010 0.8911191 1.2036843 0.7596548 0.2316328 0.7427801 0.8372773 0.1141232 0.3228224 0.7735811 0.9065953 1.869602 0.2939880 0.3530909 -0.5310481 china asia
uganda 4.000517 3.959368 2017 -1.4348918 0.7399561 -1.3117724 0.7723441 0.0717200 0.8157700 0.7033756 0.4000257 0.6551091 -0.7574137 -1.2584507 2.673379 0.6682585 0.4332500 1.6997086 uganda africa
united kingdom 7.103273 7.117429 2017 1.2020815 0.9374953 1.3311943 0.8127333 0.2810460 0.4186113 0.7585718 0.2095724 0.4401208 1.1509768 1.5931270 1.615450 0.2274234 0.3420000 -0.6479590 united kingdom europe
united states 6.991759 6.966566 2017 1.5174761 0.9210029 0.9902774 0.8684967 0.1816574 0.6811913 0.8265553 0.2682691 0.3865350 1.0957146 1.4408134 1.880688 0.2689864 0.4068000 1.5084886 united states north america

Conclusion

And what does a machine learning model have to say about which variables it found most useful to predict?

plot(varImp(model_rf))

The continents are obviously very useful. But the most important predictor is GDP!

Let me know your comments, questions, feedback in the comments below.. Thanks for reading!

comments powered by Disqus