pydatatable
by Selva Prabhakaran | Posted on August 31, 2019
101 Python datatable Exercises (pydatatable)
Python datatable is the newest package for data manipulation and analysis in Python. It carries the spirit of R’s data.table with similar syntax. It is super fast, much faster than pandas and has the ability to work with out-of-memory data. Looking at the performance it is on path to become a must-use package for data manipulation in python.
101 Python datatable Exercises (pydatatable). Photo by Jet Kim.
1. How to import datatable package and check the version?
Difficulty Level: L1
import datatable as dt
dt.__version__
# '0.8.0'You need to import datatable as dt for the rest of the codes in this exercise to work.
2. How to create a datatable Frame from a list, numpy array, pandas dataframe?
Difficulty Level: L1
Question: Create a datatable Frame from a list, numpy array and pandas dataframe.
Input:
import pandas as pd
import numpy as np
my_list = list('abcedfghijklmnopqrstuvwxyz')
my_arr = np.arange(26)
my_df = pd.DataFrame(dict(col1=my_list, col2=my_arr))Desired Output:
import pandas as pd
import numpy as np
import datatable as dt
# Inputs
my_list = list('abcedfghijklmnopqrstuvwxyz')
my_arr = np.arange(26)
my_df = pd.DataFrame(dict(col1=my_list, col2=my_arr))
# Solution
dt_df1 = dt.Frame(my_list)
dt_df2 = dt.Frame(my_arr)
dt_df3 = dt.Frame(my_df)
dt_df4 = dt.Frame(A=my_arr, B= my_list)3. How to import csv file as a pydatatable Frame?
Difficulty Level: L1
Question: Read files as datatable Frame.Show Solution
Input: BostonHousing dataset
# Solution
import datatable as dt
df = dt.fread('https://raw.githubusercontent.com/selva86/datasets/master/BostonHousing.csv')
df.head(5)
4. How to read first 5 rows of pydatatable Frame ?
Difficulty Level: L1
Question: Read first 5 rows of datatable Frame.
Input URL for CSV file: https://raw.githubusercontent.com/selva86/datasets/master/BostonHousing.csv
# Solution
import datatable as dt
df = dt.fread('https://raw.githubusercontent.com/selva86/datasets/master/BostonHousing.csv', max_nrows= 5)
df
5. How to add new column in pydatatable Frame from a list?
Difficulty Level: L1
Question: Read first 5 rows of datatable Frame and add a new column of length 5.
Input URL for CSV file: https://raw.githubusercontent.com/selva86/datasets/master/BostonHousing.csv
# Input
import datatable as dt
df = dt.fread('https://raw.githubusercontent.com/selva86/datasets/master/BostonHousing.csv', max_nrows= 5)
# Solution
df[:,"new_column"] = dt.Frame([1,2,3,4,5])
df
6. How to do addition of existing columns to get a new column in pydatatable Frame?
Difficulty Level: L1
Question: Add age and rad columns to get a new column in datatable Frame.
Input: BostonHousing dataset
# Input
import datatable as dt
df = dt.fread('https://raw.githubusercontent.com/selva86/datasets/master/BostonHousing.csv')
# Solution
df[:,"new_column"] = df[:, dt.f.age + dt.f.rad]7. How to get the int value of a float column in a pydatatable Frame?
Difficulty Level: L1
Question: Get the int value of a float column dis in datatable Frame.
Input: BostonHousing datasetShow Solution
# Input
import datatable as dt
df = dt.fread('https://raw.githubusercontent.com/selva86/datasets/master/BostonHousing.csv')
# Solution
df[:, "new_column"] = df[:, dt.int32(dt.f.dis)]
df.head(5)8. How to create a new column based on a condition in a datatable Frame?
Difficulty Level: L2
Question: Create a new column having value as ‘Old’ if age greater than 60 else ‘New’ in a `datatable` Frame.
Input: BostonHousing datasetShow Solution
import datatable as dt
df = dt.fread('https://raw.githubusercontent.com/selva86/datasets/master/BostonHousing.csv')
df[:, "new_column"] = dt.Frame(np.where(df[:, dt.f.age > 60], 'Old', 'New'))
df.head(5)9. How to left join two datatable Frames?
Difficulty Level: L1
Question: join two Frames.
Input:
import datatable as dt
df1 = dt.Frame(A=[1,2,3,4],B=["a", "b", "c", "d"])
df2 = dt.Frame(A=[1,2,3,4,5],C=["a2", "b2", "c2", "d2", "e2"])Primary Key : AShow Solution
import datatable as dt
df1 = dt.Frame(A=[1,2,3,4],B=["a", "b", "c", "d"])
df2 = dt.Frame(A=[1,2,3,4,5],C=["a2", "b2", "c2", "d2", "e2"])
df2.key = "A"
output = df1[:, :, dt.join(df2)]
output10. How to rename a column in a pydatatable Frame?
Difficulty Level: L1
Question: Rename column zn to zn_new in a datatable Frame.
Input: BostonHousing dataset
import datatable as dt
df = dt.fread('https://raw.githubusercontent.com/selva86/datasets/master/BostonHousing.csv')
df.names = {'zn': 'zn_new'}
df.head(5)11. How to import every 50th row from a csv file to create a datatable Frame?
Difficiulty Level: L2
Question: Import every 50th row of [BostonHousing dataset] (BostonHousing.csv) as a dataframe.
Input: BostonHousing dataset
# Solution: Use csv reader. Unfortunately there isn't an option to do it directly using fread()
import datatable as dt
import csv
with open('local/path/to/BostonHousing.csv', 'r') as f:
reader = csv.reader(f)
for i, row in enumerate(reader):
row = [[x] for x in row]
# 1st row
if i == 0:
df = dt.Frame(row)
header = [x[0] for x in df[0,:].to_list()]
df.names = header
del df[0,:]
# Every 50th row
elif i%50 ==0:
df_temp = dt.Frame(row)
df_temp.names = header
df.rbind(df_temp)
df.head(5)12. How to change column values when importing csv to a Python datatable Frame?
Difficulty Level: L2
Question: Import the boston housing dataset, but while importing change the 'medv' (median house value) column so that values < 25 becomes ‘Low’ and > 25 becomes ‘High’.
Input: BostonHousing datasetShow Solution
# Solution: Use csv reader
import datatable as dt
import csv
with open('https://raw.githubusercontent.com/selva86/datasets/master/BostonHousing.csv', 'r') as f:
reader = csv.reader(f)
for i, row in enumerate(reader):
row = [[x] for x in row]
if i == 0:
df = dt.Frame(row)
header = [x[0] for x in df[0,:].to_list()]
df.names = header
del df[0,:]
else:
row[13] = ['High'] if float(row[13][0]) > 25 else ['Low']
df_temp = dt.Frame(row)
df_temp.names = header
df.rbind(df_temp)
df.head(5)13. How to change value at particular row and column in a Python datatable Frame?
Difficulty Level: L1
Question: Change value at row number 2 and column number 1 as 5 in a datatable Frame.
Input: BostonHousing dataset
# Solution: It follows row, column indexing. No need to use "loc", ".loc"
import datatable as dt
df = dt.fread('https://raw.githubusercontent.com/selva86/datasets/master/BostonHousing.csv')
df[2,1] = 5
df.head(5)14. How to delete specific cell, row, column, row per condition in a datatable Frame?
Difficulty Level: 2
Questions:
Delete the cell at position
2,1.Delete the
3rdrow.Delete the
chascolumn.Delete rows where column
znis having 0 value.
Input: BostonHousing dataset
# Solution: It follows row,colume indexing. No need to use "loc", ".loc"
import datatable as dt
df = dt.fread('https://raw.githubusercontent.com/selva86/datasets/master/BostonHousing.csv')
# Delete the cell at position `2,1`.
del df[2,1]
# Delete the `3rd` row.
del df[3,:]
# Delete the `chas` column.
del df[:,"chas"]
# Delete rows where column `zn` is having 0 value.
del df[dt.f.zn == 0,:]
df.head(5)15. How to convert datatable Frame to pandas, numpy, dictionary, list, tuples, csv files?
Difficulty Level: L1
Question: Convert datatable Frame to pandas, numpy, dictionary, list, tuples, csv files.
Input: BostonHousing dataset
# Solution
import datatable as dt
df = dt.fread('https://raw.githubusercontent.com/selva86/datasets/master/BostonHousing.csv')
# to pandas df
pd_df = df.to_pandas()
# to numpy arrays
np_arrays = df.to_numpy()
# to dictionary
dic = df.to_dict()
# to list
list_ = df[:,"indus"].to_list()
# to tuple
tuples_ = df[:,"indus"].to_tuples()
# to csv
df.to_csv("BostonHousing.csv")16. How to get data types of all the columns in the datatable Frame?
Difficulty Level: L1
Question: Get data types of all the columns in the datatable Frame.
Input: BostonHousing dataset
Desired Output:
crim : stype.float64
zn : stype.float64
indus : stype.float64
chas : stype.bool8
nox : stype.float64
rm : stype.float64
age : stype.float64
dis : stype.float64
rad : stype.int32
tax : stype.int32
ptratio : stype.float64
b : stype.float64
lstat : stype.float64
medv : stype.float64Show Solution
17. How to get summary stats of each column in datatable Frame?
Difficulty Level: L1
Questions:
For each column:
Get the sum of the column values.
Get the max of the column values.
Get the min of the column values.
Get the mean of the column values.
Get the standard deviation of the column values.
Get the mode of the column values.
Get the modal value of the column values.
Get the number of unique values in column.
Input: BostonHousing dataset
# Solution
import datatable as dt
df = dt.fread('https://raw.githubusercontent.com/selva86/datasets/master/BostonHousing.csv')
df.sum()
df.max()
df.min()
df.mean()
df.sd()
df.mode()
df.nmodal()
df.nunique()18. How to get the column stats of particular column of the datatable Frame?
Difficulty Level: L1
Question: Get the max value of zn column of the datatable Frame
Input: BostonHousing dataset
Desired Output: 100
# Input
import datatable as dt
df = dt.fread('https://raw.githubusercontent.com/selva86/datasets/master/BostonHousing.csv')
df[:,dt.max(dt.f.zn)]19. How to apply group by functions in datatable Frame?
Difficulty Level: L1
Question: Find the mean price for every manufacturer using Cars93 dataset.
Input: Cars93
Desired Output:
Manufacturer C0
0 None 28.550000
1 Acura 15.900000
2 Audi 33.400000
3 BMW 30.000000
4 Buick 21.625000
5 Cadillac 37.400000
..
..
30 Volkswagen 18.025000
31 Volvo 22.700000# Solution
import datatable as dt
df = dt.fread('https://raw.githubusercontent.com/selva86/datasets/master/Cars93_miss.csv')
df[:, dt.mean(dt.f.Price), dt.by("Manufacturer")].head(5)20. How to arrange datatabe Frame in ascending order by column value?
Difficulty Level: L1
Question: Arrange datatable Frame in ascending order by Price.
Input: Cars93
Desired Output:
Manufacturer Model Type Min.Price Price Max.Price MPG.city \
0 Saturn SL Small 9.2 NaN 12.9 NaN
1 Toyota Camry Midsize 15.2 NaN 21.2 22.0
2 Ford Festiva Small 6.9 7.4 7.9 31.0
3 Hyundai Excel Small 6.8 8.0 9.2 29.0
4 Mazda 323 Small 7.4 8.3 9.1 29.0
Width Turn.circle Rear.seat.room Luggage.room Weight Origin \
0 68.0 40.0 26.5 NaN 2495.0 USA
1 70.0 38.0 28.5 15.0 3030.0 non-USA
2 63.0 33.0 26.0 12.0 1845.0 USA
3 63.0 35.0 26.0 11.0 2345.0 non-USA
4 66.0 34.0 27.0 16.0 2325.0 non-USA
Make
0 Saturn SL
1 Toyota Camry
2 Ford Festiva
3 Hyundai Excel
4 Mazda 323 import datatable as dt
df = dt.fread('https://raw.githubusercontent.com/selva86/datasets/master/Cars93_miss.csv')
# Solution1
df.sort("Price")
# Solution2
df[:,:, dt.sort(dt.f.Price)].head(5)21. How to arrange datatabe Frame in descending order by column value?
Difficulty Level: L1
Question: Arrange datatable Frame in descending order by Price.
Input: Cars93
Desired Output:
Manufacturer Model Type Min.Price Price Max.Price MPG.city \
0 Mercedes-Benz 300E Midsize 43.8 61.9 80.0 19.0
1 Infiniti Q45 Midsize 45.4 47.9 NaN 17.0
2 Cadillac Seville Midsize 37.5 40.1 42.7 16.0
3 Chevrolet Corvette Sporty 34.6 38.0 41.5 17.0
4 Audi 100 Midsize NaN 37.7 44.6 19.0
MPG.highway AirBags DriveTrain ... Passengers Length \
0 25.0 Driver & Passenger Rear ... 5.0 NaN
1 22.0 None Rear ... 5.0 200.0
2 25.0 Driver & Passenger Front ... 5.0 204.0
3 25.0 Driver only Rear ... 2.0 179.0
4 26.0 Driver & Passenger None ... 6.0 193.0
Wheelbase Width Turn.circle Rear.seat.room Luggage.room Weight \
0 110.0 69.0 37.0 NaN 15.0 3525.0
1 113.0 72.0 42.0 29.0 15.0 4000.0
2 111.0 74.0 44.0 31.0 NaN 3935.0
3 96.0 74.0 43.0 NaN NaN 3380.0
4 106.0 NaN 37.0 31.0 17.0 3405.0
Origin Make
0 non-USA Mercedes-Benz 300E
1 non-USA Infiniti Q45
2 USA Cadillac Seville
3 None Chevrolet Corvette
4 non-USA Audi 100 import datatable as dt
df = dt.fread('https://raw.githubusercontent.com/selva86/datasets/master/Cars93_miss.csv')
# Solution
df[::-1,:, dt.sort(dt.f.Price)].head()22. How to repeat(append) the same data in datatable Frame?
Difficulty Level: L1
Question: Repeat(append) the same data 5 times in datatable Frame.
Input: Cars93Show Solution
23. How to replace string with another string in entire datatable Frame?
Difficulty Level: L1
Question: Replace Audi with My Dream Car in entire datatable Frame.
Input: Cars93Show Solution
# Input
import datatable as dt
df = dt.fread('https://raw.githubusercontent.com/selva86/datasets/master/Cars93_miss.csv')
# Solution
dt.repeat(df, 5)24. How to extract the details of a particular cell with given criterion??
Difficulty Level: L1
Question: Extract which manufacturer, model and type has the highest Price.
Input: Cars93
Desired Output:
Manufacturer Model Type
Mercedes-Benz 300E MidsizeShow Solution
25. How to rename a specific columns in a dataframe?
Difficulty Level: L2
Question: Rename the column Model as Car Model.
Input: Cars93Show Solution
# Input
import datatable as dt
df = dt.fread('https://raw.githubusercontent.com/selva86/datasets/master/Cars93_miss.csv')
# Solution
old_col_name = "Model"
new_col_name = "Car Model"
df.names = [new_col_name if x == old_col_name else x for x in df.names]
df.head(5)26. How to count NA values in every column of a datatable Frame?
Difficulty Level: L1
Question: Count NA values in every column of a datatable Frame.
Input: Cars93
Desired Output:
Manufacturer Model Type Min.Price Price Max.Price MPG.city \
0 4 1 3 7 2 5 9
MPG.highway AirBags DriveTrain ... Passengers Length Wheelbase \
0 2 6 7 ... 2 4 1
Width Turn.circle Rear.seat.room Luggage.room Weight Origin Make
0 6 5 4 19 7 5 3Show Solution
27. How to get a specific column from a datatable Frame as a datatable Frame instead of a series?
Difficulty Level: L1
Question :Get the column (Model) in datatable Frame as a datatable Frame (rather than as a Series).
# Input
import datatable as dt
df = dt.fread('https://raw.githubusercontent.com/selva86/datasets/master/Cars93_miss.csv')
# Solution
df[:,"Model"].head(5)28. How to reverse the order of columns of a datatable Frame?
Difficulty Level: L1
Question : Reverse the order of columns in Cars93 datatable Frame.
Input: Cars93Show Solution
# Input
import datatable as dt
df = dt.fread('https://raw.githubusercontent.com/selva86/datasets/master/Cars93_miss.csv')
# Solution 1
df.head()
df[:,::-1].head(5)29. How to format or suppress scientific notations in Python datatable Frame?
Difficulty Level: L2
Question: Suppress scientific notations like ‘e-03’ in df and print upto 6 numbers after decimal.
Input
import datatable as dt
df = dt.Frame(random=np.random.random(4)**10)
df
# random
0 3.518290e-04
1 5.104371e-02
2 5.895886e-06
3 1.274671e-09Desired Output
random random2
0 3.518290e-04 0.000352
1 5.104371e-02 0.051044
2 5.895886e-06 0.000006
3 1.274671e-09 0.000000# Solution
import datatable as dt
df = dt.Frame(random=np.random.random(4)**10)
df[:,"random2"] = dt.Frame(['%.6f' % x for x in df[:,"random"].to_list()[0]])
df30. How to filter every nth row in a pydatatable?
Difficulty Level: L1
Question: From df, filter the 'Manufacturer', 'Model' and 'Type' for every 20th row starting from 1st (row 0).
Input: Cars93Show Solution
# Input
import datatable as dt
df = dt.fread('https://raw.githubusercontent.com/selva86/datasets/master/Cars93_miss.csv')
# Solution
df[::20, ['Manufacturer', 'Model', 'Type']]31. How to reverse the rows of a python datatable Frame?
Difficulty Level: L2
Question: Reverse all the rows.
Input: Cars93Show Solution
# Input
import datatable as dt
df = dt.fread('https://raw.githubusercontent.com/selva86/datasets/master/Cars93_miss.csv')
# Solution
df[::-1,:]32. How to find out which column contains the highest number of row-wise maximum values?
Difficulty Level: L2
Question: What is the column name with the highest number of row-wise maximum’s.
Input: BostonHousing dataset
Desired Output:
taxShow Solution
# Input
import datatable as dt
df = dt.fread("https://raw.githubusercontent.com/selva86/datasets/master/BostonHousing.csv")
# Solution
for i in range(len(df.names)):
if df.sum()[0:1,:].to_list()[i] == max(df.sum()[0:1,:].to_list()):
print(df.names[i])33. How to normalize all columns in a dataframe?
Difficulty Level: L2
Questions:
Normalize all columns of
dfby subtracting the column mean and divide by standard deviation.Range all columns of
dfsuch that the minimum value in each column is 0 and max is 1.
Don’t use external packages like sklearn.
Input: BostonHousing dataset
Desired Output:
crim zn indus chas nox rm age dis \
0 0.000000 0.18 0.067815 0.0 0.314815 0.577505 0.641607 0.269203
1 0.000236 0.00 0.242302 0.0 0.172840 0.547998 0.782698 0.348962
2 0.000236 0.00 0.242302 0.0 0.172840 0.694386 0.599382 0.348962
3 0.000293 0.00 0.063050 0.0 0.150206 0.658555 0.441813 0.448545
4 0.000705 0.00 0.063050 0.0 0.150206 0.687105 0.528321 0.448545
rad tax ptratio b lstat medv
0 0.000000 0.208015 0.287234 1.000000 0.089680 0.422222
1 0.043478 0.104962 0.553191 1.000000 0.204470 0.368889
2 0.043478 0.104962 0.553191 0.989737 0.063466 0.660000
3 0.086957 0.066794 0.648936 0.994276 0.033389 0.631111
4 0.086957 0.066794 0.648936 1.000000 0.099338 0.693333# Input
import datatable as dt
df = dt.fread("BostonHousing.csv")
# Solution
for i in df.names:
df[:,i] = df[:,(dt.f[i] - df[:,dt.min(dt.f[i])][0,0])/(df[:,dt.max(dt.f[i])][0,0] - df[:,dt.min(dt.f[i])][0,0])]
df.head(5)34. How to compute grouped mean on datatable Frame and keep the grouped column as another column?
Difficulty Level: L1
Question: In df, Compute the mean price of every fruit, while keeping the fruit as another column instead of an index.
Input
df = dt.Frame(fruit = ['apple', 'banana', 'orange'] * 3,
rating = np.random.rand(9),
price = np.random.randint(0, 15, 9))Desired Output:
fruit C0
0 apple 7.666667
1 banana 5.000000
2 orange 8.333333Show Solution
35. How to join two datatable Frames by 2 columns?
Difficulty Level: L2
Question: Join dataframes df1 and df2 by ‘A’ and ‘B’.
Input
df1 = dt.Frame(A=[1, 2, 3, 4],
B=["a", "b", "c", "d"],
D=[1, 2, 3, 4])
df2 = dt.Frame(A=[1, 2, 4, 5],
B=["a", "b", "d", "e"],
C=["a2", "b2", "d2", "e2"])Desired Output:
A B D C
0 1 a 1 a2
1 2 b 2 b2
2 3 c 3
3 4 d 4 d2Show Solution
36. How to create leads (column shifted up by 1 row) of a column in a datatable Frame?
Difficulty Level: L2
Question: Create new column in df, which is a lead1 (shift column A up by 1 row).
Input:
df = dt.Frame(A=[1,2,3,4],B=["a", "b", "c", "d"],d=[1,2,3,4])Desired Output:
A B d A.1
0 1 a 1 2
1 2 b 2 3
2 3 c 3 4
3 4 d 4 NaN# Input
import datatable as dt
df1 = dt.Frame(A=[1, 2, 3, 4], B=["a", "b", "c", "d"], D=[1, 2, 3, 4])
df2 = dt.Frame(A=[1, 2, 4, 5], B=["a", "b", "d", "e"], C=["a2", "b2", "d2", "e2"])
# Solution
df2.key = ["A","B"]
output = df1[:, :, dt.join(df2)]
outputMachine Learning Exercise
36. How to use FTRL Model to calculate the probability of a person having diabetes?
Difficulty Level: L3
Question 1: Use Follow the Regularized Leader (Ftrl) Model to calculate the probability of a person having diabetes.
Question 2: Find the feature importance of the features used in model.
Input:
Training Data : pima_indian_diabetes_training_data.csv
Testing Data : pima_indian_diabetes_testing_data.csvShow Solution
import datatable as dt
from datatable.models import Ftrl
# Import data
train_df = dt.fread('pima_indian_diabetes_training_data.csv')
test_df = dt.fread('pima_indian_diabetes_testing_data.csv')
# Create Ftrl model
ftrl_model = Ftrl()
# add parameter values while creating model
ftrl_model = Ftrl(alpha = 0.1, lambda1 = 0.5, lambda2 = 0.6)
# change paramter of existing model
ftrl_model.alpha = 0.1
ftrl_model.lambda1 = 0.5
ftrl_model.lambda2 = 0.6
# Prepare training and test dataset
train_df[:,"diabetes"] = dt.Frame(np.where(train_df[:, dt.f["diabetes"] == "pos"], 1,0))
test_df[:,"diabetes"] = dt.Frame(np.where(test_df[:, dt.f["diabetes"] == "pos"], 1,0))
x_train = train_df[:, ["pregnant", "glucose", "pressure", "mass", "pedigree", "age"]]
y_train = train_df[:, ["diabetes"]]
x_test = test_df[:, ["pregnant", "glucose", "pressure", "mass", "pedigree", "age"]]
y_test = test_df[:, ["diabetes"]]
# training the model
ftrl_model.fit(x_train,y_train)
# predictions of the model
targets = ftrl_model.predict(x_test)
print(targets.head(5))
# feature importance
fi = ftrl_model.feature_importances
fiAuthor: Ajay Kumar
Reference : https://www.machinelearningplus.com/data-manipulation/101-python-datatable-exercises-pydatatable/
Last updated
Was this helpful?