# Tutorial, Project Ideas, and Tips
## Intermediate Python Refresher: Tutorial, Project Ideas, and Tips
[May 12th 2020](https://hackernoon.com/archives/2020/05/12)
### This article is to teach Python beginners and developers some key concepts used in Python that aren't taught from the get-go.
If you can create a quadratics root solver, you'll be able to understand this article. These are some concepts I didn't learn in one day but rather a couple years as I am self-taught.I tried to make this into a video, but the video ended up being 1:45\
hours long; I myself would need a lot of motivation to watch a\
“tutorial” that long, and I prefer articles to get my information\
since I can pick out relevant details rather than trying to skip\
sections of a video.
### How It All Started
I learned Python basics through [CS Circles,](https://cscircles.cemc.uwaterloo.ca/?ref=hackernoon.com) and then proceeded to improve/test my problem solving skills. I did this by doing [CCC ](https://cemc.uwaterloo.ca/contests/computing.html?ref=hackernoon.com)questions which you can find (among other contest problems) at [DMOJ](https://dmoj.ca/?ref=hackernoon.com). Other sites to improve your algorithmic problem solving skills include [HackerRank](https://hackerrank.com/?ref=hackernoon.com) and [LeetCode](https://leetcode.com/?ref=hackernoon.com) (although this is mainly for interview preparations).While I was doing this, I was coding with the default IDLE editor! I did this for 3 months and then I found out about [PyCharm ](https://www.jetbrains.com/pycharm/download/?ref=hackernoon.com)which\
has a slight learning curve but is loads better in terms of features\
and at improving productivity. Nowadays, I use both PyCharm and [Visual Studio Code](https://code.visualstudio.com/?ref=hackernoon.com).I personally have an entire folder dedicated to snippets of code I could\
use in the future, so I suggest you also do that and maybe even add some\
of the code snippets in this article in there so that you can read your\
own examples instead of Googling them or coming back to this article.
### General Tips
These are some tips that are not bound to programming but just life and productivity in general.**Know your keyboard shortcuts**\
Know both the program specific ones (browser, explorer, IDE of choice, etc.) and also OS specific ones (e.g. Win + R for run).\
\
**Know the command line**Instead of doing a calculation by hand or opening an IDE to create and run a script, you can actually execute Python code from the command line.\
Aside from the common batch functions (e.g. ls, cd), knowing how to use\
Python from the command line will save you a lot of time**Know how to Google**Google (or your search engine of choice), is my best friend and should also be yours. It has saved me a lot of time and so it could also save you a\
lot of time. It can’t do that if you don’t use it or don’t know how to\
use it. When you Google something, your query needs to be general enough that you can find answers, but also specific enough so that those\
answers are relevant.**Problem Breakdown Strategy**This goes hand in hand with Googling. Suppose you have a problem/project. You need to break it down into smaller parts. You then need to analyze each of these parts and see if they are small enough for you to complete each of them. If not, either your missing some knowledge that you should Google or the part is too big and needs to be broken down again. You keep doing this recursive procedure until your project has been\
split into solvable parts so that you can complete them and then weave\
together a project. When I search and find answers through Google, I\
don’t expect them to be 100% what I need. I usually need to *remix*\
them into what I want and that’s what you should also expect: the bare\
minimum solution that takes you at least one step forward.With these tips stated, you can do a couple of different things next. You\
can skim the rest of the document and make notes on the snippets of code\
I feature (what I would do personally), read only the headings, skip to\
the project ideas section, or stop reading altogether as my tips are so\
useful.
### Refresher
In CS Circles, they bring abou\`t the print function and some of its\
optional parameters but it’s easy to forget about them so here it is\
again.
```python
>>> # The default parameters for print are sep=' ', and end='\n'
>>> print('4', '21', 2020, sep='/', end='\n---------\n')
4/21/2020
---------
```
### Let’s Start
**Input function and String Formatting**The input function has an optional parameter so that it can also act as a\
prompt and if you are using Python 3.6+, you can make use of f-strings.
```python
name = input('Enter your name: ')
print(f'Hello {name}!') # modern way of string formatting
# if input='reader', output: Hello reader!
```
**For Loops**\
\
I want to make clear to you that a for loop, is not a while loop as it is\
in other languages. In Python, a for loop is an iteration over an iterable object.The range function has three parameters, two of them being optional. Do not write use the range function with an explicit start value of 0 because 0 is the default start value, (unless of course you are modifying the default step value of 1).In this example, I will show you exactly what I mean by “not a while loop” and how a for loop (specifically range) does not add to the temporary value.
```python
# range(start=0, stop, step=1)
# range(5) == range(0, 5) == range(0, 5, 1)
for i in range(5):
print(i)
i += 2
# Guess the output. HINT: i += 2does nothing
```
If you run this code, you’ll notice that the output is increasing by 1\
each time even if we are adding 2 to i at every loop. This is because *i* is\
set to the next value in range and isn’t actually being increased by\
one each time. This means that we can actually iterate over all sorts of\
iterable objects, like lists, without having to use range and indexing.
```python
some_letters = ['a', 'b', 'c', 'd', 'e']
for letter in some_letters:
# do something
pass
```
Here I introduced the keyword pass, this is to avoid errors in otherwise empty blocks.If you do want to keep track of the index as well as the item, you still\
don’t have to use range, you can use the built-in function enumerate.
```python
for i, letter in enumerate(some_letters, start=0):
print(f'item at index {i} is {letter}')
```
You can think of enumerate as turning an iterable into an iterable of pairs (index, item of iterable at index).You can also use the next function to retrieve the next value in an\
iterable (if there is no next item, an error will be raised).
### File I/O
I’m including this because if you search “how to read files in python” on Google, you are given [this](https://www.guru99.com/reading-and-writing-files-in-python.html?ref=hackernoon.com) which teaches it the old way and not the modern approach.
```python
# make sure test.txt exists with text in it
# OLD
f = open('test.txt') # note default is mode='r'
# do something with f here
f.close()
with open('test.txt') as f: # NEW; no close() needed
print(f.read())
# f.read() moves the "cursor" to the end of the file
assert not f.read()
f.seek(0)
assert f.read()
# f.read() returns a string now (unless test.txt is empty)
with open('test.txt', 'w') as f:
# f.read() ERROR do not do this
f.write('this is a test\n') # note there is no end parameter
f.writelines(['line1\n', 'line2\n']) # note no auto newline# other modes: a for append, rb for reading-bytes, wb for writing bytes, and r+/w+ for both reading and writing at the same time
```
If you are curious why the r+/w+ is not the default, think about how a file\
cannot be opened to be written to if it is being “written” to by\
another program. If you just need to read a file, it being open in\
another program means that you won’t be interfering with the other\
program.
### Error Handling
```python
# handling the error
try:
raise RuntimeWarning('Something could go wrong')
except RuntimeWarning as e: # as e is optional
# handle the exception here
# ignoring the error
# old
try:
raise Exception('BOO')
except Exception: pass
# new
from contextlib import suppress
def ignore_error(exception: Exception):
"""
Use three quotes for docstrings or long strings
"""
# : for type hinting (in a dynamic typed language!) and
# yes you can pass exceptions and functions as parameters
with suppress(exception):
raise exception('BOO')
print('not printed')
ignore_error(RuntimeError)
print('this gets printed')
```
By this point if you are following along in PyCharm, you would have seen some squiggly lines, especially under “Exception” in the above code.\
These squiggly lines help you to avoid syntax errors, follow style\
guidelines, and bring attention to code that could be doing something\
you didn’t want it to be doing.
### More Data Types
So what are these iterables I keep mentioning? Yes a list is an iterable and so are tuples (which you should already know of).There are also dictionaries, sets and generators (not discussed here).\
Dictionaries are like hash tables in other languages, because they\
“hash” the key to store information.
```python
empty_dict = {} # or dict()
my_dict = {'key': 'value'}
# How to get value from dict
my_dict['a'] # raises KeyError if 'a' not in dictionary
my_dict.get('a', DEFAULT_VALUE)
if 'key' in my_dict:
val = my_dict['key']
val = my_dict.get('key', None)
if val is not None: pass
with suppress(KeyError):
val = my_dict['key']
# iterations
for k in my_dict: pass # or for k in my_dict.keys()
for v in my_dict.values(): pass
for k, v in my_dict.items():
# since items() generates the items as the iteration happens,
# my_dict cannot be modified in this loop.
# For modification use tuple(my_dict.items())
pass
# remove key from dict
del my_dict['key'] # can raise KeyError
# if you want to use the value, use .pop() and define a default
# value to avoid KeyErrorsmy_dict.pop('key', DEFAULT_VALUE)
# set
empty_set = set() # {} would initialize an empty dict
my_set = {1, 2, 3}
if 1 in set: pass
# there are many set methods, go check them out yourself
# some include: union, intersect, difference
# you can use + and - as well
```
**Data Structure Usage (Efficiency)**\
\
The data structure you use is very important to writing good code.use dictionaries if order doesn’t matter + each key has information (value) associated with ituse sets if order doesn’t matter + no values per key (e.g. keeping track of what you have ‘used’ per se)use tuples if you need ordered data but don’t need to modify the data (e.g. coordinates)use lists if you need order and mutability (most flexible)You can’t use sets or dictionaries or sets if you need to keep track of\
duplicates. That’s because sets and dictionaries hash the keys so that\
it is super fast (O(1)) to check if a key is in a dictionary. This does\
mean that you can’t use lists, sets, and generators as keys (but you can definitely use tuples as long as lists are not nested).Dictionaries are also like JSON objects so you can actually use the json module to export them to a JSON file. Note that if you’re using sets as values, they are converted to lists in an exported json file.**Miscellaneous Functions**Sometimes you will see functions like
```python
func(*args, **kwargs)
```
```python
# args = a list of arguments
# kwargs = keyword arguments
# (in the function it'll be a dictionary)
# *args: list in the function **kwargs: dict in the function
def complex_func(*args, **kwargs):
pass
def normal_func(a, b, c, sample_param=5):
pass
sample_args = {'sample_param': 3}
args = [0, 1, 2]
complex_func(1, 2, 3, test='true') # how you'd call it
complex_func(*args, **sample_args) # also works on normal functions
normal_func(*args, **sample_args)
```
### List Comprehension and Ternary
One of the most beautiful parts of Python is list comprehensions; one liners to create lists.
```python
# example: input is space separated integers
integers = [int(x) for x in input.split()]
# split(sep=' ', maxsplit=-1), -1 means no limit
no_negatives = [x for x in integers if x > 0] # only if
positives = [x if x > 0 else -x for x in integers] # if and else
back_to_str = ' '.join((str(x) for x in integers))
# items in the list to join need to be of type str
print(integers)
# this next case demonstrates the ternary operator _ if _ else _
print('list is', 'not empty' if integers else 'empty')
```
You can also use list comprehensions to create dictionaries and sets
```python
set_example = {x for x in range(3)}
dict_example = {x: x for x in range(3)}
generator_example = (x for x in range(3)) # use sparingly
```
The third example is a generator. There are some use cases for it, so do your research before using them as they are an advanced topic not for this article.
### Iterables vs. Primitives
There is one very important distinction between primitive variables and iterable variables. For example.
```python
a = 5
b = a
a = 6
print(a == b) # false
# vs.
a = [1, 2, 3]
b = a
c = [1, 2, 4]
a[2] = 4
print(a == b == c) # true
print(a is b) # true; same refrence
print(a is c) # false
```
This is especially important when dealing with nested iterables with how you create nested iterables and also *copy* them. Try out these examples yourself.
```python
lols = [[0] for i in range(3)] # [0] is created 3 times
lols[0][0] = 5
print(lols) # [[5], [0], [0]]
# vs.
a = [[0]]
lols = a * 3 # same as lols = [[0] * 3]
lols[0][0] = 5
print(lols) # [[5], [5], [5]]
```
**Copying Iterables**To make a shallow copy, use .copy(). BUT, note that for any nested iterables, only the reference is copied, not the actual nested list. That’s why it’s called a shallow copy. To *deepcopy*, we can use the copy module.
```python
new_copy = lols.copy() # I prefer this over using [:]
reversed_list = lols[::-1]
# I rarely use this^ as reversed() and .reverse() exist
new_copy[0][0] = 6 # lols == [[6], [6], [6]]
assert lols == new_copy and not lols is new_copy
from copy import deepcopy
new_copy = deepcopy(list_of_lists)
new_copy = list_of_lists
new_copy[0][0] = 4 # [[4], [4], [4]] because 3x of the same list
assert lols != new_copy and lols is not new_copy
```
### Memoization
Memoization is the caching of function return results in order to speed up repetitive calculations. An example would be the recursive implementation of the Fibonacci sequence.
```python
def memo(func): # remove print statements in a practical setting
cache = {}
def _helper(x): # constrained to one param (in this case)
# you could have multiple params (x, y, ...) and then
# cache using tuple keys
if x not in cache:
print('not in cache')
cache[x] = func(x)
else:
print('in cache')
return cache[x]
return _helper
@memo # square = memo(square) <-- what it means
def square(x):
return x * x
for i in range(3):
square(i), square(i) # second one uses cached result
```
An exercise is to make an addition function (a, b) that uses memoization.
### Lambdas
Usually used in place of a function parameter if the calculation is short. For example, sorting.
```python
['aa', 'Bb', 'Cc', 'dD'].sort(key=lambda string: string.upper())
[('a', 1), ('b', 0)].sort(key=lambda pair: pair[1])
sorted([('a', 1), ('b', 0)], key=lambda pair: pair[1])
max([('a', 1), ('b', 0)], key=lambda pair: pair[1]) # and min
```
### Modules
Modules play a big part in projects you will do. Some built-in ones are os, shutil, copy, glob, and threading.**os**
```python
import os
os.mkdir() # to make a NEW dir
os.chdir() # choose a current working dir
os.getcwd() # get current working dir
os.path.exists()
os.rename()
os.remove() # for existing files only
os.rmdir()
os.getenv('key') # gets an environmental variable
# use the shutil module for directories with sub directoriese
```
**Environmental variables**Specify project secrets in a
```python
.env
```
file
```python
# in .env
KEY=VALUE
# in *.py
with open('.env') as f:
for line in f.read().splitlines():
k, v = line.split('=', 1)
os.environ[k] = v
```
**glob**Used for getting a list of files/folders
```python
from glob import glob
print(glob('*.py')) # get all .py files in cwd, * is a wildcard
# exercise: find out how to get all .py files in cwd + its subdirs
```
**Threading**I have already written some threading example for you to look at [here](https://gist.github.com/elibroftw/ddce68086e4dfad4df053a7f995905fc?ref=hackernoon.com). Just follow the instructions in the gist.
### Third Party Modules
You need to use \`pip install module\_name\` to install modules.Some modules that are common are requests, beautifulsoup4, PIL, and flask. If you’re working on a big project, you’ll probably end up using 3rd party modules.
### Advanced Topics (future Python learning)
**Classes**I did not cover classes because that is more about OOP than Python programming and the use cases for classes are very small. One thing you should know when you are learning classes is *\_\_slots\_\_* property, so do search that up on your own.**Generators**Again this is an advanced topic and learning about it now will only lead to confusion, its best to learn this on your own or in a practical setting.**Decorators**I covered only the basics of decorators. There are decorators used by lots of other 3rd party libraries and different use cases (e.g. timing functions) so I suggest you do your own research on them as well.**git and git workflow**This is very important when your collaborating with others or are working for a company. Git is a versioning tool used so that mistakes don’t hurt you, and for letting you work on multiple features at the same time.**other builtin modules**Such as itertools, threading, multiprocessing, and more.
### Project Ideas
You can find a list of project ideas below:
```
PRACTICAL PROJECT IDEAS
Your own website/portfolio (no bootstrap, use raw HTML/CSS/JS + flask backend). Use Heroku (free) to host it [Hard]
A program that applies a blur to an image you select [Easy]
(use tkinter/SG so that you don't have to hard code the filename every time someone uses the program)
A program that sets your desktop wallpaper and cycles through a folder of images, [Okay]
change the wallpaper every x (you decide) seconds.
Web Scraping & Data Parsing [Hard]
Go to https://www.cia.gov/library/publications/the-world-factbook/rankorder/2004rank.html and use inspect elements,
Your job is to parse this web page and filter the data so that only the data with the latest "date of information"
remains. Output should be order-preserved and in a CSV file (filename is arbitrary)
To make it easier (not recommended), you can assume the latest date of info is 2017 EST.
To make it even easier (not recommended) just preserve the order and output all of the data in a CSV
hint: use requests, bs4, and the csv module
Another challenge is to download the data as a txt file and convert that text file to a CSV file using Python
hint: you may find ' '.join(iterable) useful
Using an API [Okay - Hard]
Spotify Reference
You'll have to get a Spotify API key
See https://developer.spotify.com/documentation/web-api/reference/search/search/ and
make a program that lets users search for tracks using the Spotify API (you return a list of tracks)
See https://elijahlopez.herokuapp.com/search-album-art/ for a demo but with album art
You just need to spit out a list of tracks
To make this harder, you will sort the tracks by popularity (you'll have to search the documentation yourself)
To make this even harder, you can add a sort by artists feature and MORE
# requests, base64 module,
Try to send an email to yourself (your email -> your email) using Python [Easy - Hard]
To make this harder, try to also send an attachment
module: at least smptlib
You can try to make your own key logger [Okay - Hard]
At first just append the contents to a .log or .txt file
Use the 'a' open type to append to bottom.
HARDER: YOU WILL INCLUDE COPIED AND PASTED ITEMS WITH custom syntax
# (e.g. USER_COPIED("")), (e.g. USER_PASTED(""))
Make your own soundboard with your own sounds [Hard because tedious]
You PySimpleGUI for the GUI (I recommend but you don't have to)
- Let your soundboard include support for different "packs"
- Specify a design/package guide for other users to create use their sounds on your soundboard
default: your default pack
PACKS_DIR/pack_name/sound_name/sound .mp3/ogg
# pygame, playsound
Make your own game using pygame (OOP)
Turn one of these projects into an executable [Okay but Tedious]
- use sys.exit() instead of quit()
pyinstaller, or cx_freeze
Make your own discord bot [variable difficulty] (async)
Email me or comment for bigger programming (not specifically Python) project ideas
```
Comment if you have any questions.Thanks for reading and good luck to your learning journey.
Reference :
---
# Agent Instructions: Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter:
```
GET https://yo-sarawut.gitbook.io/tutorials/articles/python-1/tutorial-project-ideas-and-tips.md?ask=
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.