Work with CSV files, read and write like a boss.

Example 1: Read a CSV file.

df = pd.read_csv('file.csv')

Example 2: Write a DataFrame to CSV.

df.to_csv('output.csv', index=False)

Handle CSVs like a data wizard.

Pandas JSON

Handle JSON data with ease.

Example 1: Read JSON.

df = pd.read_json('file.json')

Example 2: Write JSON.

df.to_json('output.json', orient='records')

JSON is your gateway to modern data formats.

Pandas Cleaning Data

Clean data faster than your roommate avoids doing dishes.

Example 1: Strip whitespace.

df['Column'] = df['Column'].str.strip()

Example 2: Standardize case.

df['Column'] = df['Column'].str.lower()

Clean data, clean mind.

Pandas Missing Values

Handle missing values like a pro.

Example 1: Fill missing values.

df = pd.DataFrame({'A': [1, None, 3]})
df.fillna(0, inplace=True)

Example 2: Drop rows with missing data.

df.dropna(inplace=True)

Because no one likes a blank space (thanks, Taylor Swift).

Pandas Handling Wrong Format

Fix your messy data formats.

Example 1: Convert strings to dates.

df['Date'] = pd.to_datetime(df['Date'])

Example 2: Convert strings to numbers.

df['Numeric'] = pd.to_numeric(df['Numeric'], errors='coerce')

Wrong format? Not anymore.

Pandas Handling Wrong Data

Replace invalid values.

Example: Replace placeholders with None.

df['Column'] = df['Column'].replace('?', None)

No more wrong data ruining your day.

Pandas Get Dummies

Convert categories to indicator variables.

Example: Create dummy variables.

dummies = pd.get_dummies(df['Category'])
print(dummies)

Perfect for machine learning models.

Pandas Categorical

Optimize memory usage with categorical data types.

Example: Convert to categorical.

df['Category'] = df['Category'].astype('category')

Save memory, save the planet.

Pandas Datetime

Work with datetime like a pro.

Example 1: Extract the year.

df['Year'] = pd.to_datetime(df['Date']).dt.year

Example 2: Extract the month.

df['Month'] = pd.to_datetime(df['Date']).dt.month

DateTime makes time manipulation a breeze.

Pandas Aggregate Functions

Summarize data efficiently.

Example: Group by and aggregate.

grouped = df.groupby('Category').agg({'Value': 'sum'})
print(grouped)

Aggregations are the secret sauce to insights.

Pandas Group By

Group data for analysis.

Example: Group and calculate mean.

print(df.groupby('Category').mean())

Group by like a data Jedi.

Pandas Filtering

Filter rows based on conditions.

Example: Filter rows.

df[df['Value'] > 10]

Keep what you love, ditch the rest.

Pandas Sort

Sort data like a pro.

Example 1: Sort rows by a column.

df.sort_values(by='Value', ascending=False, inplace=True)

Example 2: Sort by index.

df.sort_index(inplace=True)

Sorting is as easy as pie.

Pandas Correlation

Check correlations between numerical columns.

Example: Compute correlation.

print(df.corr())

Correlation = your shortcut to patterns.

Pandas Plot

Visualize data without leaving Pandas.

Example: Line plot.

df.plot(kind='line')

Example: Bar plot.

df.plot(kind='bar')

Pandas plotting is your gateway to quick insights.

Pandas Histogram

Understand distributions with histograms.

Example 1: Simple histogram.

df['Value'].hist()

Example 2: Multiple histograms.

df[['A', 'B']].hist()

Histograms: because data speaks louder than words.

View and analyze your data frames with built-in Pandas methods. Stats made simple!

Example 1: Get a quick statistical summary.

df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})
print(df.describe())

# Output:
#          A    B
# count  3.0  3.0
# mean   2.0  5.0
# std    1.0  1.0
# min    1.0  4.0
# 25%    1.5  4.5
# 50%    2.0  5.0
# 75%    2.5  5.5
# max    3.0  6.0

Example 2: Calculate column-wise means.

df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})
print(df.mean())

# Output:
# A    2.0
# B    5.0
# dtype: float64

Example 3: Return the first n rows in a data frame.

data = {
    "Name": ["John", "Alice", "Bob", "Emma", "Mike"],
    "Age": [25, 30, 35, 28, 32],
    "City": ["New York", "Paris", "London", "Sydney", "Tokyo"],
}
df = pd.DataFrame(data)
print(df.head(3))

# Output:
#     Name  Age      City
# 0   John   25  New York
# 1  Alice   30     Paris
# 2    Bob   35    London

Example 4: Return the last n rows in a data frame.

data = {
    "Name": ["John", "Alice", "Bob", "Emma", "Mike"],
    "Age": [25, 30, 35, 28, 32],
    "City": ["New York", "Paris", "London", "Sydney", "Tokyo"],
}
df = pd.DataFrame(data)
print(df.tail(3))

# Output:
#    Name  Age    City
# 2   Bob   35  London
# 3  Emma   28  Sydney
# 4  Mike   32   Tokyo

Example 5: Get the data frame information.

data = {
    "Name": ["John", "Alice", "Bob", "Emma", "Mike"],
    "Age": [25, 30, 35, 28, 32],
    "City": ["New York", "Paris", "London", "Sydney", "Tokyo"],
}
df = pd.DataFrame(data)
print(df.info())

# Output:
# <class 'pandas.core.frame.DataFrame'>
# RangeIndex: 5 entries, 0 to 4
# Data columns (total 3 columns):
#  #   Column  Non-Null Count  Dtype
# ---  ------  --------------  -----
#  0   Name    5 non-null      object
#  1   Age     5 non-null      int64
#  2   City    5 non-null      object
# dtypes: int64(1), object(2)
# memory usage: 248.0+ bytes
# None

Pandas DataFrame Manipulation

Add, update, or drop columns and rows with ease.

Example 1: Add and drop columns.

data = {
    "Name": ["John", "Alice", "Bob", "Emma", "Mike"],
    "Age": [25, 30, 35, 28, 32],
    "City": ["New York", "Paris", "London", "Sydney", "Tokyo"],
}
df = pd.DataFrame(data)

df["Country"] = ["United States", "France", "United Kingdom", "Australia", "Japan"]
print(df.head(2))

print("================")

df.drop("Age", axis=1, inplace=True)
# To drop multiple: df.drop(["Age", "City"], axis=1, inplace=True)
# Using columns: df.drop(columns="Age", inplace=True)
print(df.head(2))

#     Name  Age      City        Country
# 0   John   25  New York  United States
# 1  Alice   30     Paris         France
# ================
#     Name      City        Country
# 0   John  New York  United States
# 1  Alice     Paris         France

Example 2: Insert and drop rows.

data = {
    "Name": ["John", "Alice", "Bob", "Emma", "Mike"],
    "Age": [25, 30, 35, 28, 32],
    "City": ["New York", "Paris", "London", "Sydney", "Tokyo"],
}
df = pd.DataFrame(data)

df.loc[len(df.index)] = ["Drake", 32, "Bangkok"]
# To replace: df.loc[2] = ["Drake", 32, "Bangkok"]
print(df.tail(2))

print("================")

df.drop(1, axis=0, inplace=True)
# To drop multiple: df.drop([1, 3], axis=0, inplace=True)
# Using index: df.drop(index=1, inplace=True)
print(df.head(2))

#     Name  Age     City
# 4   Mike   32    Tokyo
# 5  Drake   32  Bangkok
# ================
#    Name  Age      City
# 0  John   25  New York
# 2   Bob   35    London

Example 3: Rename column names and indexes.

data = {
    "Name": ["John", "Alice", "Bob", "Emma", "Mike"],
    "Age": [25, 30, 35, 28, 32],
    "City": ["New York", "Paris", "London", "Sydney", "Tokyo"],
}
df = pd.DataFrame(data)

df.rename(columns={"City": "Address"}, inplace=True)
# Using mapper: df.rename(mapper={"City": "Address"}, axis=1, inplace=True)
print(df.head(2))

print("================")

df.rename(index={1: 100}, inplace=True)
# Using mapper: df.rename(mapper={1: 100}, axis=0, inplace=True)
print(df.head(2))

#     Name  Age   Address
# 0   John   25  New York
# 1  Alice   30     Paris
# ================
#       Name  Age   Address
# 0     John   25  New York
# 100  Alice   30     Paris

Think of it like playing Tetris but with data.

Pandas Indexing and Slicing

Access data with labels or positions—no guessing required.

Example 1: Indexing by column or row.

df = pd.DataFrame({'A': [10, 20], 'B': [30, 40]})
print(df['A'])  # Access column
print(df.iloc[0])  # Access first row by position
print(df.loc[0])  # Access first row by label

Example 2: Slice rows and columns.

print(df.iloc[:1])  # First row
print(df[['A', 'B']])  # Selected columns

Indexing in Pandas is like peeling an onion: layer by layer.

Pandas Select

Filter specific rows or data based on conditions.

Example 1: Simple condition.

print(df[df['A'] > 15])

Example 2: Multiple conditions.

print(df[(df['A'] > 10) & (df['B'] < 40)])

It’s like swiping right on the rows you love.

Pandas Multiindex

For when a single index just isn’t enough.

Example: Create and use a MultiIndex.

arrays = [['A', 'A', 'B'], [1, 2, 1]]
index = pd.MultiIndex.from_arrays(arrays, names=('Letter', 'Number'))
df = pd.DataFrame({'Value': [10, 20, 30]}, index=index)
print(df)

MultiIndex is for the overachievers. You know who you are.

Pandas Reshape

Reshape your data with melt and pivot.

Example 1: Use melt for long format.

df = pd.DataFrame({'ID': [1, 2], 'Value': [10, 20]})
melted = pd.melt(df, id_vars='ID')
print(melted

Example 2: Use pivot for wide format.

pivoted = melted.pivot(index='ID', columns='variable', values='value')
print(pivoted)

Shape your data like a pro.

Pandas Duplicate Values

Find and remove duplicate rows. Because no one likes redundancy.

Example 1: Detect duplicates.

df = pd.DataFrame({'A': [1, 1, 2], 'B': [3, 3, 4]})
print(df.duplicated())

Example 2: Drop duplicates.

df.drop_duplicates(inplace=True)
print(df

Duplicates, begone!

Pandas Pivot

Reorganize your data with pivot.

Example: Pivot data.

data = {'Date': ['2024-01-01', '2024-01-02'], 'Value': [10, 20]}
df = pd.DataFrame(data)
pivoted = df.pivot(index='Date', columns='Value')
print(pivoted)

Pivots make data feel fancy.

Pandas Pivot Table

Summarize data with a pivot table.

Example: Create a pivot table.

data = {'Category': ['A', 'A', 'B'], 'Value': [10, 20, 30]}
df = pd.DataFrame(data)
pivot_table = df.pivot_table(values='Value', index='Category', aggfunc='sum')
print(pivot_table)

You’re basically at Excel-level now.

Pandas is a powerful Python library for data manipulation and analysis. Pandas is your gateway to managing data effectively. First step: bring it to your project.

It’s like Excel, but smarter and much less likely to crash when you sort something.

import pandas as pd

Pandas Series

A Pandas Series is a one-dimensional labeled array. Think of it as a fancy list with labels (indices). We can access values by their indices.

Example 1: Create a basic Series from the List.

s = pd.Series([10, 20, 30])
print(s)

# Output:
# 0    10
# 1    20
# 2    30
# dtype: int64

Example 2: Assign custom indices to the elements.

s = pd.Series([10, 20, 30], index=["a", "b", "c"])
print(s)

# Output:
# a    10
# b    20
# c    30
# dtype: int64

Example 3: Create a Series from the dictionary.

s = pd.Series({"bmw": 10, "honda": 9, "suzuki": 8})
print(s)

# Output:
# bmw       10
# honda      9
# suzuki     8
# dtype: int64

Example 4: Create a Series by selecting specific keys from the dictionary.

s = pd.Series({'bmw': 10, 'honda': 9, 'suzuki': 8}, index=['bmw', 'honda'])
print(s)

# Output:
# bmw      10
# honda     9
# dtype: int64

Pandas DataFrame

A DataFrame is a 2D-labeled data structure. Think of it as rows and columns. It’s the bread and butter of Pandas.

Example 1: Create a DataFrame from a dictionary.

data = {
    "Name": ["Alice", "Bob"],
    "Age": [25, 30],
}
df = pd.DataFrame(data)
print(df)

# Output:
#     Name  Age
# 0  Alice   25
# 1    Bob   30

Example 2: Create a DataFrame from lists.

data = [[1, "John"], [2, "Bob"]]
df = pd.DataFrame(data, columns=["ID", "Name"])
print(df)

# Output:
#    ID  Name
# 0   1  John
# 1   2   Bob

Pandas Index

The Index is a label array of stored rows and columns. Labels can also be customized.

Example 1: Setting a column as an Index.

data = {
    "Name": ["Alice", "Bob"],
    "Age": [25, 30],
}
df = pd.DataFrame(data)
df.set_index("Name", inplace=True)
print(df)

# Output:
#        Age
# Name
# Alice   25
# Bob     30

Example 2: Setting a range as an Index.

data = {
    "Name": ["Alice", "Bob"],
    "Age": [25, 30],
}
df = pd.DataFrame(data, index=pd.RangeIndex(100, 102, name="Index"))
print(df)

# Output:
#         Name  Age
# Index
# 100    Alice   25
# 101      Bob   30

Example 3: Renaming an Index.

data = {
    "Name": ["Alice", "Bob"],
    "Age": [25, 30],
}
df = pd.DataFrame(data, index=pd.RangeIndex(100, 102, name="Index"))
df.rename(index={101: "First"}, inplace=True)
print(df)

# Output:
#         Name  Age
# Index
# 100    Alice   25
# First    Bob   30

Example 4: Resetting an Index.

data = {
    "Name": ["Alice", "Bob"],
    "Age": [25, 30],
}
df = pd.DataFrame(data)
df.set_index("Name", inplace=True)
df.reset_index(inplace=True)
print(df)

# Output:
#     Name  Age
# 0  Alice   25
# 1    Bob   30

Example 5: Listing Indices.

data = {
    "Name": ["Alice", "Bob"],
    "Age": [25, 30],
}
df = pd.DataFrame(data)
print(df.index)
print(df.index.values)

# Output:
# RangeIndex(start=0, stop=2, step=1)
# [0 1]

Pandas Array

Pandas integrates with NumPy arrays for efficient data storage.

Example 1: Create a pandas array and pandas Series.

arr = pd.array(["John", "Alice", "Bob"])
arr_series = pd.Series(arr)
print(arr_series)

# Output:
# 0     John
# 1    Alice
# 2      Bob
# dtype: string

Example 2: Convert a column to a NumPy array.

data = {
    "Name": ["Alice", "Bob"],
    "Age": [25, 30],
}
df = pd.DataFrame(data)
arr = df["Name"].to_numpy()
print(arr)

# Output:
# ['Alice' 'Bob']

Example 3: Modify data using NumPy.

data = {
    "Name": ["Alice", "Bob"],
    "Age": [25, 30],
}
df = pd.DataFrame(data)
arr = df["Age"].to_numpy()
df["Age"] = arr + 2
print(df)

# Output:
#     Name  Age
# 0  Alice   27
# 1    Bob   32

NumPy is a powerful Python library for numerical computing. It simplifies numerical computations by performing efficient operations on large multidimensional arrays and matrices.

Say goodbye to slow loops and hello to blazing speed!

import numpy as np

NumPy Array Creation

Create arrays using array() or functions such as zeros() and ones(). Think of it as building blocks for your data.

arr = np.array([1, 2, 3])
zeros = np.zeros((2, 2))
ones = np.ones((3, 3))

NumPy N-d Array Creation

Supports the creation of multi-dimensional arrays. Because 2D is cool, but N-D is cooler.

nd_array = np.array([[1, 2], [3, 4]])
higher_dim = np.array([[[1, 2], [3, 4]], [[5, 6], [7, 8]]])

NumPy Data Types

Specify or inspect array data types. Useful when you want to avoid unexpected data-type surprises.

arr = np.array([1.0, 2.0], dtype=np.float32)
print(arr.dtype)  # float32
int_arr = np.array([1, 2, 3], dtype=np.int32)
print(int_arr.dtype)  # int32

NumPy Array Attributes

Inspect the shape, size, and dimensions of arrays. It’s like peeking under the hood of your array.

arr = np.array([[1, 2, 3], [4, 5, 6]])
print(arr.shape, arr.size, arr.ndim)  # (2, 3), 6, 2

NumPy Input Output

Save and load arrays easily. Think of it as bookmarking your progress.

np.save('array.npy', arr)
loaded = np.load('array.npy')
np.savetxt('array.txt', arr, delimiter=',')
loaded_txt = np.loadtxt('array.txt', delimiter=',')

NumPy Array Indexing

Access elements by indices. Remember, NumPy arrays are 0-indexed!

arr = np.array([10, 20, 30])
print(arr[0])  # 10
print(arr[-1])  # 30

NumPy Array Slicing

Extract sub-arrays using slicing. It’s like cutting a slice of your data pizza.

arr = np.array([1, 2, 3, 4, 5])
sub_arr = arr[1:4]  # [2, 3, 4]
every_other = arr[::2]  # [1, 3, 5]

NumPy Array Reshaping

Change the shape without altering data. Rearrange your data like a Rubik’s cube.

arr = np.array([1, 2, 3, 4, 5, 6])
reshaped = arr.reshape((2, 3))  # [[1, 2, 3], [4, 5, 6]]
flattened = reshaped.flatten()  # [1, 2, 3, 4, 5, 6]

NumPy Arithmetic Array Operations

Perform element-wise operations. Because who wants to loop through elements manually?

arr = np.array([1, 2, 3])
result_add = arr + 10  # [11, 12, 13]
result_mul = arr * 2  # [2, 4, 6]

NumPy Array Functions

Built-in functions like sum and mean make life easier. They’re like your data’s best friends.

arr = np.array([1, 2, 3, 4])
print(np.sum(arr))  # 10
print(np.mean(arr))  # 2.5

NumPy Comparison/Logical Operations

Compare arrays element-wise. Great for filtering data with conditions.

arr = np.array([1, 2, 3, 4])
print(arr > 2)  # [False, False, True, True]
print(np.logical_and(arr > 1, arr < 4))  # [False, True, True, False]

NumPy Math Functions

Apply math functions directly. No need for calculators anymore.

arr = np.array([1, 4, 9])
print(np.sqrt(arr))  # [1. 2. 3.]
print(np.power(arr, 2))  # [1 16 81]

NumPy Constants

Use predefined constants like pi. For when you don’t want to remember 3.14159.

print(np.pi)  # 3.141592653589793
print(np.e)  # 2.718281828459045

NumPy Statistical Functions

Compute stats like median, and variance. Perfect for understanding your data’s personality.

arr = np.array([1, 2, 3, 4])
print(np.median(arr))  # 2.5
print(np.var(arr))  # 1.25

NumPy String Functions

Operate on strings in arrays. Because even text data needs some love.

names = np.array(['Alice', 'Bob'])
print(np.char.upper(names))  # ['ALICE' 'BOB']
print(np.char.replace(names, 'o', '0'))  # ['Alice' 'B0b']

NumPy Broadcasting

Perform operations on arrays with different shapes. It’s like magic, but with math.

arr = np.array([1, 2, 3])
broadcasted = arr + np.array([10])  # [11, 12, 13]
expanded = arr + np.array([[10], [20]])  # [[11, 12, 13], [21, 22, 23]]

NumPy Matrix Operations

Matrix multiplication, inversion, etc. Linear algebra geeks, rejoice!

matrix = np.array([[1, 2], [3, 4]])
print(np.dot(matrix, matrix))  # Matrix multiplication
print(np.linalg.inv(matrix))  # Matrix inversion

NumPy Set Operations

Find unique elements and intersections. Useful for deduplication and comparisons.

set1 = np.array([1, 2, 3])
set2 = np.array([2, 3, 4])
print(np.union1d(set1, set2))  # [1 2 3 4]
print(np.setdiff1d(set1, set2))  # [1]

NumPy Vectorization

Efficient operations on entire arrays. Skip the loops and embrace speed.

arr = np.array([1, 2, 3])
vectorized = np.vectorize(lambda x: x ** 2)(arr)  # [1, 4, 9]
vectorized_add = np.vectorize(lambda x: x + 10)(arr)  # [11, 12, 13]

NumPy Boolean Indexing

Filter elements based on conditions. Let your data speak for itself.

arr = np.array([1, 2, 3, 4])
filtered = arr[arr > 2]  # [3, 4]
even = arr[arr % 2 == 0]  # [2, 4]

NumPy Fancy Indexing

Access specific elements with lists/arrays. Fancy indeed!

arr = np.array([10, 20, 30, 40])
print(arr[[0, 2]])  # [10, 30]
print(arr[[1, 3]])  # [20, 40]

NumPy Random

Generate random numbers. Perfect for simulations and shuffling data.

rand_arr = np.random.rand(3, 3)  # Uniform distribution
rand_ints = np.random.randint(0, 10, (2, 2))  # Random integers

NumPy Linear Algebra

Handle linear algebra operations. Your math professor would approve.

matrix = np.array([[1, 2], [3, 4]])
print(np.linalg.det(matrix))  # Determinant
print(np.linalg.eig(matrix))  # Eigenvalues and eigenvectors

NumPy Histogram

Create histograms. Visualize data distribution like a pro.

arr = np.array([1, 1, 2, 3, 3, 3, 4])
hist, bins = np.histogram(arr, bins=3)
print(hist)  # [2 1 4]
print(bins)  # [1. 2. 3. 4.]

NumPy Interpolation

Interpolate data. Filling gaps has never been easier.

x = [0, 1, 2]
y = [0, 1, 4]
print(np.interp(1.5, x, y))  # 2.5
print(np.interp([0.5, 1.5], x, y))  # [0.5, 2.5]

NumPy Files

Read/write text/binary files. Share or save your work effortlessly.

arr = np.array([[1, 2], [3, 4]])
np.savetxt('data.txt', arr)
loaded = np.loadtxt('data.txt')
np.save('data.npy', arr)
loaded_bin = np.load('data.npy')

NumPy Error Handling

Handle errors gracefully. Because no one likes crashing code.

try:
    result = np.sqrt(-1)
except FloatingPointError as e:
    print(e)  # Domain error
try:
    bad_index = arr[100]
except IndexError as e:
    print(e)  # Index out of bounds

NumPy Date and Time

Work with date/time data. Time travel, but for data.

dates = np.arange('2023-01-01', '2023-01-10', dtype='datetime64[D]')
duration = np.timedelta64(1, 'D')
print(dates + duration)  # Increment dates by one day

NumPy Data Visualization

Combine with libraries like Matplotlib. Because pictures speak louder than numbers.

import matplotlib.pyplot as plt
arr = np.array([1, 2, 3, 4])
plt.plot(arr)  # Line plot
plt.hist(arr)  # Histogram
plt.show()

NumPy Universal Function

Operate element-wise using ufuncs. Fast and functional, just like NumPy.

arr = np.array([1, 2, 3])
print(np.add(arr, 2))  # [3, 4, 5]
print(np.multiply(arr, 2))  # [2, 4, 6]

In this post, I’ll show you how I manage Django settings. It's a lifesaver when you have to deal with different environments.

Why Bother with Django Settings?

Imagine working on your app locally, everything looks perfect. Then you push it to production, and boom! Something breaks. Maybe you left debug mode on, or worse, exposed all your secrets.

Managing settings is like having different outfits for different occasions. You wouldn’t wear pajamas to a job interview, right? Same with Django, your settings should change depending on where you’re running your app.

Using Environment Variables: Keep Your Secrets Hidden

You don’t want your secret keys, database passwords, or API keys lying around in your code for everyone to see. Trust me, it's not a good look.

I use python-decouple to handle this. It helps me load environment variables and keep my settings clean.

First, install it:

pip install python-decouple

Now, in settings.py, grab your environment variables:

# settings.py
from decouple import config

SECRET_KEY = config('SECRET_KEY')
DEBUG = config('DEBUG', cast=bool)

Store the real secrets in a .env file. Don’t forget to add this file to your .gitignore. You don’t want to share your secrets like I did.

# .env
SECRET_KEY='supersecretkey'
DEBUG=True

Congrats! Now, your sensitive data is safe from prying eyes.

Handling Multiple Environments (Because One is Never Enough)

You probably have different setups for development and production. In development, you might use SQLite. But in production, you need something fancier like PostgreSQL.

Here’s how I handle multiple environments without losing my sanity.

Step 1: Split Settings

First, I split my settings into three files:

• base.py for shared settings.

• development.py for local work.

• production.py for the big leagues (aka production).

It looks something like this:

/project
    ├── settings/
    │   ├── __init__.py
    │   ├── base.py
    │   ├── development.py
    │   └── production.py

Step 2: Common Settings in base.py

In base.py, I put everything that’s the same for all environments.

# settings/base.py
from decouple import config

INSTALLED_APPS = [
    # Your apps here
]

SECRET_KEY = config('SECRET_KEY')

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': config('DB_NAME', default='db.sqlite3'),
    }
}

Step 3: Custom Settings for Each Environment

In development.py, I set up things like debugging and using SQLite.

# settings/development.py
from .base import *

DEBUG = True

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': config('DB_NAME', default='db.sqlite3'),
    }
}

In production.py, I switch to PostgreSQL and turn off debugging. No need to let the world see your errors!

# settings/production.py
from .base import *

DEBUG = False

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': config('DB_NAME'),
        'USER': config('DB_USER'),
        'PASSWORD': config('DB_PASSWORD'),
    }
}

Step 4: Load the Right Settings Automatically

In settings/__init__.py, I decide which settings to use based on an environment variable. Fancy, huh?

# settings/__init__.py
from decouple import config

ENV = config('DJANGO_ENV', default='development')

if ENV == 'production':
    from .production import *
else:
    from .development import *

In your .env, you can set the environment like this:

# .env
DJANGO_ENV='development'  # Or 'production'

Now, when you deploy, Django will use the right settings without any extra effort from you.

Static and Media Files: The Pesky Duo

Managing static and media files in different environments can be annoying. Locally, I just serve them from my app. In production, I need something more serious like AWS S3.

For development:

# settings/development.py
STATIC_URL = '/static/'
MEDIA_URL = '/media/'

For production:

# settings/production.py
STATIC_URL = '/static/'
STATIC_ROOT = '/path/to/static/'

MEDIA_URL = '/media/'
MEDIA_ROOT = '/path/to/media/'

Now, my files work everywhere without me worrying about broken images or missing styles.

Keep Secrets Out of Git (Seriously)

Make sure your .env file is in .gitignore. I learned this the hard way. Once, I committed my secret keys to GitHub, and bots immediately started sniffing around my repo.

# .gitignore
.env

Don’t be like me. Keep your secrets secret.

If you think you know everything about them, think again! I recently found a cool new way to use generators that totally surprised me.

What Are Python Generators?

Think of Python generators as smart vending machines for data. Instead of giving you all the items at once, they hand them out one by one.

You get a snack here, a snack there—one yield at a time.

Here’s a simple example:

def count_up_to(max):
    count = 1
    while count <= max:
        yield count
        count += 1

When you use this generator, it’s like having a conveyor belt of numbers:

counter = count_up_to(5)
for number in counter:
    print(number)

And you get:

1
2
3
4
5

No need to store all the numbers at once—just a steady stream of digits.

What I Already Knew

Before my recent discovery, I thought I had generators all figured out. I knew how to make them, use them, and even had some tricks up my sleeve. Here’s what I was doing:

Simple Generator Function

A basic generator function that yields items one by one:

def simple_gen():
    yield 'a'
    yield 'b'
    yield 'c'

You can use this generator like this:

for char in simple_gen():
    print(char)

And you get:

a
b
c

Generator Expressions

I also used generator expressions for creating sequences on the fly:

squares = (x * x for x in range(5))

You can iterate over this generator like so:

for square in squares:
    print(square)

And you get:

0
1
4
9
16

Using next with Generators

Another cool trick I used was the next function, which lets you manually fetch the next item from a generator. Here’s an example:

def count_up_to(max):
    count = 1
    while count <= max:
        yield count
        count += 1

counter = count_up_to(3)

print(next(counter))  # Outputs: 1
print(next(counter))  # Outputs: 2
print(next(counter))  # Outputs: 3
print(next(counter))  # Raises StopIteration

With next, you can grab items one at a time, and when the generator is done, it raises a StopIteration exception to let you know.

Generators were my go-to for handling large data. They helped me read big files line by line and deal with large queries without breaking a sweat.

What I Discovered

I was pretty surprised when I discovered a new way to use generators that made me feel like I was missing out on a great tool.

Nested Generator Function

Enter yield from—a way to use multiple generators together.

Here’s how it works:

def gen1():
    yield 1
    yield 2
    yield 3

def gen2():
    yield 'a'
    yield 'b'
    yield 'c'

def combined_gen():
    print("From First Generator")
    yield from gen1()

    print("From Second Generator")
    yield from gen2()

When you use combined_gen(), it’s like getting data from both gen1 and gen2:

for item in combined_gen():
    print(item)

And you get:

From First Generator
1
2
3
From Second Generator
a
b
c

Using next with Nested Generators

You can also use next to step through nested generators:

def gen1():
    yield 1
    yield 2
    yield 3

def gen2():
    yield 'a'
    yield 'b'
    yield 'c'

def combined_gen():
    print("From First Generator")
    yield from gen1()

    print("From Second Generator")
    yield from gen2()

gen = combined_gen()

print(next(gen))  # Prints: "From First Generator" & Outputs: 1
print(next(gen))  # Outputs: 2
print(next(gen))  # Outputs: 3
print(next(gen))  # Prints: "From Second Generator" & Outputs: a
print(next(gen))  # Outputs: b
print(next(gen))  # Outputs: c
print(next(counter))  # Raises StopIteration

In this example, yield from combines data from two generators, and using next lets you step through the output one item at a time.

It’s like having a remote control for your generator's data stream.

Why This Matters

Finding out about yield from felt like discovering a hidden feature on a tool you already use. It makes your code cleaner and easier to manage, especially when working with multiple data sources.

So, if you think you’ve mastered generators, give yield from a try. It might just make your coding life a lot easier and cooler.

But what’s the point of having a perfectly tuned app if no one can access it? It was time to take that final step: going public.

This post is all about that last-mile journey—where I took my app from "everything works in production" to "everything is live, stable, and ready for the public." Let me walk you through how I did it.

Cloudflare R2: Offloading Static and Media Files

One of the first things I tackled was moving my static and media files off my EC2 instance and into Cloudflare R2. Managing those files directly on the server was never going to scale well, so I decided to offload them to R2. It's like Cloudflare’s version of S3—perfect for my needs.

I set up the bucket and linked it to my app. It didn’t take long, and before I knew it, my app was serving static and media files from R2 instead of the EC2.

That change alone made my setup feel more polished and took some weight off the server, freeing it up to do the heavy lifting.

Monitoring with Prometheus and Grafana

With the app running smoothly, my next concern was keeping tabs on everything. After all, if something crashes, I want to know about it before my users do. Time to use Prometheus and Grafana.

Prometheus started scraping all kinds of metrics from my Docker containers—everything from CPU and memory usage to request counts. Grafana took all that data and made it pretty with some slick dashboards.

Now I can pull up a real-time view of how the app is performing anytime I want. Plus, I set up some alerts to notify me if anything starts going off the rails.

Of course, that meant I needed secure access to these monitoring tools without exposing them directly to the internet. This is where Tailscale came in handy.

Setting Up DNS for Prometheus and Grafana with Tailscale

I configured Prometheus and Grafana to run in my private Docker network alongside my other containers. But rather than leaving these services open to the public, I decided to route them through Nginx using Tailscale’s private VPN.

I have already set up Tailscale, which gave me a secure, private IP that could only be accessed within my VPN network. Then, I configured DNS entries in Cloudflare for both Prometheus and Grafana as well, similar to Portainer.

Also, I had to get SSL certificates for new subdomains and set them up in the Nginx config.

The DNS entries pointed to my private Tailscale IP, and from there, Nginx routed the traffic to the appropriate services within the Docker network.

With this setup, I could access prometheus.mydomain.com and grafana.mydomain.com securely through Tailscale, while keeping both services completely invisible to the outside world. All the management and monitoring power, none of the public exposure.

Wrapping It Up

After getting all that setup, offloading static files, monitoring performance, and securely accessing Prometheus and Grafana through Tailscale, it was time to flip the switch.

And that’s it, the app is now public, live, and working as expected. I’m happy with where it’s at, and I’ve got everything in place to keep it running smoothly.

Setting Up the Domain and SSL

First off, I grabbed a domain—nothing fancy, just the usual process of buying from a registrar. For this setup, I used Cloudflare to manage the DNS. Here’s how it went down:

Pointing the Domain to Cloudflare

I signed up on Cloudflare, added my domain, and followed their instructions to point my domain to their nameservers. Simple enough, right? Just had to update the nameservers with my domain registrar.

Configuring SSL/TLS

Next up, I wanted to make sure everything was secure. In Cloudflare, I set the SSL/TLS encryption mode to "Strict". This mode ensures that traffic between Cloudflare and my origin server is encrypted.

I also generated a certificate for my origin server. This certificate would come into play later when I set up Nginx.

Dockerizing the App

Alright, with the domain and SSL sorted, it was time to Dockerize my app and get things rolling on AWS.

I kicked things off by Dockerizing my app, and that meant writing up a Docker Compose file. Nothing too crazy—just setting up Nginx alongside my app in the same file.

Remember that origin certificate I created earlier in Cloudflare? Well, I mounted that crt and key files in the Nginx container and used it for SSL.

Then, I made sure everything was working by opening my app through Nginx. Success!

Setting Up AWS

With Docker set up, it was time to get AWS involved. I launched an EC2 instance, making sure to configure it with the right VPC, subnet, route table, and security groups to allow traffic on ports 22, 80, and 443. I added my PC's SSH key as well.

It was time to put Docker to work for what it was built for. So, I went ahead and installed Docker on my EC2 machine and copied my project over using SCP.

Once that was done, I SSHed into the EC2 instance and started up the container. Simple as that!

I verified everything was set up correctly by opening the public IP of my EC2 in the browser.

Yahhooo! My hobby project was officially live on the internet. But hold up—was I really going to stick with the IP address? Do I have to type out the IP instead of using mydomain.com?

Final Touches: DNS and Portainer

With the app running, it was time to polish things off.

Configuring Cloudflare DNS

I went back to Cloudflare and added an A Record pointing to my EC2’s public IP. I also added a CNAME Record for the www subdomain. And voilà—my website was live with my own domain!

Thank god I didn't have to use IP anymore.

Managing Docker Containers with Portainer

To keep an eye on my Docker containers, I went ahead and installed Portainer, hooking it up to the Nginx network.

Then, I tweaked the Nginx config to route traffic to portainer.mydomain.com. I also added WebSocket config in Nginx since I needed to access the container's shell in Portainer, and WebSockets makes that happen.

After that, I jumped back into Cloudflare and added a subdomain for Portainer, pointing it right at the EC2’s public IP.

I checked to make sure Portainer was set up properly, and yep, everything was working smoothly—I could access everything without a hitch. But, there was a catch: everything was out there on the public internet.

Portainer’s own docs even mention that its default authentication isn’t exactly recommended for security.

Setting Up Tailscale VPN

After setting up Portainer for managing my Docker containers, I realized I didn’t want it to be accessible over the public internet because Portainer has more powers than Superman himself.

The solution? Using Tailscale, a mesh VPN that makes it easy to securely access services over a private network.

First, I needed to install Tailscale on my EC2 instance. And I did exactly that. Then, started Tailscale and logged in with my account to connect the EC2 instance to my Tailscale network.

Once Tailscale was up and running, it assigned a private IP to my EC2 instance, something like 100.x.x.x. I noted this IP down because I’d be using it to access Portainer securely.

Now that my EC2 instance had a private Tailscale IP, I changed A record for portainer.mydomain.com in Cloudflare to point at that private IP.

With everything set up, I connected my laptop to the Tailscale network, which gave me access to the private IP of the EC2 instance.

Now, whenever I go to portainer.mydomain.com, it securely routes the traffic through the VPN, keeping Portainer unreachable from the public internet.

There was another hiccup: Cloudflare’s certificate only covers Cloudflare and my server, so when I used a private IP for Portainer, SSL went kaput.

To fix that, I grabbed an SSL certificate for the subdomain using ZeroSSL and installed it in Nginx. Problem solved!

Final Words

Finally, I set up GitHub Actions for automatic deployments.

And that’s a wrap! From Dockerizing my app to setting up AWS and automating deployments, it’s been a wild ride.

I hope this helps if you’re on a similar journey. Feel free to drop any questions or thoughts in the comments. Cheers to seamless deployments!

A credit card is compulsory to create an account.

Once the account is created, the name and other details can be changed via Account Settings from the top right menu in the AWS Console Dashboard.

Identity Access Management (IAM)

AWS has a feature to manage multiple users in the same account. These users are called IAM users and can have different roles and permissions, which AWS calls groups and policies.

IAM users can be created by searching IAM in the top search bar and clicking on IAM under the services section. Doing this within an account is good practice.

AWS Region Selector

Certain services in AWS are region-dependent. For example, EC2 instances can be created in different locations.

However, this is not true for every service in AWS. Services like Cloudfront, S3 Buckets, etc. are not created in any one specific region.

The active region can be changed from the top right dropdown (besides the Account menu). If any service created does not appear in the dashboard, the wrong region selection could be an issue.

Over Billing Story

There are multiple stories about AWS overbilling rumored in the market. They are true.

Since AWS has a Pay as you Go policy, there are no monthly or annual costs. AWS charges for services according the usage. If an instance costs $1 per hour, the total amount for 2 instances in one hour would be $2.

If there are services that are created by mistake, or left on for a long time, it affects the billing. Even the default values while creating an instance should be noted carefully. Sometimes, larger variants are selected by default whereas smaller variants could be enough.

There are multiple AWS services to control the budget. Services like Budget Alerts and Billing Alarm could be used to control the spending on AWS.

There is also a free tier available. Visit the AWS pricing website for more details.

Cloud computing is the practice of using network servers hosted on the internet to store, manage, and process data rather than a local server or a personal computer.

Evolution of Cloud Hosting

1. Dedicated Server

2. Virtual Private Server (VPS)

3. Shared Hosting

4. Cloud Hosting

Amazon

Amazon is an American multinational computer technology corporation headquartered in Seattle, Washington. It was founded in 1994 by Jeff Bezos as an online bookstore.

Amazon Services

• Amazon Web Services

• Amazon Prime Video, Amazon Prime Music, Twitch.tv

• Amazon Store

• AI and Satellites

• Many many more.

Currently, the CEO of Amazon is Andy Jossy.

Amazon Web Services (AWS)

Founded in 2006 as a Cloud Service Provider (CSP). However, Simple Queue Service (SQS) existed before, i.e. was launched in 2004.

After that, Simple Storage Service (S3) and Elastic Compute Cloud (EC2) were launched in March 2006 and August 2006 respectively.

Since April 2013, Amazon also started AWS Certification. Currently, the CEO of AWS is Matt Garman and the CTO of AWS is Werner Vogels.

Cloud Service Providers (CSPs)

The company that provides multiple cloud services that can be chained together to create cloud architectures are called Cloud Service Providers.

All the features are accessible via a single unified API and utilize the metered billing (cost per usage). These providers also provide rich monitoring and IaaS (Infrastructure as a Service).

Example: AWS

If the company does not meet most of the requirements, they are referred to as Cloud Platforms. For example; Twillio, Hashicorp, etc.

Landscape of CSPs

1. Tier 1

   AWS, Azure, Google Cloud, Alibaba Cloud

2. Tier 2

   IBM Cloud, Oracle Cloud, Huawei Cloud, Tencent

3. Tier 3

   Vultr, Digital Ocean, Akamai Connected Cloud (Linode)

4. Tier 4

   OpenStack, Apache CloudStack

Common Cloud Services

AWS has 200+ cloud services such as; Analytics, Game Tech, Machine Learning, etc.

Evolution of Computing

1. Dedicated

   Physical server utilized by a single customer.

2. Virtual Machines

   Multiple virtual machines on one machine.

3. Containers

   Multiple containers running on a virtual machine.

4. Functions

   Serverless computing that runs only upon triggered.

Types of Cloud Computing

• SaaS (Software as a Service)

  These types of services are managed by service providers. They are specifically designed for customers.

• PaaS (Platform as a Service)

  These services focus on the deployment and management of the application in the cloud. These services are targeted for developers.

• IaaS (Infrastructure as a Service)

  This service provides access to the full architecture. These services are mostly targeted at admins and DevOps.

Cloud Computing Deployment Models

Public Cloud

• Everything is built on the CSP

• Also known as cloud-first or cloud-native

• Suitable for startups, SaaS offerings

Private Cloud

• Everything is built on the company's data center

• Also known as on-premise

• Suitable for banks, fintech companies

Hybrid

• Using both, on-premise and CSPs

• Suitable for large enterprises

Cross-Cloud

• Using multiple CSPs at once

• Also known as multi-cloud

• Suitable for enterprises with distributed environment

Recursion in programming is a technique where a function calls itself directly or indirectly. This approach allows us to break down complex problems into smaller, more manageable chunks.

For an example,

def recursion():
    return recursion()

It is essential to note that, there must be a base case and a recursive case in a function. If the base case is not mentioned, the function will call itself continuously and the system's stack will be full.

That's how StackOverflow was found. Wait, what? Oops, that's when Stack Overflow exception is raised.

Stack

A stack is a special area of computer memory that stores information about a computer program's active subroutines or functions.

A new frame is added to the call stack each time a function is called. This frame contains the return address, local variables, parameters, and saved registers.

Yes yes, the behavior is similar to stack that we read in data structures.

Base Case and Recursive Case

• Base case: A condition under which the function should stop calling itself, to prevent the infinite recursion.

• Recursive Case: The part of a function where it calls itself with a modified argument towards the base case.

Let's take an example,

def print_upto_one(number):
    print(number)

    if number == 1:
        return

    print_upto_one(number-1)

Here, the function print_upto_one prints the passed number, calls itself again with decreased value. Also note that there is a condition to stop calling itself, i.e., when the value of number is 1.

• if number == 1: return is a base case where the function will stop calling itself.

• print_upto_one(number-1) is a recursive case when the function calls itself with a decreased value.

Recursion vs Infinite Loop

Wait, this sounds a lot like infinite loop. What's the difference?

Not exactly. Let's understand it this way,

Recursion is like Russian nesting dolls where each doll contains a smaller doll inside until you reach the smallest one.

Infinite Loop is like a song stuck on repeat, playing the same part endlessly.

A new stack frame will be allocated to each function call in recursion whereas an infinite loop runs in the same amount of memory. That's why endless recursion will raise stack overflow error but infinite loop will keep running until the user or the system terminates it explicitly.

Detailed Working of Recursion

Recursion works in stack memory. Each function called inside the function returns the value from where it is called.

Meaning, let's assume we have a function A and each function call as A1, A2, A3 and so on.

Now, when A1 calls A2 and A2 calls A3, A3 will return value to A2 and A2 will return value to A1 that the caller function can process.

Let's understand this with an example. This function calculates the sum of n natural numbers.

def sum(n):
    if n == 1:
        return 1
    inner_func_value = sum(n - 1)
    return n + inner_func_value

Visual Representation

Here, the function calls itself with a decreased value until the passed value reaches to 1. Then, every function output value is returned to the outer function, and so on.

I know I know, this is a bit confusing while getting started.

Factorial of a number using Recursion

Let's understand Recursion with one more example. This function will calculate the factorial of a number.

def factorial(n):
    if n == 1:
        return 1
    return n * factorial(n - 1)

Now, let's also understand the function calls step by step.

factorial(3)                        # 1st call with value 3
    3 * factorial(2)                # 2nd call with value 2 (3-1)
        3 * (2 * factorial(1))      # 3rd call with value 1 (2-1)
        3 * (2 * 1)                 # 3rd call returns 1 (base case)
    3 * 2                           # 2nd call returns 2 (2 * 1)
6                                   # 1st call returns 6 (3 * 2)

Let's break down each step:

1. First call: factorial(3)

   • Calculates 3 * factorial(2)

2. Second call: factorial(2)

   • Calculates 2 * factorial(1)

3. Third call: factorial(1)

   • Base case: Returns 1 since factorial(1) = 1

4. Return to second call:

   • Calculates 2 * 1 = 2

   • Returns 2

5. Return to first call:

   • Calculates 3 * 2 = 6

   • Returns 6

So, the factorial of 3 is correctly calculated as 6.