01. Generative AI Foundations, Transformers & Embeddings

1. What is Generative AI?

🇬🇧 English Definition

Generative AI is a class of artificial intelligence systems that learn patterns from training data and generate brand new content such as text, code, images, audio, video, and structured data.

🇮🇳 Hinglish Definition

Simple language mein: traditional software rules follow karta hai, jabki GenAI learned patterns ke basis par new original content generate karta hai. Chatbots, image generators aur coding assistants iske major examples hain.

// Basic Conceptual Representation of GenAI Inference
def generate_content(user_prompt, model_parameters):
    // 1. Process and tokenize user input prompt
    tokens = tokenize(user_prompt)
    // 2. Pass tokens through neural network layers
    predictions = model_parameters.forward(tokens)
    // 3. Decode output token probabilities into text
    output_text = decode(predictions)
    return output_text

print(generate_content("Explain gravity", model_params=None))
            

🔍 Line-by-Line Code Breakdown

def generate_content(user_prompt, model_parameters): Defines function taking prompt and model weight parameters.
tokens = tokenize(user_prompt): Converts raw text into numerical token representation.
predictions = model_parameters.forward(tokens): Computes probabilistic next token distribution.
output_text = decode(predictions): Converts numeric output IDs back into readable natural language string.

🎯 Interview Focus & Questions

Q: What differentiates Generative AI from Discriminative AI?
A: Discriminative AI predicts labels or decisions given input data (P(Y|X)), whereas Generative AI models the underlying data distribution (P(X) or P(X,Y)) to produce whole new data samples.

2. AI vs ML vs Deep Learning vs Generative AI

🇬🇧 English Definition

Artificial Intelligence is the broad umbrella of intelligent systems. Machine Learning is the data-driven subset. Deep Learning uses multi-layered artificial neural networks. Generative AI is a subfield of Deep Learning focused specifically on content generation.

🇮🇳 Hinglish Definition

Hierarchy ko yaad rakho: AI (Broadest Umbrella) → Machine Learning (Data-driven approach) → Deep Learning (Multi-layer Neural Nets) → Generative AI (Content Generation using Deep Nets).

// Hierarchy representation in Python
ai_ecosystem = {
    "AI": "Broad umbrella of smart machines",
    "ML": "Algorithms learning from data without hardcoding",
    "DL": "Neural networks with multiple hidden layers",
    "GenAI": "Deep learning models specialized in producing synthetic text, images & media"
}
print("GenAI Scope:", ai_ecosystem["GenAI"])
            

🔍 Line-by-Line Code Breakdown

ai_ecosystem dictionary: Maps each layer of the AI paradigm to its specific core objective.
GenAI: Highlights that modern text/image GenAI relies heavily on deep transformer architectures.

🎯 Interview Focus & Questions

Q: Does all AI use Neural Networks?
A: No. AI includes expert systems and search algorithms. However, modern Generative AI relies almost exclusively on deep neural networks like Transformers and Diffusion models.

3. Generative Models & Core Concepts

🇬🇧 English Definition

Generative models learn joint probability distributions P(X) or conditional distributions P(X|Y) to sample new realistic instances. Autoregressive models predict tokens sequentially, while Diffusion models generate images through iterative noise reduction.

🇮🇳 Hinglish Definition

Generative model training data se patterns seekhta hai. Text models token-by-token next word predict karte hain (Autoregressive), jabki Image models noise ko clean/denoise karke image banate hain (Diffusion).

// Autoregressive Token Generation Simulation
import numpy as np

def sample_next_token(logits):
    // Convert logits into probabilities via Softmax
    probs = np.exp(logits) / np.sum(np.exp(logits))
    // Sample next token ID based on probability distribution
    token_id = np.random.choice(len(probs), p=probs)
    return token_id

logits_example = [1.2, 3.5, 0.4, 2.1]
print("Sampled Token Index:", sample_next_token(logits_example))
            

🔍 Line-by-Line Code Breakdown

np.exp(logits) / np.sum(...): Softmax function normalizing raw model logits into a valid probability distribution.
np.random.choice(...): Samples a token index proportional to its computed probability.

🎯 Interview Focus & Questions

Q: What is an Autoregressive Model?
A: An autoregressive model predicts the current output token conditioned on all previously generated tokens in the sequence (e.g., GPT family).

4. Neural Networks for GenAI

🇬🇧 English Definition

Neural networks are parameterized continuous functions composed of input layers, hidden weight transformations, activation functions, and output layers that adjust weights via backpropagation and gradient descent.

🇮🇳 Hinglish Definition

Neural network ko ek giant mathematical function samjho. Training mein error compute karke weights update hote hain (Backpropagation) taaki model better output generate kar sake.

// Multi-layer Artificial Neuron Forward Pass
import numpy as np

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def neuron_layer(X, W, b):
    // Linear matrix multiplication plus bias
    Z = np.dot(X, W) + b
    // Non-linear activation function
    A = sigmoid(Z)
    return A

X = np.array([0.5, 1.2])
W = np.array([[0.8, -0.4], [0.3, 0.9]])
b = np.array([0.1, -0.2])
print("Layer Output:", neuron_layer(X, W, b))
            

🔍 Line-by-Line Code Breakdown

sigmoid(x): Activation function mapping inputs into range (0, 1).
np.dot(X, W) + b: Linear algebra transformation representing weighted input sum plus bias term.

🎯 Interview Focus & Questions

Q: Why are non-linear activation functions crucial in Neural Networks?
A: Without non-linear activations (like ReLU, GELU, Sigmoid), stacking multiple neural layers would collapse mathematically into a simple single linear transformation.

5. Transformers

🇬🇧 English Definition

The Transformer is a deep learning architecture relying on self-attention mechanisms, eliminating recurrent loops and enabling massive parallel training across sequence tokens.

🇮🇳 Hinglish Definition

Transformer modern GenAI ka foundation hai. Self-attention mechanism tokens ke beech ke rich contextual relationships ko process karta hai aur parallel GPU computation allow karta hai.

// Scaled Dot-Product Attention Concept
import numpy as np

def scaled_dot_product_attention(Q, K, V):
    d_k = Q.shape[-1]
    // 1. Matrix multiplication of Query and Key transpose
    scores = np.matmul(Q, K.T) / np.sqrt(d_k)
    // 2. Softmax normalization over last axis
    weights = np.exp(scores) / np.sum(np.exp(scores), axis=-1, keepdims=True)
    // 3. Multiply weights with Value matrix
    output = np.matmul(weights, V)
    return output, weights

Q = np.random.randn(4, 64)
K = np.random.randn(4, 64)
V = np.random.randn(4, 64)
attn_out, weights = scaled_dot_product_attention(Q, K, V)
print("Attention Output Shape:", attn_out.shape)
            

🔍 Line-by-Line Code Breakdown

np.matmul(Q, K.T) / np.sqrt(d_k): Calculates pairwise token similarity scores scaled by vector dimension sqrt(d_k).
weights * V: Aggregates contextual information from Values based on computed attention weights.

🎯 Interview Focus & Questions

Q: What are Query, Key, and Value vectors in Attention?
A: Query represents what token seeks, Key represents what token contains, and Value represents the actual vector information transferred when Query matches Key.

6. Tokens & Tokenization

🇬🇧 English Definition

Tokens are the fundamental atomic units processed by language models. Tokenization converts raw text strings into numerical token IDs using algorithms like Byte-Pair Encoding (BPE) or WordPiece.

🇮🇳 Hinglish Definition

LLM direct English ya Hindi words nahi samajhta. Tokenizer text ko subword pieces / IDs mein break karta hai aur final answer generate hone par IDs ko wapas human text mein convert karta hai.

// Tokenization Demo with Python Regex & Dict mapping
text = "Generative AI is revolutionary!"
// Simple token splitter simulation
subwords = ["Gen", "erative", " AI", " is", " rev", "olution", "ary", "!"]
vocab_map = {word: idx for idx, word in enumerate(subwords)}

token_ids = [vocab_map[w] for w in subwords]
print("Tokens:", subwords)
print("Token IDs:", token_ids)
            

🔍 Line-by-Line Code Breakdown

vocab_map: Dictionary mapping string tokens to numerical integer identifiers.
token_ids: Numerical sequence fed directly into embedding layer.

🎯 Interview Focus & Questions

Q: Why is subword tokenization preferred over word or character tokenization?
A: Subword tokenization balances vocabulary size and out-of-vocabulary (OOV) issues by breaking rare words into common subword chunks while keeping frequent words intact.

7. Embeddings

🇬🇧 English Definition

An embedding is a continuous dense numerical vector representation of text or tokens in a high-dimensional vector space where semantically similar items reside close to each other.

🇮🇳 Hinglish Definition

Embedding text ka mathematical vector hota hai. Jab do sentences ka matlab similar hota hai, to unki embedding vectors geometric space mein bilkul paas hoti hain.

// Cosine Similarity between Embeddings
import numpy as np

def cosine_similarity(v1, v2):
    return np.dot(v1, v2) / (np.linalg.norm(v1) * np.linalg.norm(v2))

vec_king = np.array([0.25, 0.88, 0.12])
vec_queen = np.array([0.28, 0.85, 0.15])
vec_apple = np.array([-0.90, 0.10, 0.82])

print("Similarity (King, Queen):", cosine_similarity(vec_king, vec_queen))
print("Similarity (King, Apple):", cosine_similarity(vec_king, vec_apple))
            

🔍 Line-by-Line Code Breakdown

np.dot(v1, v2): Dot product measuring vector directional agreement.
np.linalg.norm(v1): Magnitude normalization scaling cosine similarity output between -1 and +1.

🎯 Interview Focus & Questions

Q: How do dense embeddings differ from sparse one-hot encodings?
A: One-hot vectors are high-dimensional and sparse with zero semantic relationship, whereas dense embeddings compress semantic features into fixed lower-dimensional continuous vectors.

8. Large Language Models (LLMs)

🇬🇧 English Definition

Large Language Models are massive transformer-based neural networks trained on vast text corpora to model language semantics, follow complex instructions, and perform multi-domain reasoning.

🇮🇳 Hinglish Definition

LLM billion-parameter deep neural network hota hai jo extensive data par pretrain hakar text generate karta hai. Remember: LLMs probability distribution par operate karte hain, automated fact lookup DB nahi hain.

// High-Level Model Text Generation Loop Simulation
class DummyLLM:
    def __init__(self):
        self.vocab = ["The", "future", "of", "AI", "is", "bright", "."]
    
    def generate(self, prompt, max_tokens=5):
        output = prompt.split()
        for i in range(max_tokens):
            next_word = self.vocab[i % len(self.vocab)]
            output.append(next_word)
        return " ".join(output)

llm = DummyLLM()
print(llm.generate("User Input:"))
            

🔍 Line-by-Line Code Breakdown

DummyLLM.generate(...): Simulates step-by-step next word prediction append loop common in autoregressive LLMs.

🎯 Interview Focus & Questions

Q: What are hallucinations in LLMs?
A: Hallucinations occur when an LLM generates plausible-sounding but factually incorrect, ungrounded, or completely fabricated information due to probabilistic token sampling.

9. Model Parameters & Context Window

🇬🇧 English Definition

Model parameters represent learned neural weights. The context window defines the maximum token capacity (prompt + history + response + retrieval) a model can process in a single inference call.

🇮🇳 Hinglish Definition

Parameters model ki learned knowledge & capacity hote hain. Context window wo maximum limit hai jitne tokens (system prompt, user input, chat memory, retrieved context) ek time par process ho sakte hain.

// Context Window Budget Calculator in Python
def check_context_limit(system_prompt, user_query, doc_context, max_window=8192):
    // Estimate 1 token ~= 4 chars
    total_chars = len(system_prompt) + len(user_query) + len(doc_context)
    estimated_tokens = total_chars // 4
    
    remaining = max_window - estimated_tokens
    if remaining < 0:
        return False, f"Overflow by {-remaining} tokens"
    return True, f"Valid. {remaining} tokens remaining for response."

print(check_context_limit("You are helpful assistant.", "What is RAG?", "Deep document content..."*1000))
            

🔍 Line-by-Line Code Breakdown

estimated_tokens = total_chars // 4: Practical rule-of-thumb estimating token count from character length.
remaining: Ensures total context fits strictly within model window constraint.

🎯 Interview Focus & Questions

Q: Does a larger context window eliminate the need for Retrieval-Augmented Generation (RAG)?
A: No. Larger context windows incur high computational cost, high latency, and needle-in-a-haystack retrieval drop-offs, whereas RAG selectively retrieves precise relevant facts.

10. Pretraining

🇬🇧 English Definition

Pretraining is the initial self-supervised training phase where an LLM learns language syntax, grammar, general reasoning, and world knowledge by predicting masked or next tokens across billions of web documents.

🇮🇳 Hinglish Definition

Pretraining base phase hai jahan model billions of text lines par self-supervised mode (next-token prediction) mein train hota hai. Complex compute & huge hardware cost is level par lagti hai.

// Self-Supervised Next Token Prediction Loss Concept
import torch
import torch.nn as nn

loss_fn = nn.CrossEntropyLoss()
// Logits: [batch_size, vocab_size]
logits = torch.tensor([[2.0, 0.5, 0.1], [0.1, 1.8, 0.3]])
// Target Token IDs
targets = torch.tensor([0, 1])

loss = loss_fn(logits, targets)
print("Pretraining Cross-Entropy Loss:", loss.item())
            

🔍 Line-by-Line Code Breakdown

nn.CrossEntropyLoss(): Measures loss between model token probability distribution and actual target token ID.
loss.item(): Scalar value backpropagated to update model weights during pretraining.

🎯 Interview Focus & Questions

Q: What dataset objective is used in GPT pretraining?
A: Causal Language Modeling (CLM), where the model predicts token X_t given previous tokens [X_1, X_2, ..., X_{t-1}].