![\"Chrome](\"https://blogs.pranjalkumar.in/assets/images/chrome-device-bound-session-credentials-explained%20by%20Pranjal%20Kumar%20-%20software%20engineer-6ef0d66599ca8cdfcf5f8915764e0b0d.png\")
\nMost authentication systems are built around a comforting illusion: once a user successfully logs in, the system believes it knows who is on the other side of the connection.\nMost authentication systems are built around a comforting illusion: once a user successfully logs in, the system believes it knows who is on the other side of the connection.
In reality, most web applications do not continuously know who is using the session. They know only that each request carries a valid cookie.
\nThat distinction is not academic. It is the reason session theft remains one of the most effective ways to bypass passwords, MFA, passkeys, risk checks, and even carefully designed identity-provider flows.
\nChrome's Device Bound Session Credentials, commonly called DBSC, is aimed directly at this weakness. It changes the trust model of web sessions from:
\nBearer session model:
If you have the cookie, you are the session.
To:
\nDevice-bound proof-of-possession model:
If you have the cookie, you must also prove you are still on the original device.
That is a fundamental architectural shift.
\nDBSC is not just another browser feature. It is part of a broader movement away from portable bearer credentials and toward credentials that are bound to context, device, and proof of possession.
\nA traditional session cookie is a bearer credential. “Bearer” means possession is enough.
\nThe server does not ask:
\nIf the cookie is valid and unexpired, the request is generally treated as authenticated.
\nThat gives us a simple but dangerous flow:
\nUser logs in with password + MFA
↓
Server issues session cookie
↓
Malware steals cookie
↓
Attacker imports cookie elsewhere
↓
Server accepts attacker as logged-in user
This is why cookie theft can bypass MFA. MFA protects the login ceremony, but once a session has been established, the cookie often becomes the real proof of authentication.
\nThe attacker does not need to defeat cryptography. They do not need to brute-force a password. They do not need to intercept a one-time password. They only need the browser's already-issued session artifact.
\nThat is why infostealer malware is so effective. It targets browser session state, extracts cookies or tokens, and replays them elsewhere.
\nThe uncomfortable truth is this:
\n\n\nIf a credential is portable, attackers will build an economy around stealing and replaying it.
\n
DBSC exists because the web's dominant session model made authenticated sessions too easy to move from one device to another.
\nDBSC does not try to make cookies impossible to steal. That would be unrealistic.
\nIf malware runs with sufficient access on a user's machine, it may still be able to read browser-accessible state, observe network activity, or abuse the active browser context.
\nInstead, DBSC attacks the attacker's next step: replay.
\nThe goal is not:
\nNo attacker can ever copy a cookie.
The goal is:
\nA copied cookie should not be enough to continue the session from another machine.
DBSC does this by binding session continuity to a cryptographic key pair generated on the user's device.
\nAt a high level:
\nSo the new session equation becomes:
\nValid continued session = short-lived cookie + proof of device-held private key
That is the core DBSC idea.
\nThe attacker may steal the cookie, but once that short-lived cookie expires, they cannot renew the session from another machine unless they can also prove possession of the original device-bound private key.
\nDBSC introduces three important backend concepts:
\nThe important design choice is that DBSC is additive. Applications do not need to rewrite every endpoint to understand a brand-new authentication system. Instead, DBSC is concentrated around login, registration, and refresh.
\nMost application endpoints can continue checking cookies as they already do.
\nStep 1: user logs in normally
\nThe user authenticates using the site's existing login mechanism:
\nDBSC does not replace initial authentication. It strengthens what happens after authentication.
\nThat distinction matters.
\nPasskeys and MFA answer:
\nCan this user authenticate now?
DBSC answers:
\nCan this already-authenticated session still prove it belongs to the original device?
Those are related, but different, security questions.
\nStep 2: server asks Chrome to start a device-bound session
\nAfter login, the server sends a DBSC registration instruction using an HTTP response header such as:
\nSecure-Session-Registration: ...
Conceptually, the server is telling Chrome:
\nFor this authenticated session, create a device-bound credential and register it with me.
This is the bootstrap step.
\nStep 3: Chrome generates a public/private key pair
\nChrome generates a new asymmetric key pair for the session.
\nPublic key → sent to server
Private key → protected on the device
The public key is not secret. The server stores it and later uses it to verify signatures.
\nThe private key is the important artifact. It should not be exportable like a normal file. Ideally, it is protected by hardware-backed security such as a TPM on Windows or Secure Enclave-style protection on macOS.
\nStep 4: Chrome registers the public key
\nChrome calls the site's registration endpoint and sends the public key and related session material.
\nThe server may store records similar to:
\nuser_id
session_id
public_key
refresh_endpoint
cookie_scope
registration_time
expiration_policy
risk_policy
The private key never leaves the device.
\nStep 5: server issues short-lived cookies
\nThis is one of the most important parts of DBSC.
\nThe server should not continue issuing long-lived bearer cookies. Instead, it transitions to short-lived cookies.
\nThe point is not that cookies become impossible to steal. The point is that stolen cookies become much less valuable because they expire quickly and cannot be refreshed without the device-bound private key.
\nThe model changes from:
\nLong-lived bearer cookie
To:
\nShort-lived cookie + hardware-backed/session-bound renewal
Step 6: Chrome refreshes the session with proof of possession
\nWhen Chrome needs to make a request and notices that the DBSC-controlled cookie is missing or expired, Chrome can pause the original request and call the configured refresh endpoint.
\nThe refresh flow looks like this:
\nOriginal request needs valid cookie
↓
Cookie expired or missing
↓
Chrome calls refresh endpoint
↓
Server returns challenge
↓
Chrome signs challenge using device-held private key
↓
Server verifies signature using stored public key
↓
Server issues fresh short-lived cookie
↓
Original request resumes
This lets the user experience remain smooth while session continuation becomes cryptographically bound to the original device.
\nThat is the beauty of DBSC: it adds a stronger trust boundary without turning every session renewal into a visible user challenge.
\nA natural question is:
\n\n\nWhy not just make all session cookies expire every five minutes?
\n
Because short expiry alone creates a usability problem.
\nIf cookies expire too quickly, users are constantly forced to reauthenticate. To avoid that, most systems introduce a long-lived refresh token or long-lived session cookie.
\nBut then the refresh artifact becomes the new bearer credential.
\nShort-lived access cookie + long-lived bearer refresh token
If malware steals the refresh token, the attacker can continue minting new session cookies.
\nDBSC changes that equation.
\nTraditional refresh:
Refresh token is a bearer credential.
DBSC refresh:
Refresh requires proof of possession of a device-held private key.
This is a materially better security/usability tradeoff.
\nUsers do not need to log in repeatedly, but attackers cannot simply copy the refresh capability to another machine.
\nThe TPM is one of the most important pieces of the DBSC story.
\nA Trusted Platform Module, or TPM, is a hardware-backed cryptographic component designed to generate, protect, and use cryptographic keys. It can create keys whose private material is difficult or impossible to export through ordinary software APIs.
\nFor DBSC, the browser needs a private key with three properties:
\nThat is exactly the kind of problem TPMs are designed to support.
\nImagine Chrome generated a DBSC private key and stored it as a normal file:
\n~/Chrome/Profile/DBSC/private-key.pem
That would not meaningfully solve cookie theft.
\nThe same malware that steals cookies could steal the private key file too.
\nThen the attacker would have:
\nstolen cookie + stolen private key = replayable session
At that point, DBSC would collapse back into a bearer-like model.
\nThe key cannot merely be “local.” It must be protected in a way that makes extraction hard.
\nThis is where TPM-backed storage matters.
\nA TPM-backed key behaves conceptually like this:
\nApplication: Please sign this challenge.
TPM: Here is the signature.
Application: Please give me the private key.
TPM: No.
The browser can use the key, but it cannot casually export the raw private key material.
\nIn DBSC, the TPM-backed key does not need to decrypt data or reveal secrets. It primarily needs to sign challenges.
\nThe flow is conceptually:
\nServer creates challenge:
nonce + session_id + timestamp + origin context
Chrome asks protected key:
sign(challenge)
Chrome sends to server:
challenge + signature
Server verifies:
verify(public_key, challenge, signature)
If verification succeeds, the server knows that the client can still access the original device-bound private key.
\nThis nuance is critical.
\nTPM-backed DBSC proves:
\nThe session can still access the original device-bound key.
It does not prove:
\nThe device is malware-free.
The user is physically present.
The browser process is uncompromised.
The OS is fully trustworthy.
If malware is running on the same machine and can control the browser or trigger signing operations, it may still abuse the active session locally.
\nThis is sometimes called the signing oracle problem.
\nA non-exportable TPM key prevents this:
\nExport private key → use it anywhere
But it may not fully prevent this:
\nCompromised local device → ask browser/TPM to sign challenge locally
So DBSC changes the attacker's path.
\nBefore DBSC:
\nSteal cookie → replay anywhere
After DBSC:
\nSteal cookie → short-term access only
Need original device key → must compromise or operate from original device
That is still a huge improvement.
\nIt turns a scalable remote replay attack into a harder local compromise problem.
\nDBSC should not be understood only as a browser feature. It should be understood as an economic attack against infostealer malware.
\nInfostealers are profitable because session credentials are portable.
\nA typical attacker workflow looks like this:
\nCompromise endpoint
↓
Extract browser cookies
↓
Upload cookies to attacker infrastructure
↓
Sell or replay sessions
↓
Bypass MFA because login already happened
The scalability comes from portability. Once a cookie is stolen, the attacker can use it elsewhere.
\nDBSC reduces that value.
\nWith DBSC:
\nCookie stolen today
↓
Cookie works only until short-lived expiry
↓
Attacker cannot refresh without original device-bound private key
This does not make attacks impossible, but it damages the attacker's return on investment.
\nThat is what good security architecture often does. It does not merely block one technique; it changes the economics of the attack.
\nDBSC belongs to a broader family of proof-of-possession and sender-constrained credential systems.
\nOAuth DPoP, standardized in RFC 9449, follows a similar principle for OAuth tokens. Instead of allowing a token to be used by anyone who possesses it, DPoP binds token usage to proof that the client possesses a private key.
\nThe comparison looks like this:
\nBearer token:
token alone is enough
DPoP:
token + proof of private key required
DBSC:
cookie + proof of device-bound private key required
The philosophy is the same: make stolen credentials harder to replay.
\nThe difference is where they operate.
\nDPoP is primarily an OAuth/API token mechanism.
\nDBSC is a browser/session-cookie mechanism.
\nThat makes DBSC especially relevant for traditional web applications that still rely heavily on cookies and browser-managed session state.
\nDBSC is not the first attempt to bind credentials to a client.
\nEarlier approaches, such as Token Binding, tried to bind tokens at the TLS layer. The idea was elegant: cryptographically bind tokens to the TLS connection so stolen tokens could not be replayed elsewhere.
\nBut deployment reality made this hard.
\nModern web infrastructure often looks like this:
\nBrowser
↓ TLS
CDN / edge proxy
↓ internal routing
Load balancer
↓
API gateway
↓
Service mesh
↓
Application service
↓
Auth/session store
The service that owns the session may not be the component terminating the original TLS connection.
\nIf the proof is too low in the stack, application teams may not be able to reliably consume it.
\nDBSC is more pragmatic because it operates closer to HTTP and application session semantics. The auth system that issues and refreshes cookies can also verify device proof.
\nThat is good architecture: put the security control where the ownership boundary actually exists.
\nAny device-binding mechanism immediately raises a privacy concern:
\n\n\nAre we creating a stable device identifier?
\n
If every website received the same hardware-backed public key, DBSC would become a tracking disaster.
\nSites could correlate users across origins and sessions.
\nA responsible design must avoid that.
\nDBSC's design uses session-specific keys rather than exposing a universal device identity. The goal is to bind this session to this device, not reveal a stable global hardware identity.
\nThe difference is huge.
\nGood design:
\nSite A session key ≠ Site B session key
Session 1 key ≠ Session 2 key
No global device identifier exposed
Bad design:
\nSame public key exposed everywhere
Device becomes trackable across the web
Security features that create persistent tracking surfaces will face resistance from browsers, users, regulators, and privacy engineers.
\nDBSC is interesting because it tries to improve session security without turning the browser into a device-identification oracle.
\nDBSC is strongest against off-device replay.
\nIt helps when the attacker flow is:
\nMalware steals cookie
↓
Attacker imports cookie into another browser/device
↓
Attacker tries to continue the session
DBSC breaks this flow because the attacker does not have the original device-bound private key needed to refresh the session.
\nIt also improves defender visibility.
\nIf attackers can no longer replay cookies from their own infrastructure, they may be forced to act from the compromised user's device. That makes endpoint detection, enterprise device management, browser telemetry, and cleanup more relevant.
\nIn security terms, DBSC converts a portable credential problem into a device-compromise problem.
\nThat is not perfect, but it is meaningfully better.
\nA serious engineering analysis must be clear about boundaries.
\nDBSC does not fully stop:
\nThis does not make DBSC weak. It makes its threat model precise.
\nDBSC is an anti-exfiltration and anti-replay control. It is not a complete endpoint integrity solution.
\nThe correct mental model is:
\nDBSC does not prove the user is safe.
DBSC proves the session can still access the original device-bound key.
That is still extremely valuable.
\nPeople may confuse DBSC with passkeys because both use public/private key cryptography and device-associated keys.
\nBut they solve different problems.
\nPasskey:
Can this user authenticate to this account?
DBSC:
Can this already-authenticated session still prove it belongs to the original device?
Passkeys protect the front door.
\nDBSC protects the session after the user has entered.
\nA strong modern authentication architecture should combine both:
\nPasskey login
+
Device-bound session renewal
+
Short-lived cookies
+
Risk-based step-up authentication
+
Session revocation
+
Endpoint posture and detection
That is the direction web authentication is moving: fewer reusable secrets and more contextual proof.
\nThe YouTube video framing around DBSC and password/session sharing is understandable.
\nIf session state is no longer portable across machines, then copying a logged-in browser profile or cookie jar to another device becomes less useful.
\nIn that sense, DBSC can make casual session sharing harder.
\nBut from a security engineering perspective, password sharing is not the deepest point.
\nThe deeper point is:
\nAttackers should not be able to cheaply replay stolen authenticated sessions at scale.
For consumer subscription services, DBSC may also help enforce that sessions remain tied to legitimate devices. But if DBSC is marketed only as anti-sharing, users may perceive it as restrictive DRM.
\nThe stronger and more accurate framing is session theft protection.
\nIf I were designing DBSC adoption for a large-scale company, I would not let every product team independently bolt it onto its own application.
\nI would centralize it in the authentication platform.
\nA mature architecture would look like this:
\nIdentity Provider / Auth Gateway
↓
DBSC registration service
↓
DBSC refresh service
↓
Session store / public key registry
↓
Risk engine
↓
Observability + security analytics
The registration service should:
\nThe refresh service is even more critical.
\nIt should:
\nA DBSC rollout without observability is dangerous.
\nTeams need dashboards for:
\nDBSC registration success rate
DBSC refresh success rate
Refresh latency
Proof validation failures
Fallback frequency
Unsupported browser/device percentage
Sessions still using long-lived bearer cookies
Suspicious repeated refresh failures
DBSC turns the refresh path into part of the authentication control plane. It must be designed with the same seriousness as login infrastructure.
\nA clean backend design treats DBSC as a session-hardening layer.
\n ┌────────────────────┐
│ User authenticates │
└─────────┬──────────┘
│
▼
┌───────────────────────┐
│ Auth service issues │
│ registration header │
└─────────┬─────────────┘
│
▼
┌───────────────────────┐
│ Browser creates key │
│ TPM/Secure Enclave │
└─────────┬─────────────┘
│
▼
┌───────────────────────┐
│ Registration endpoint │
│ stores public key │
└─────────┬─────────────┘
│
▼
┌───────────────────────┐
│ Short-lived cookie │
│ issued │
└─────────┬─────────────┘
│
▼
┌───────────────────────┐
│ Refresh endpoint │
│ verifies signed proof │
└───────────────────────┘
For high-scale systems, the refresh endpoint must be:
\nIf the refresh service goes down, users may appear logged out even if the main application is healthy.
\nThat makes DBSC infrastructure part of the critical path.
\nDBSC creates new security signals. The question is what the server should do with them.
\nFor example:
\nDBSC proof succeeds:
Continue session normally.
DBSC proof fails once:
Retry or gracefully degrade.
DBSC proof repeatedly fails:
Trigger step-up authentication.
DBSC proof fails from suspicious context:
Revoke session and alert.
DBSC unavailable on device:
Use shorter session lifetime and stronger risk checks.
This is where DBSC becomes more than protocol plumbing. It becomes part of adaptive access control.
\nA mature system should not treat every failure identically. Some failures are benign: browser updates, device migration, corrupted profile state, unsupported hardware, enterprise policy restrictions.
\nOther failures may be suspicious: repeated refresh attempts, impossible travel, mismatched device posture, or refresh attempts after known compromise.
\nThe policy layer matters as much as the cryptography.
\nThe real lesson from DBSC is bigger than Chrome.
\nFor decades, web security has over-weighted the login moment and under-weighted the session lifetime.
\nMost systems ask:
\nDid the user authenticate correctly at time T0?
Attackers increasingly operate after T0.
\nThey wait until the user is already authenticated, steal the resulting session, and bypass the expensive login security controls.
\nModern security needs to ask better questions:
\nShould this session still be trusted now?
Is it still on the original device?
Has the risk posture changed?
Can the client prove possession of the original key?
Should this action require step-up authentication?
DBSC is valuable because it moves web sessions toward continuous trust validation without constantly annoying the user.
\nThat is the future of authentication: not more prompts, but stronger invisible guarantees.
\nDBSC is not the death of cookies.
\nIt is the hardening of cookies.
\nIt does not replace MFA. It does not replace passkeys. It does not replace endpoint security. It does not prove the device is uncompromised.
\nBut it fixes one of the worst properties of web sessions:
\n\n\nA stolen cookie should not be a portable identity.
\n
By using device-bound cryptographic keys, ideally protected by TPM or similar hardware-backed mechanisms, DBSC makes session renewal depend on proof from the original device.
\nThe old model was:
\nTrust follows the cookie.
The DBSC model is:
\nTrust follows the cookie only while the original device can prove possession.
That is not a small implementation detail. That is a better trust boundary.
\nAnd in security architecture, better trust boundaries are how the game actually changes.
\n![\"Bloom](\"https://blogs.pranjalkumar.in/assets/images/probabilistic-data-structures-bloom-filters-explained%20by%20Pranjal%20Kumar%20-%20software%20engineer-94b8b0ef12c3f69d6dc226c2af08b46d.png\")
\n\nHere's a question that sounds impossible: can you check if an element is in a set of 1 billion items using only 1.2 GB of RAM, with lookups completing in constant time, and never producing a false negative? Oh, and the data structure is a single contiguous array of bits; no pointers, no allocations, no garbage collection pauses.
\nThe answer is yes. It's called a Bloom filter, and once you understand it, you'll see it everywhere; from the database engine under your app to Chrome's malicious URL checker to the network switches routing your packets.
\nThis post goes deep. Not \"here's a pretty diagram\" deep. \"I need to tune the false positive rate for a production system handling 400K QPS\" deep.
\n
Let's start with a confession: deterministic data structures are expensive.
\nA HashSet<String> holding 1 billion URLs? That's roughly:
You're looking at ~120 GB of RAM for a set membership test. And every single read might chase a pointer into some random heap location; cache miss city. Your L3 cache is crying.
\nProbabilistic data structures trade a small, bounded error rate for massive savings in space and time. They answer a subtly different question: instead of \"is X in the set?\" they answer \"is X definitely not in the set, or probably in the set?\"
\nThat sounds like a downgrade. But in practice? It's one of the most powerful engineering tradeoffs you can make.
\nBefore we go deep on Bloom filters, let me give you the lay of the land:
\n| Structure | What it answers | Error type | Space |
|---|---|---|---|
| Bloom filter | \"Is X in the set?\" | False positives (never false negatives) | ~10 bits per element for 1% FPR |
| Counting Bloom filter | Same, but supports deletion | Same | ~40 bits per element |
| Cuckoo filter | Same, supports deletion, often better FPR | False positives | ~12 bits per element for <1% FPR |
| HyperLogLog | \"How many distinct elements?\" | ±0.8% error | 12 KB (!) for billions of elements |
| Count-Min Sketch | \"How many times has X appeared?\" | Over-estimates only | O(1/ε × log(1/δ)) |
| Quotient filter | \"Is X in the set?\" (cache-friendly) | False positives | ~10-12 bits per element |
| XOR filter | \"Is X in the set?\" (immutable, faster) | False positives | ~9 bits per element |
Each one is beautiful in its own way. But Bloom filters are the gateway drug; once you grok them, the rest click into place.
\n\n\nFPR: False Positive Rate
\n
A Bloom filter is:
\nm bits, all initialized to 0k independent hash functions, each mapping an element to a position in [0, m)To add element x: compute h₁(x), h₂(x), ..., hₖ(x) and set those bit positions to 1.
To query element x: compute the same hashes. If all k positions are 1, answer \"probably yes.\" If any position is 0, answer \"definitely no.\"
That's it. That's the whole thing. No linked lists, no trees, no rebalancing, no resizing. Just bits and hashes.
\nThink about it: if we added element x, we set bits at positions h₁(x) through hₖ(x). We never unset bits. So when we query x later, those positions will still be 1. Always. The filter never forgets what it's seen.
But other elements might have also set those same positions. If elements a, b, and c collectively lit up the same bit positions that h₁(x) through hₖ(x) would need; then querying x returns \"probably yes\" even though x was never inserted. The bits don't remember who set them.
This is the tradeoff. And it's quantifiable.
\nI know, I know; \"just tell me what parameters to use.\" But if you don't understand the math, you can't tune these things in production. And you will need to tune them. Trust me.
\nFor a Bloom filter with m bits, k hash functions, and n inserted elements:
The probability a specific bit is still 0 after all insertions:
\nP(bit=0)=(1−m1)kn≈e−kn/m
\nThe false positive probability (all k bits are 1 for an element not in the set):
P(FP)=(1−e−kn/m)k
\nGiven m and n, the optimal k that minimizes false positive rate:
kopt=nmln2≈0.693⋅nm
\nThis is a beautiful result. It says: for a given bits-per-element ratio, there's a sweet spot for the number of hashes. Too few hashes → not enough discrimination. Too many hashes → you fill up the bit array too fast.
\nIf you want a false positive rate of p:
nm=−(ln2)2lnp≈−1.44⋅log2p
\nLet's make this concrete:
\n| Desired FP rate | Bits per element | Hash functions (k) |
|---|---|---|
| 10% | 4.8 | 3 |
| 1% | 9.6 | 7 |
| 0.1% | 14.4 | 10 |
| 0.01% | 19.2 | 13 |
| 0.001% | 24.0 | 17 |
The takeaway: each additional order of magnitude in accuracy costs ~4.8 bits per element. That's incredibly space-efficient. A 1% false positive Bloom filter uses ~1.2 bytes per element regardless of element size; whether you're storing 10-byte integers or 10 KB URLs.
\nThe textbook says \"use k independent hash functions.\" In practice, nobody does this. Computing k independent hashes is expensive, and you don't actually need independence.
Kirsch-Mitzenmacker optimization (2006): You can simulate k hash functions using only two hash functions:
gi(x)=h1(x)+i⋅h2(x)modm,i=0,1,...,k−1
\nThis gives the same asymptotic false positive rate as k truly independent hashes. In practice, people use:
Here's what this looks like in practice:
\nfn get_bit_positions(item: &[u8], num_hashes: u32, num_bits: u64) -> Vec<u64> {
let hash = murmurhash3_x64_128(item, 0);
let h1 = hash as u64;
let h2 = (hash >> 64) as u64;
(0..num_hashes)
.map(|i| {
h1.wrapping_add((i as u64).wrapping_mul(h2)) % num_bits
})
.collect()
}
Two hashes, k bit positions. Simple, fast, and proven correct.
The bit array seems trivial; just malloc(m / 8) bytes, right? In theory, yes. In practice, there are a few gotchas that separate a toy implementation from a production one:
// Setting a bit
void bloom_set(uint8_t *filter, uint64_t pos) {
filter[pos >> 3] |= (1 << (pos & 7));
}
// Testing a bit
bool bloom_test(uint8_t *filter, uint64_t pos) {
return filter[pos >> 3] & (1 << (pos & 7));
}
Cache line awareness: On modern CPUs, a cache line is 64 bytes (512 bits). If your k hash outputs are spread uniformly across a large filter, each query potentially touches k different cache lines. For a 1 GB filter with k=7, you're looking at 7 cache misses per lookup; at ~100 ns each, that's 700 ns per query.
Blocked Bloom filters solve this: partition the bit array into cache-line-sized blocks. Hash the element to pick a block, then do all k hash probes within that block. You lose a bit of theoretical FP optimality, but you gain cache-friendliness that more than compensates in practice.
Traditional: k probes scattered across m bits → k cache misses
Blocked: 1 hash to select block + k probes within 512 bits → 1 cache miss
I've seen this take query time from ~500 ns to ~50 ns on real hardware. A 10x improvement just from thinking about cache lines.
\nFor filters that don't fit in RAM (or that you want to persist across process restarts), mmap is your friend:
int fd = open(\"bloom.dat\", O_RDWR | O_CREAT, 0644);
ftruncate(fd, filter_size_bytes);
uint8_t *filter = mmap(NULL, filter_size_bytes,
PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
The OS handles paging; hot parts stay in RAM, cold parts stay on disk. This is exactly how databases like RocksDB implement their Bloom filters: they're part of the SST file format and get mmap'd when the file is opened.
This isn't academic. Bloom filters are everywhere in production systems. Let me walk you through the ones I find most interesting.
\nThis is probably the most impactful use. In an LSM-tree, data lives in sorted immutable files (SSTables) across multiple levels. A point lookup (GET key) might need to check every SSTable for the key.
Without Bloom filters: reading a non-existent key requires checking all SSTables → potentially dozens of disk reads → unacceptable latency.
\nWith Bloom filters: each SSTable has an associated Bloom filter (stored in the file's metadata block). Before reading the SSTable's data blocks, check the filter. If it says \"definitely not here\" → skip the file entirely. Zero disk I/O for that SSTable.
\nImpact: RocksDB's documentation states that Bloom filters reduce read amplification by 10-100x for point lookups. Cassandra uses them on every SSTable, and a misconfigured bloom_filter_fp_chance (their tuning knob) is one of the most common causes of read latency spikes I've seen in production.
# Cassandra table options
CREATE TABLE users (
...
) WITH bloom_filter_fp_chance = 0.01; -- 1% FP rate, ~10 bits/key
Changing this from 0.01 to 0.001 doubles memory usage but reduces unnecessary disk reads by 10x. Whether that's worth it depends entirely on your read/write ratio.
\nYour browser maintains a local Bloom filter of ~4 million malicious URLs. When you navigate to a page, Chrome checks the URL against this filter locally; no network request needed. If the filter says \"definitely not malicious\" (which is >99.99% of the time), you proceed instantly.
\nIf the filter says \"maybe malicious,\" Chrome then makes an API call to Google's Safe Browsing service for a definitive answer. The Bloom filter turned a network round-trip on every single navigation into a network round-trip on 0.01% of navigations.
\nWhen a CDN edge node receives a request, it needs to know if the content is cached somewhere in the cluster before fetching from origin. The nodes share Bloom filter summaries of their cache contents. A request arrives → check all peer filters → if any say \"maybe here\" → ask that peer → otherwise fetch from origin.
\nThis is why CDNs can maintain hit rates >95% even with object counts in the hundreds of billions.
\nHardware routers use Bloom filters in TCAM (Ternary Content-Addressable Memory) for longest-prefix match in routing tables. At 100 Gbps line rate, you have ~6.7 nanoseconds per packet. A Bloom filter in SRAM gives you constant-time membership checks at wire speed.
\nLightweight Bitcoin wallets use Bloom filters to request only relevant transactions from full nodes without revealing which addresses they own (BIP 37). The wallet sends a Bloom filter of its addresses to the full node, which then only forwards transactions matching the filter. Privacy through false positives; the node can't tell which matches are real and which are noise.
\nThe original 1970 Bloom filter paper by Burton Bloom was literally about spell checking. Store a dictionary in a Bloom filter. Check each word in the document. If the filter says \"not in dictionary\" → it's definitely misspelled. If it says \"in dictionary\" → it's probably spelled correctly (and you can do a more expensive exact check if needed).
\nThe problem: standard Bloom filters don't support deletion. You can't unset a bit because you don't know if other elements also need that bit.
\nThe solution: replace each bit with a counter (typically 4 bits). Insert increments, delete decrements. A position is \"set\" if its counter > 0.
\nStandard: [0][1][0][1][1][0][0][1] ← 1 bit per position
Counting: [0][3][0][1][2][0][0][1] ← 4 bits per position
Tradeoff: 4x the space (4 bits instead of 1 per position). And there's a counter overflow risk; if more than 15 elements hash to the same position, the 4-bit counter wraps. In practice, the probability of this is negligible for properly sized filters.
\nWhen to use: when your set changes over time and you need to remove elements. Think: \"which users are currently online?\"; users go offline, and you need to remove them.
\nI think cuckoo filters are underrated. They were introduced by Fan et al. in 2014 and in many practical scenarios they're just... better.
\nHow they work: instead of k bit positions, store a fingerprint (compact hash) of each element in one of two candidate bucket positions (using cuckoo hashing; hence the name).
Bucket 0: [fp₁][fp₂][ ][ ] ← 4 slots per bucket
Bucket 1: [fp₃][ ][ ][ ]
Bucket 2: [fp₄][fp₅][fp₆][ ]
...
To insert: compute two candidate buckets via h₁(x) and h₂(x) = h₁(x) ⊕ hash(fingerprint(x)). If either has space, place the fingerprint. If both are full, evict an existing fingerprint to its alternative bucket (cuckoo displacement). Repeat until placed or max iterations reached.
To query: check both candidate buckets for the fingerprint.
\nAdvantages over Bloom filters:
\nDisadvantages:
\nHere's the crossover point I use as a rule of thumb:
\nStandard Bloom filters require you to know n (number of elements) upfront. What if you don't? Scalable Bloom filters (Almeida et al., 2007) solve this:
This is what Redis uses in its BF.ADD/BF.EXISTS implementation; it auto-scales without requiring you to pre-declare the capacity.
If your set is static (built once, queried many times), XOR filters are the new hotness:
\nThe downside: you can't add elements after construction. It's an immutable snapshot. Perfect for SSTable-level filters in databases (you build the filter when the SSTable is written and never modify it), but useless for streaming membership tracking.
\nHere's my mental model for sizing a production Bloom filter:
\nimport math
def bloom_filter_size(n_elements, fp_rate):
\"\"\"Calculate optimal Bloom filter parameters.\"\"\"
m = -n_elements * math.log(fp_rate) / (math.log(2) ** 2)
k = (m / n_elements) * math.log(2)
return {
'bits': int(math.ceil(m)),
'bytes': int(math.ceil(m / 8)),
'megabytes': m / 8 / 1024 / 1024,
'hash_functions': int(math.ceil(k)),
'bits_per_element': m / n_elements
}
# Example: 100 million URLs, 0.1% false positive rate
params = bloom_filter_size(100_000_000, 0.001)
# → ~172 MB, 10 hash functions, 14.4 bits/element
Rule of thumb: 10 bits per element gives you 1% FP. Each additional 5 bits per element gives you ~10x better FP rate.
\nYour Bloom filter's actual FP rate drifts as you insert more elements than planned. Monitor it:
\n# Approximate current FP rate based on fill ratio
def estimated_fp_rate(m_bits, k_hashes, n_inserted):
return (1 - math.exp(-k_hashes * n_inserted / m_bits)) ** k_hashes
Set an alert when the estimated FP rate exceeds your threshold. When it fires, you need to resize; which for a standard Bloom filter means building a new, larger filter and reinserting everything. Plan for this.
\nBloom filters are just byte arrays. They serialize trivially:
\nBut transferring a 100 MB filter over the network on every update is wasteful. Two tricks:
\nDelta compression: if you're syncing filters between nodes, XOR the old and new filter. The delta is sparse (mostly zeros) and compresses beautifully with LZ4 or zstd.
\nPartitioned filters: split the filter into segments. Only transfer segments that changed since the last sync. This is how distributed systems like Cassandra share filter updates during anti-entropy repair.
\nBloom filters are naturally read-safe without locks; reading bits is idempotent. For concurrent writes, you have options:
\n__sync_fetch_and_or(&filter[byte_pos], bit_mask); lock-free, correct, minimal overheadIn my experience, atomic bit-set is the sweet spot for most workloads. The false positive rate increases very slightly due to non-atomic multi-bit operations (a query might see a partially-inserted element), but in practice this is within noise.
\nA few years back, I was working on a system that processed event deduplication at ~400K events/second. Each event had a UUID, and we needed to detect duplicates within a 24-hour window.
\nThe naive approach: a Redis SET with 24-hour TTL. At 400K/sec × 86400 seconds × ~40 bytes per UUID = ~1.4 TB of Redis. At AWS ElastiCache pricing, that's... not cheap.
\nThe Bloom filter approach:
\nOn the rare false positive (0.01% × 24 filters ≈ 0.24% worst case), we'd do a more expensive exact check against a backing store. In practice, the composite FP rate was much lower because events rarely hash identically across multiple hourly filters.
\nResult: 17x less memory, easily fit on a few commodity machines, and the latency per dedup check went from ~1 ms (Redis network round trip) to ~200 ns (in-process Bloom filter). The infra cost savings alone justified the engineering effort in the first sprint.
\n1. Not accounting for growth
\n\"We have 10 million users, so we'll size for 10 million.\"
\nSix months later, you have 25 million users. Your 1% FP rate is now 15%, and your database is getting hammered with false-positive-triggered lookups. Always size for 2-3x your expected growth, or use scalable Bloom filters.
\n2. Choosing k based on vibes
\nI've seen codebases that hardcode k=3 or k=7 without any justification. Use the formula: k = (m/n) × ln(2). If your bits-per-element ratio is 10, optimal k is 7. If it's 20, optimal k is 14. Getting k wrong by even 2-3 costs you 2-5x in FP rate for free.
3. Using a bad hash function
\nhashCode() in Java is a 32-bit hash with terrible distribution for Bloom filter use. I once saw a production Bloom filter using String.hashCode() with k=7; but since the hash is only 32 bits, the entropy was exhausted after ~4 hash functions. The last 3 were essentially correlated noise. FP rate was 10x worse than predicted.
Use a 128-bit hash (MurmurHash3-128 or xxHash-128) and split it into two 64-bit values for the Kirsch-Mitzenmacker trick. This gives you effectively unlimited k without entropy exhaustion.
\n4. Ignoring the \"definitely not\" guarantee in application logic
\nThe whole point of a Bloom filter is the asymmetry: \"no\" means NO. \"Yes\" means MAYBE. I've seen code that treats both answers as probabilistic:
\n# WRONG: treating \"not found\" as probabilistic
if not bloom.might_contain(key):
result = \"probably not in set\" # NO! It's DEFINITELY not in set!
Build your system logic around the certainty of negatives. That's where the value is.
\n5. Not versioning your hash function
\nIf you change the hash function (or its seed), every existing filter becomes invalid. All queries will return nonsense. Version your filters with a header that includes the hash algorithm and seed. This sounds obvious until you're doing a zero-downtime migration and half your nodes are running the old hash and half the new one.
\nHere's what I want you to walk away with: Bloom filters aren't just a clever trick. They represent a fundamental insight about information theory.
\nShannon taught us that information has an irreducible cost. If you need to distinguish N items with error rate ε, you need at minimum log₂(1/ε) bits per item. A Bloom filter uses 1.44 × log₂(1/ε) bits per item; only 44% above the information-theoretic minimum. That's astounding for a structure this simple.
The reason they're everywhere in systems engineering isn't because engineers love clever algorithms. It's because the tradeoff they offer; certainty of absence in exchange for ambiguity of presence; maps perfectly onto how distributed systems actually work:
\nIn all these cases, the certain negative is more valuable than the uncertain positive. Bloom filters are the mathematical embodiment of this asymmetry.
\nBloom filter = bit array + k hashes
Insert(x): set bits at h₁(x), h₂(x), ..., hₖ(x)
Query(x): if ALL bits at h₁(x)...hₖ(x) are 1 → \"maybe\"
if ANY bit is 0 → \"definitely not\"
Size it: m = -n·ln(p) / (ln2)²
Tune it: k = (m/n)·ln(2)
Hack it: gᵢ(x) = h₁(x) + i·h₂(x) mod m (only need 2 hashes)
Use it when: \"definitely not\" is more valuable than \"definitely yes\"
The next time someone proposes a HashMap for a membership check on a billion items, you'll know there's a better way. And now you know exactly how to size it, tune it, and keep it healthy in production.
Bloom, Burton H. \"Space/time trade-offs in hash coding with allowable errors.\" Communications of the ACM 13.7 (1970): 422-426. — Still worth reading. The paper is 4 pages long and changed systems engineering forever.
\n![\"Diagram](\"https://blogs.pranjalkumar.in/assets/images/how-internet-works-explained%20by%20Pranjal%20Kumar%20-%20software%20engineer-3a40dbfe9cdd13f7e81698a1ecc5d34e.png\")
\n\nEvery time I read a \"how the internet works\" article, I cringe - they all stop at DNS → TCP → HTTP → HTML. That's like explaining a car engine by saying \"fuel goes in, wheels spin.\" Super helpful when the engine is on fire.
\nThis post is what I wish someone had handed me when I was staring at a waterfall chart at 3 AM, wondering why a page took 4 seconds to load on a 1 Gbps connection. We're going to trace every system boundary crossing - from the electrical signal your keyboard fires to the GPU draw call that lights up the final pixel.
\n
You press Enter. That's a physical key-switch completing a circuit on a matrix scan. Your keyboard's microcontroller detects the state change, debounces it (~5 ms), and sends a USB HID scancode (0x28 for Enter) over the USB bus.
\nThe host controller raises a hardware interrupt (IRQ). The kernel's USB HID driver picks it up, translates the scancode to a virtual keycode, and enqueues a KEY_ENTER event into the input subsystem (/dev/input/eventN on Linux, IOKit HID on macOS, Raw Input on Windows).
The window manager dispatches this as a keydown event to your browser. The browser's UI thread captures it, recognizes you're in the URL bar, and kicks off navigation. Chrome's NavigationController fires and the network stack starts work before the UI thread finishes its current animation frame. They're racing against your perception of speed.
Before we go further, here's Chrome's process model - it shapes everything that follows:
\n┌─────────────────────────────────────────────────────────┐
│ Browser Process (1 per Chrome instance) │
│ - UI thread (address bar, tabs, bookmarks) │
│ - Network Service (DNS, TCP, TLS, HTTP - all of it) │
│ - Storage Service (cookies, cache, IndexedDB) │
├─────────────────────────────────────────────────────────┤
│ Renderer Process (1 per site / site isolation) │
│ - Main thread (DOM, CSSOM, JS, Layout, Paint) │
│ - Compositor thread (layer tree, tile management) │
│ - Raster threads (pixel generation via Skia) │
├─────────────────────────────────────────────────────────┤
│ GPU Process (1 shared across all renderers) │
│ - Composites layers from all renderer processes │
│ - Issues GL/Vulkan/Metal draw calls │
└─────────────────────────────────────────────────────────┘
When you press Enter, the Browser Process owns the entire navigation until the first response bytes arrive. Only then does it spin up (or reuse) a Renderer Process and hand off the byte stream. This is a security boundary - the renderer is sandboxed and can't touch the network directly. Every fetch goes through the Browser Process via IPC.
\nThis also explains site isolation: each cross-origin site gets its own renderer process with its own V8 isolate. A malicious script on evil.com literally cannot read memory from bank.com's renderer because they're separate OS processes.
Now the browser has a string: https://example.com/path?q=1. You'd think parsing a URL is trivial - and you'd be wrong. I've seen production bugs from URL parsing edge cases.
The browser runs it through URL parsing (RFC 3986): normalizes the scheme, Punycode-encodes IDN labels (münchen.de → xn--mnchen-3ya.de), percent-encodes illegal characters, and resolves the default port (443 for HTTPS).
Then, before any network I/O:
\nHSTS Preload check. Chrome ships a static preload list baked into the binary - domains that must always use HTTPS. If the domain is on this list and you typed http://, the browser internally redirects to https:// without sending a single plaintext byte to the network.
Status Code: 307 Internal Redirect
Non-Authoritative-Reason: HSTS
There's also the dynamic HSTS store: if the server previously sent a Strict-Transport-Security header, Chrome stored that policy. It's persisted at ~/.config/google-chrome/Default/TransportSecurity - just a JSON file. I once debugged a redirect loop that turned out to be a stale HSTS entry from a dev environment. Knowing this file exists saves you hours.
DNS is one of those things everyone thinks they understand. \"It's like a phone book for the internet!\" Sure. And a CPU is like a calculator. Technically correct, practically useless for debugging.
\nBefore anything hits the network, the browser goes through a caching chain:
\nchrome://net-internals/#dns) with per-record TTL. Negative caches (NXDOMAIN) are stored too. Chrome will happily cache a failure for minutes while dig shows the record is fine.systemd-resolved on Linux, mDNSResponder on macOS./etc/hosts - still checked. I've seen senior engineers spend hours debugging DNS issues that turned out to be a forgotten entry from a hackathon two years ago.dnsmasq or equivalent.If all miss, we go to the recursive resolver (let's say 8.8.8.8).
The stub resolver sends a UDP datagram (port 53). The recursive resolver performs the actual recursion:
\nStub → Recursive Resolver (8.8.8.8)
└─→ Root server → \"Try .com TLD at 192.5.6.30\"
└─→ .com TLD server → \"Try example.com NS at 93.184.216.34\"
└─→ Authoritative NS → \"A record: 93.184.215.14, TTL: 86400\"
In theory, that's 3 round trips. In practice, the recursive resolver has cached the root and TLD responses (hit billions of times daily), so you usually pay for only the final authoritative query. But on a cold resolver, you'll feel all three hops - that's why the first page load after changing DNS settings feels sluggish.
\nA few things worth knowing about DNS as it exists today:
\nDNSSEC validates the chain of trust from root → TLD → domain using signed records. A failed validation returns SERVFAIL - hard error, not a redirect.
DNS-over-HTTPS (DoH) encrypts queries as HTTP/2 POSTs to providers like https://dns.google/dns-query. Your ISP can't see what you're resolving anymore, but it requires an existing TLS connection to the DoH provider (bootstrapped via hardcoded IPs).
Happy Eyeballs (RFC 8305) - my favorite name in networking. If both IPv4 and IPv6 records come back, the browser starts IPv6 first, gives it a 250ms head start, then races it against IPv4. Whichever SYN-ACK arrives first wins. Users shouldn't suffer because their ISP's IPv6 is half-broken.
\nTCP was designed in the 1970s and still causes performance headaches its creators never imagined.
\nClient → Server: SYN (seq=x), Window Scale, MSS, SACK Permitted, Timestamps
Server → Client: SYN-ACK (seq=y, ack=x+1), Window Scale, MSS, SACK Permitted
Client → Server: ACK (ack=y+1)
Cost: 1 RTT. For a US West → Europe server, that's ~150 ms before any data flows.
\nThe options negotiated during SYN matter more than people think:
\n| Option | What it does |
|---|---|
| Window Scale | Allows receive windows > 64KB (max 1 GB window) |
| MSS | Maximum Segment Size - typically 1460 bytes for Ethernet |
| SACK | Selective Acknowledgments - smarter retransmission |
| ECN | Explicit Congestion Notification - routers signal congestion without dropping packets |
After the handshake, the kernel's congestion control algorithm dictates how fast data flows. This will fundamentally change how you think about web performance.
\nIf your critical CSS is 30 KB, you're paying an extra RTT - maybe 100+ ms on a cross-continent connection - just because you shipped 15 KB too much CSS. This is why the \"14 KB rule\" exists, and why every byte in your critical path matters far more than raw bandwidth suggests.
\nBrowsers maintain up to 6 TCP connections per origin (HTTP/1.1). With HTTP/2, everything multiplexes over a single connection, amortizing the handshake cost across all requests.
\nI've spent more hours staring at TLS handshake failures than I'd like to admit. TLS 1.3 collapsed the handshake from 2 RTTs (TLS 1.2) to 1 RTT by doing key exchange and authentication in the same flight:
\nClient → Server: ClientHello
+ key_share (X25519 ECDH public key)
+ server_name (SNI: \"example.com\")
+ ALPN (h2, http/1.1)
Server → Client: ServerHello + key_share
{EncryptedExtensions}
{Certificate}
{CertificateVerify}
{Finished}
Client → Server: {Finished}
[Application Data starts HERE]
Both sides derive traffic keys from the shared ECDH secret via HKDF. All data is now encrypted with AES-256-GCM or ChaCha20-Poly1305.
\nHere's a dirty secret: the server_name extension (SNI) is sent in plaintext in the ClientHello. Any network observer - your ISP, a corporate proxy, a nation-state firewall - can see which domain you're connecting to, even though the content is encrypted. Encrypted Client Hello (ECH) fixes this by encrypting the ClientHello under a public key from the domain's DNS HTTPS record.
If the client previously connected to this server, it may have cached a PSK from the server's NewSessionTicket. On reconnection, it sends application data with the ClientHello - zero round trips for TLS.
The catch: 0-RTT data is replayable. An attacker who captures the ClientHello can replay it. This is why only idempotent requests (GET) go in 0-RTT, and servers need anti-replay mechanisms.
\nThe chain is validated: leaf cert → intermediate(s) → root CA in the OS trust store. Chrome uses CRLSets (a compressed revocation list pushed via component updates) instead of slow online OCSP checks, and requires Certificate Transparency SCTs to prevent rogue CA issuance.
\nWe finally have an encrypted pipe. Now let's shove data through it.
\nHTTP/2's killer feature: multiple request/response pairs share a single TCP connection as independent streams. No more 6-connection limit.
\nBut here's the cruel irony - TCP delivers bytes in order. If one segment is lost, all streams stall waiting for retransmission. Your CSS stream is held hostage by a lost packet from the JavaScript stream. This head-of-line blocking drove the creation of QUIC.
\nHPACK compresses headers using a static table of common values plus a per-connection dynamic table. The second request to the same origin typically sends ~20 bytes of headers instead of ~800 bytes.
\nHTTP/3 runs over QUIC (RFC 9000) - reliable delivery rebuilt on top of UDP. When I first heard this, I thought it was insane. Then I read the design and realized it was brilliant:
\n| Protocol | RTTs to first byte |
|---|---|
| HTTP/1.1 + TLS 1.2 | 3 |
| HTTP/2 + TLS 1.3 | 2 |
| HTTP/3 (QUIC) | 1 |
| HTTP/3 0-RTT | 0 |
Browsers discover HTTP/3 via the Alt-Svc header. The first request always goes over TCP; subsequent requests migrate to QUIC if it works.
What actually goes over the wire - not the textbook version:
\nGET /path?q=1 HTTP/2
:authority: example.com
:scheme: https
accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
accept-encoding: gzip, br, zstd
sec-ch-ua: \"Chromium\";v=\"128\"
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: none
The Sec-Fetch-* headers are criminally underused. They tell the server the context of the request - and they're unforgeable (the browser sets them, not JavaScript). If you're building an API and not checking these, you're missing a free layer of CSRF defense.
\nThe request hits a load balancer (Envoy, nginx), gets routed to an app server, which generates the response:
\nHTTP/2 200
content-type: text/html; charset=utf-8
content-encoding: br
cache-control: public, max-age=3600, stale-while-revalidate=86400
strict-transport-security: max-age=63072000; includeSubDomains; preload
content-security-policy: default-src 'self'; script-src 'self' 'nonce-r4nd0m';
The body is Brotli-compressed. The browser starts decompressing as chunks arrive - it doesn't wait for the full response. The HTML parser starts building the DOM while the last bytes are still in flight.
\nThis is where most articles wave their hands and say \"the browser renders the page.\" It's not one step. It's an absurdly complex pipeline.
\nWhile the main HTML parser runs on the main thread, Chrome fires up a preload scanner on a separate thread. It scans ahead looking for resource URLs (<link>, <script src>, <img>) and kicks off network requests before the main parser reaches those nodes. If you bury a critical CSS <link> below a huge inline script, you've negated this optimization. I've seen this cost 500+ ms.
Network bytes → Content decoding (streaming)
→ Character encoding detection
→ Tokenizer (state machine per HTML5 spec)
→ Tree builder
→ DOM Tree
The parser blocks on <script> tags without async or defer - pauses DOM construction entirely until the script is fetched, compiled, and executed. Every synchronous script in your <head> is a toll booth on the highway to first paint.
A <link rel=\"stylesheet\"> blocks rendering (not parsing) - the browser won't paint until CSSOM is ready, because painting without styles causes a flash of unstyled content.
All stylesheets merge into a CSSOM. The DOM and CSSOM combine into a layout tree - elements with display: none excluded, pseudo-elements added, computed styles resolved.
The engine walks the tree and computes geometry. Block layout, inline layout, flex/grid constraint solving. This is where layout thrashing kills you - reading a layout property after modifying the DOM forces a synchronous reflow:
\n// BAD: forces layout on every iteration
for (let i = 0; i < 1000; i++) {
el.style.width = el.offsetWidth + 1 + 'px';
}
After layout, Chrome records display items (draw commands). Then the page is split into layers (elements with will-change: transform, 3D transforms, <video>, <canvas>). Each layer is divided into tiles and rasterized by GPU threads into textures. Finally, the GPU process composites all layers together and submits the framebuffer.
On a 60 Hz display, this entire pipeline must complete in 16.67 ms. On 120 Hz, 8.33 ms. You have 8 milliseconds from a DOM change to photons. That's why UI frameworks obsess over minimizing re-renders.
\nWhen the parser hits a <script>, V8 takes over:
Lazy parsing: V8 only fully parses functions when they're called, not declared. If you ship a 2 MB bundle but only execute 200 KB on first load, V8 won't waste time parsing the rest upfront. This is why code splitting works.
\nTiered compilation:
\nCold code: Ignition (interpreter)
Warm code: Maglev (mid-tier compiler, ~10x faster)
Hot code: Turbofan (full optimizing compiler, ~100x faster)
Turbofan does type specialization, inlining, escape analysis, and dead code elimination based on runtime feedback from Inline Caches.
\nDeoptimization is the trap: if Turbofan's type assumptions are violated, it bails out back to Ignition. I've seen this cause performance cliffs in production - code runs fast for an hour, then one edge-case input triggers a deopt and that function is 100x slower until re-optimized. This is why polymorphic call sites kill performance, and why TypeScript helps the JIT too.
\nYour data needs to physically get from here to there.
\nARP resolution: Before the first SYN leaves your machine, the kernel needs the MAC address of your router. It checks the ARP cache; if missing, broadcasts \"who has 192.168.1.1?\" and caches the reply.
\nIP routing: The kernel's routing table determines the next hop. The packet gets encapsulated: [Ethernet frame [IP packet [TCP segment [TLS record [HTTP/2 frame [payload]]]]]]
The physical journey: NIC → Ethernet/Wi-Fi → router → ISP backbone (fiber, DWDM links) → Internet Exchange Point → destination CDN/ISP → server NIC.
\nLight in fiber travels at ~200,000 km/s. NY→London (5,500 km) is 27.5 ms one-way at the speed of light - real-world is ~35-40 ms due to routing and signal regeneration. No amount of engineering can beat this. Physics doesn't care about your SLA. This is why CDNs exist.
\nFor major sites, you don't hit the origin server. A CDN (Cloudflare, Fastly, Akamai) terminates your TLS connection at an edge PoP within ~10 ms of your location. The server you're talking to might be in a rack in your city.
\nThe edge checks its cache, serves on hit (TTFB ≈ 10-30 ms), or fetches from origin on miss. CDNs use tiered caching - edge PoP → regional shield → origin - so most misses never reach your actual servers.
\nEdge compute is changing the game: Cloudflare Workers, Fastly Compute, and Lambda@Edge let you run code at the edge. Authentication, A/B testing, SSR - no round trip to origin.
\n103 Early Hints is underused magic: the CDN sends preload hints while waiting for the origin response, so the browser starts fetching CSS/JS before the HTML even arrives.
\nIf the site has a service worker registered, the request may never hit the network. Service workers intercept every request and can serve from cache, synthesize responses from IndexedDB, or implement any caching strategy you want:
\nself.addEventListener('fetch', (event) => {
event.respondWith(
caches.match(event.request).then((cached) => {
if (cached) return cached;
return fetch(event.request).then((response) => {
const clone = response.clone();
caches.open('v1').then((cache) => cache.put(event.request, clone));
return response;
});
})
);
});
The cost: if the SW thread is idle (~30s of inactivity), the browser must spin it up (50-100ms on mobile). Chrome's navigation preload fires the network request simultaneously while waking the SW, so you don't pay that penalty on the critical path.
\nHere's a real navigation to https://example.com from a cold start:
| Phase | Time (typical) | Notes |
|---|---|---|
| Keyboard → browser | < 1 ms | Hardware interrupt |
| URL parse + HSTS | < 1 ms | In-memory |
| DNS resolution | 20-150 ms | Depends on cache state |
| TCP handshake | 10-150 ms | 1 RTT |
| TLS 1.3 handshake | 10-150 ms | 1 RTT (overlaps with TCP on QUIC) |
| Service worker | 0-100 ms | If registered; may skip network |
| HTTP request → first byte | 10-500 ms | Server processing |
| HTML download | 10-200 ms | Size + bandwidth |
| Critical CSS/JS fetch | 0-200 ms | Parallel, preload scanner |
| DOM + CSSOM + Layout + Paint | 15-150 ms | Complexity dependent |
| Total (cold) | ~100-1500 ms | |
| Total (warm, 0-RTT) | ~50-200 ms |
The patterns that cause real outages:
\nSlow DNS - dns_start to dns_end > 100 ms in DevTools Timing. Fix: <link rel=\"dns-prefetch\">, fast resolver.
TCP slow start penalty - first load slow, subsequent fast. Your critical path exceeds the 14.6 KB initial congestion window. Fix: keep HTML + critical CSS under 14 KB compressed.
\nTLS chain too long - extra RTT if the certificate message exceeds initial cwnd. Fix: short chains (leaf + 1 intermediate), ECDSA certs.
\nRender-blocking resources - gap between TTFB and First Paint. Fix: async/defer scripts, inline critical CSS.
Layout thrashing - long \"Layout\" blocks in Performance panel. Fix: batch reads/writes, requestAnimationFrame, CSS containment.
Keyboard IRQ → USB HID → OS input → Browser Process
→ URL parse → HSTS → DNS (cache chain → recursive resolution)
→ TCP SYN/ACK → TLS 1.3 key exchange
→ Service Worker → HTTP/2 request → CDN/origin → Response stream
→ Preload scan → DOM + CSSOM → Layout → Paint → Composite → GPU → Pixels
Every arrow is a potential latency cliff. Every one is debuggable. And now you know where to look.
\nThe internet is not magic. It's layers of engineering held together by RFCs, backwards compatibility, and the stubbornness of protocol designers who refuse to accept \"good enough.\"
\nNext time your page loads slowly, open DevTools. Look at the waterfall. Ask: which layer is bleeding time? DNS? TCP slow start? Render-blocking script? Layout thrash? You now have the mental model to answer that.
\nYour scalpels: Chrome DevTools Network panel (timing breakdown), Performance panel (frame analysis), chrome://net-internals (socket pools, DNS cache, QUIC sessions), and WebPageTest (real-device global testing).
![\"AWS](\"https://blogs.pranjalkumar.in/assets/images/aws-s3-internals-explained%20by%20Pranjal%20Kumar%20-%20software%20engineer-29252d109b924c88897631cc47096fa0.png\")
\nWhen people first hear about Amazon S3, they usually think of it as “cloud file storage.”
That description is convenient, but architecturally misleading.
\nS3 is not a giant hard drive in the sky. It is not a shared folder. It is not a filesystem with infinite space. And it is almost certainly not just a pile of premium SSDs waiting to serve your objects.
\nA better mental model is this:
\n\n\nS3 is a massive distributed object storage system that converts unreliable commodity hardware into durable, highly available, high-throughput storage through software, redundancy, metadata systems, erasure coding, checksums, repair loops, and extreme parallelism.
\n
The magic of S3 is not that AWS buys the fastest discs.
\nThe magic is that AWS operates so many discs, across so many failure domains, with so much automation, that the system behaves like something far more powerful than the sum of its parts.
\nOr, put more simply:
\n\n\nYou do not buy the fastest disk. You buy so many slow ones that they behave like a supercomputer.
\n
That sentence captures the soul of S3.
\nBut the deeper question is:
\n\n\nHow does a system made of ordinary failure-prone hardware deliver extraordinary reliability?
\n
That is where S3 becomes fascinating.
\nMany engineers assume S3 must be powered primarily by SSDs because it feels cloud-native, fast, and infinitely scalable.
\nBut S3’s economics are fundamentally different from a low-latency database or an NVMe-backed transactional system.
\nSSDs are excellent when you need low-latency random I/O and high IOPS from a small number of devices.
\nHDDs are excellent when you need massive capacity at low cost per terabyte.
\nS3 is primarily an object storage system optimized for:
\nIt is not designed as a single-digit millisecond random-mutation storage engine for tiny records.
\nThe tradeoff looks like this:
\nSSD:
Excellent random I/O
Higher IOPS per device
Lower latency
Higher cost per TB
HDD:
Lower random IOPS
Great sequential throughput
Much lower cost per TB
Excellent for massive object storage when parallelized
S3’s brilliance is that it does not try to make one disk fast.
\nIt makes thousands or millions of discs cooperate.
\nThat is a completely different optimization strategy.
\nIf you store a large object on one disk, your read throughput is bounded by that disk.
\nBut if you split the object into many pieces and distribute those pieces across many discs, servers, racks, and Availability Zones, you can read many fragments in parallel.
\nConceptually:
\nLarge object
↓
Split into shards
↓
Distribute across many drives
↓
Read shards in parallel
↓
Validate and reassemble object
This is why the torrent analogy is useful.
\nIn BitTorrent, a file downloads faster because many peers provide different pieces simultaneously. S3’s internal design is not literally BitTorrent, but the high-level performance idea is similar:
\n\n\nAggregate throughput comes from parallelism.
\n
One HDD may only provide modest IOPS.
\nBut thousands of HDDs serving predictable sequential ranges in parallel can provide enormous aggregate throughput.
\nThis is one of the most important distributed-systems lessons:
\n\n\nAt cloud scale, the performance unit is not the disk. It is the fleet.
\n
A single disk is slow.
\nA coordinated fleet of discs can behave like a storage supercomputer.
\nA filesystem gives you directories, in-place mutation, file locking, append semantics, fine-grained metadata operations, and often POSIX-like behavior.
\nS3 gives you something different:
\nBucket + key → object
An object is not just bytes. It can include:
\nThis object model is one reason S3 scales so well.
\nIt avoids many of the hardest problems in distributed filesystems:
\nInstead of pretending to be a traditional disk, S3 exposes a simpler abstraction:
\nPUT object
GET object
DELETE object
LIST objects
That simplicity is not a limitation.
\nIt is a scalability feature.
\nThe less the system promises at the API layer, the more freedom it has internally to shard, replicate, encode, route, repair, migrate, and optimize.
\nMost people think S3 is about storing bytes.
\nAt scale, storing bytes is only half the problem.
\nThe harder question is:
\n\n\nGiven this bucket and key, where are the object fragments, what version is current, what policies apply, what encryption state exists, and what should the system return right now?
\n
That is metadata.
\nFor every GET, PUT, DELETE, LIST, lifecycle transition, replication event, or restore request, S3 needs to coordinate metadata.
\nA simplified S3 read path looks like this:
\nClient sends GET bucket/key
↓
Front-end request router receives request
↓
Authentication and policy checks happen
↓
Metadata/index layer resolves object location/version
↓
Storage fleet retrieves required data fragments
↓
Fragments are validated and assembled
↓
Object is streamed back to client
This means S3 is not just a storage fleet.
\nIt is also a massive distributed metadata system.
\nAnd metadata systems are where distributed storage usually becomes difficult.
\nA system can have petabytes or exabytes of raw capacity, but if its metadata layer cannot answer location, version, permission, and lifecycle questions quickly and correctly, the storage system collapses operationally.
\nThe hidden challenge is not only:
\nWhere are the bytes?
It is:
\nWhat are the correct bytes for this request at this exact moment?
The beginner view of durability is:
\n\n\n“Store multiple copies.”
\n
The senior-engineer view is:
\n\n\n“Assume discs fail, servers fail, racks fail, networks partition, software deploys go wrong, checksums catch corruption, repair systems lag, and correlated failure is the real enemy.”
\n
At S3 scale, failure is not an exception.
\nFailure is the steady state.
\nSomewhere, a disk is failing. Somewhere else, a server is unhealthy. Somewhere else, a network device is dropping packets. Somewhere else, background repair is rebuilding redundancy.
\nThe system cannot depend on every component being healthy.
\nIt must be designed so that component failure is boring.
\nThis is the core design philosophy:
\nDo not try to prevent all hardware failure.
Design the software so hardware failure does not matter.
That idea sounds simple until you try to implement it across millions of devices.
\nThe simplest durability strategy is replication.
\nObject → copy 1
Object → copy 2
Object → copy 3
If one copy is lost, another copy survives.
\nReplication is easy to reason about and often fast to read from. But it has a major cost problem.
\nThree-way replication means storing roughly three times the data.
\nFor small systems, that may be acceptable.
\nFor S3-scale systems, that overhead becomes enormous.
\nIf you store exabytes of customer data, every extra percentage point of overhead translates into a massive physical and financial cost:
\nMore drives
More racks
More buildings
More power
More cooling
More networking
More repair traffic
More operational complexity
This is why naive replication is not enough.
\nS3 needs something more storage-efficient.
\nThat something is erasure coding.
\nErasure coding is one of the most important ideas behind large-scale storage systems.
\nInstead of storing full copies of an object, the system splits the object into data shards and computes additional parity shards.
\nYou do not need every shard to reconstruct the object.
\nYou only need a sufficient subset.
\nConceptually:
\nOriginal object
↓
Data shards: D1 D2 D3 D4 D5 D6
Parity shards: P1 P2 P3
↓
Store shards across different drives / servers / racks / AZs
If some shards are lost:
\nAvailable: D1 D2 D4 D5 D6 P2
Missing: D3 P1 P3
Original object can still be reconstructed.
This is the same high-level idea behind Reed-Solomon-style coding used in many distributed storage systems.
\nThe system adds mathematically derived parity so that missing pieces can be rebuilt.
\nThe advantage is huge:
\nNaive replication:
Simple
Fast reads
High storage overhead
Erasure coding:
More complex
Requires repair math
Much better storage efficiency
At S3 scale, storage efficiency is not a micro-optimization.
\nIt is the business model.
\nSaving even a small percentage of storage overhead can mean enormous reductions in hardware, power, cooling, network, and operational cost.
\nThis is why the story is not:
\nS3 stores three full copies of everything and calls it a day.
The real story is closer to:
\nS3 uses redundancy schemes, placement, integrity checks, and repair systems to survive failures efficiently.
S3’s famous durability target is often described as “eleven nines.”
\nBut the number itself is not magic.
\nIt emerges from an engineering model involving:
\nThe deeper insight is:
\n\n\nDurability is a race between failure and repair.
\n
If one drive fails, that is normal.
\nIf ten drives fail, that is normal at cloud scale.
\nThe dangerous case is when failures accumulate faster than the system can detect and repair lost redundancy.
\nSo S3 needs continuous background work:
\nDetect failed or unhealthy devices
↓
Identify affected objects/fragments
↓
Read surviving fragments
↓
Reconstruct missing fragments
↓
Write repaired fragments elsewhere
↓
Verify checksums
↓
Update metadata
This is not a one-time recovery operation.
\nIt is an endless repair loop.
\nAt S3 scale, the system is never in a perfectly static healthy state. Something is always failing somewhere.
\nThe goal is not to avoid failure.
\nThe goal is to repair faster than failure can accumulate.
\nThat is the real durability game.
\nDurability is not only about missing discs.
\nIt is also about silent corruption.
\nBits can rot. Drives can return bad data. Network paths can corrupt packets. Software bugs can write the wrong bytes. Memory can flip bits. Firmware can misbehave.
\nThat is why checksums matter.
\nA good storage system does not merely store bytes.
\nIt stores evidence that the bytes are still the same bytes.
\nA simplified integrity flow looks like this:
\nClient uploads object with checksum
↓
S3 validates received data
↓
S3 stores object/fragments with integrity metadata
↓
Background systems revalidate stored data
↓
Corruption is repaired from healthy redundant fragments
This is a subtle but important point:
\n\n\nRedundancy without integrity checking can preserve corrupted data.
\n
If a system has three copies but does not know which one is correct, replication alone is not enough.
\nChecksums are what let the system detect and reason about correctness.
\nWithout integrity metadata, a storage system may become very good at preserving the wrong bytes.
\nDurability asks:
\nWill my data survive?
Availability asks:
\nCan I access it right now?
These are different properties.
\nA system can be durable but temporarily unavailable.
\nFor example, your data may be safely stored, but a metadata service, network path, authorization service, or regional dependency may prevent immediate access.
\nS3 availability requires:
\nS3 must handle failures at many levels:
\nDisk failure
Server failure
Rack failure
Network failure
Availability Zone impairment
Overload
Software deployment issue
Metadata partition issue
Regional control-plane issue
A highly available system does not require every component to be healthy.
\nIt requires enough independent components to be healthy so the request can complete.
\nThat is why S3 distributes data and metadata across failure domains.
\nFor many years, S3 was famously eventually consistent for some operations.
\nThat meant after writing an object, there could be a small window where reads or listings might not immediately reflect the latest state.
\nThis created pain for data lakes and big-data systems.
\nWorkloads sometimes needed extra consistency layers or commit protocols to avoid reading stale listings or missing newly written files.
\nThen S3 evolved.
\nS3 now provides strong read-after-write consistency for key operations such as reads, writes, lists, and metadata changes.
\nThis was a major architectural milestone.
\nThe deeper point is this:
\n\n\nS3 evolved from massively scalable object storage into massively scalable strongly consistent object storage.
\n
That is not a small change.
\nStrong consistency at S3 scale is a serious distributed-systems achievement.
\nIt means customers can build simpler data pipelines because the object store behaves more predictably after writes.
\nFor data lake systems, this reduced the need for certain external consistency workarounds.
\nIf you treat S3 like one disk behind one connection, you will leave performance on the table.
\nS3 is designed to scale horizontally.
\nThe right performance model is:
\nMany concurrent requests
Multiple connections
Multipart upload for large objects
Byte-range GETs for parallel reads
Distributed prefixes for high request rates
Compute close to the bucket Region
Caching for hot content
A high-throughput S3 client does not ask:
\nHow fast is one request?
It asks:
\nHow many independent requests can I safely run in parallel?
This is the same mental model behind most cloud-scale services.
\nThroughput comes from concurrency.
\nLatency comes from placement, routing, caching, and minimizing unnecessary hops.
\nEfficiency comes from choosing the right object sizes, request patterns, and lifecycle policies.
\nMultipart upload is one of the most important S3 performance features.
\nInstead of uploading a huge object as one request, the client splits it into parts and uploads parts in parallel:
\nLarge file
↓
Part 1 ─┐
Part 2 ─┼── uploaded in parallel
Part 3 ─┤
Part 4 ─┘
↓
CompleteMultipartUpload
↓
S3 assembles object
This improves:
\nIf part 37 fails, you retry part 37.
\nYou do not restart the entire upload.
\nThis is a core pattern in distributed systems:
\n\n\nSplit large work into independently retryable chunks.
\n
Multipart upload is not just a convenience feature.
\nIt is the API exposing the same principle S3 uses internally: large work should be sharded, parallelized, and made retryable.
\nS3 can store huge numbers of objects, but not all object layouts are equally efficient.
\nA billion 1 KB objects are very different from a thousand 1 GB objects.
\nSmall objects create overhead in:
\nThis is why data lake engineers often compact small files into larger columnar files like Parquet.
\nS3 can store tiny objects.
\nBut your architecture should not accidentally create a small-file disaster.
\nA bad small-file layout can make analytics slow, lifecycle policies noisy, metadata operations expensive, and archival transitions inefficient.
\nGood object layout matters.
\nA common pattern is:
\nBad:
millions of tiny JSON files
Better:
compacted Parquet files partitioned by date/customer/region
The storage service can scale, but that does not mean every layout is equally wise.
\nS3 is really a family of storage classes with different tradeoffs.
\nAt a high level:
\nS3 Standard:
Frequently accessed data
S3 Intelligent-Tiering:
Unknown or changing access patterns
S3 Standard-IA:
Infrequently accessed but needs milliseconds retrieval
S3 One Zone-IA:
Lower-cost infrequent access in one Availability Zone
S3 Glacier Instant Retrieval:
Archive data with milliseconds access
S3 Glacier Flexible Retrieval:
Archive data with minutes-to-hours retrieval
S3 Glacier Deep Archive:
Lowest-cost archive with hours-level retrieval
This is a crucial architectural principle:
\n\n\nThe cheapest byte is the byte whose access pattern you understand.
\n
If data is hot, keep it in a hot class.
\nIf it is cold but needs instant access, use an instant retrieval class.
\nIf it is archival and rarely read, use Glacier Flexible Retrieval or Deep Archive.
\nStorage class selection is not just an AWS billing exercise.
\nIt is an architectural decision about latency, cost, retrieval behavior, and operational expectations.
\nGlacier storage classes are designed for archive workloads where retrieval latency can be minutes or hours instead of milliseconds.
\nThat tells us something important:
\nS3 Standard optimizes for frequent access.
Glacier optimizes for low-cost retention.
Some reports and discussions suggest archive systems may use very low-cost media such as tape-like systems, but AWS does not fully disclose the exact hardware implementation.
\nSo the correct engineering stance is:
\n\n\nReason from the service contract, not from unverified hardware assumptions.
\n
The service contract says Glacier trades retrieval latency for lower storage cost.
\nThat alone tells us the architecture is optimized differently from hot S3 Standard storage.
\nA good archival design should consider:
\nArchival storage is cheap only when the workflow matches the storage class.
\nIf you frequently restore from deep archive, your architecture may be fighting the product design.
\nS3 cost is not just dollars per GB.
\nIt includes:
\nStorage cost
Request cost
Retrieval cost
Data transfer cost
Lifecycle transition cost
Replication cost
KMS cost
Monitoring and automation cost
Minimum storage duration penalties
Small-object overhead
Lifecycle policies are powerful because they let storage follow data temperature.
\nA typical log retention policy might look like:
\n0–30 days:
S3 Standard
30–90 days:
S3 Standard-IA or Intelligent-Tiering
90+ days:
Glacier Flexible Retrieval
1+ year:
Glacier Deep Archive
But lifecycle policies can also create surprises.
\nBad lifecycle rules can save storage cost but increase:
\nA good lifecycle policy starts with access patterns, not pricing pages.
\nAsk:
\nHow often is this data read?
How quickly must it be restored?
Who owns restore operations?
What is the compliance requirement?
What is the minimum retention period?
What happens if restore takes 12 hours?
Cost optimization without workflow understanding is just delayed pain.
\nS3 itself provides strong security capabilities.
\nThe most common real-world S3 failures usually come from misconfiguration:
\nA mature S3 security baseline should include:
\nBlock Public Access at organization/account level
Least-privilege IAM
Bucket policies with explicit conditions
SSE-KMS for sensitive data
S3 Bucket Keys where appropriate
CloudTrail data events for sensitive buckets
S3 Inventory and Storage Lens
Object Lock for WORM/compliance use cases
Versioning for recovery
Replication for resilience and compliance
Access Analyzer for policy review
S3 security is not one setting.
\nIt is a governance system.
\nThe biggest mistake is treating S3 as “just storage” and forgetting that it often contains the organization’s most valuable data:
\nIf S3 is your data lake, then S3 is part of your security perimeter.
\nS3 encryption is often explained too simply.
\nAt a high level, there are two common server-side encryption models:
\nSSE-S3:
S3 manages encryption keys.
SSE-KMS:
AWS KMS manages keys with stronger customer control, auditability, and policy governance.
SSE-S3 is simple and low-friction.
\nSSE-KMS gives stronger governance knobs:
\nBut SSE-KMS also introduces architectural considerations:
\nFor sensitive enterprise data, SSE-KMS is often the right choice.
\nBut it should be designed deliberately.
\nEncryption is not only about cryptography.
\nIt is about ownership, auditability, key lifecycle, and operational failure modes.
\nS3 versioning is one of the most underrated safety features.
\nWithout versioning:
\nAccidental overwrite or delete → object may be gone
With versioning:
\nOverwrite creates new version
Delete creates delete marker
Older versions can be recovered
Versioning helps with:
\nObject Lock goes further by supporting write-once-read-many behavior for compliance and retention scenarios.
\nA mature backup/archive architecture often combines:
\nVersioning
Object Lock
Lifecycle policies
Cross-region replication
Restricted delete permissions
CloudTrail audit logs
The key principle is:
\n\n\nBackups are not real backups unless they are protected from the same identities and automation that can destroy production.
\n
S3 gives you primitives, but you still need a recovery architecture.
\nS3 durability inside a Region is not the same as cross-region disaster recovery.
\nIf your requirement is regional resilience, compliance locality, or faster recovery from regional disruption, you may need replication.
\nCommon replication patterns include:
\nSame-Region Replication:
Compliance, log aggregation, account separation
Cross-Region Replication:
Disaster recovery, geographic resilience, data locality
Replication Time Control:
More predictable replication SLA requirements
Replication introduces its own tradeoffs:
\nA principal engineer should distinguish clearly between:
\nDurability:
Will the object survive local device/AZ failures?
Availability:
Can I access it now?
Disaster recovery:
Can my business continue if a larger regional event occurs?
They are related, but not identical.
\nS3 is also commonly used as an event source.
\nFor example:
\nObject uploaded to bucket
↓
S3 event notification
↓
Lambda / SQS / SNS / EventBridge
↓
Processing pipeline
This pattern powers:
\nBut event-driven S3 systems require careful design.
\nYou should think about:
\nA robust S3 ingestion pipeline should assume that processing can happen more than once.
\nThe safe design is:
\nEvery object processing operation should be idempotent.
If your pipeline breaks when the same event is delivered twice, the pipeline is not production-grade.
\nA simplified mental model of S3 looks like this:
\n ┌────────────────────┐
│ Client │
└─────────┬──────────┘
│
▼
┌────────────────────┐
│ S3 Front-End/API │
└─────────┬──────────┘
│
┌────────────┴────────────┐
▼ ▼
┌──────────────────┐ ┌──────────────────┐
│ Auth/Policy Path │ │ Metadata/Index │
└──────────────────┘ └─────────┬────────┘
│
▼
┌──────────────────────┐
│ Object Placement Map │
└─────────┬────────────┘
│
┌─────────────────────────────┼─────────────────────────────┐
▼ ▼ ▼
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ Storage Fleet AZ1│ │ Storage Fleet AZ2│ │ Storage Fleet AZ3│
│ Data fragments │ │ Data fragments │ │ Parity fragments │
└──────────────────┘ └──────────────────┘ └──────────────────┘
│ │ │
└─────────────────────────────┼─────────────────────────────┘
▼
┌──────────────────────┐
│ Repair + Scrubbing │
│ Checksums + Metrics │
└──────────────────────┘
This is not AWS’s exact internal architecture.
\nIt is the right conceptual model.
\nS3 is a collection of control planes and data planes working together:
\nData plane:
PUT, GET, DELETE, LIST, multipart upload, restore
Control plane:
bucket configuration, policies, lifecycle, replication, encryption config
Background systems:
repair, checksum validation, lifecycle transitions, inventory, replication, monitoring
The background systems are not secondary.
\nThey are what make the durability promise real.
\nS3 is now more than object storage.
\nIt is the storage substrate for:
\nThe reason is simple:
\nS3 separated storage from compute.
Before S3-style architectures, storage and compute were often tightly coupled.
\nYour data lived on the same cluster that processed it.
\nS3 changed that.
\nNow you can store data once and process it with many different compute engines:
\nAthena
EMR
Glue
Redshift Spectrum
Spark
Presto/Trino
Lambda
SageMaker
Snowflake external tables
Databricks
Custom EC2/EKS workloads
This separation is one of the biggest architectural shifts in cloud computing.
\nIt allowed organizations to build data lakes where storage is durable, cheap, and independent of the compute engines that query it.
\nS3 is not impressive because it stores files.
\nS3 is impressive because it turns failure into a normal operating condition.
\nIt assumes:
\nDiscs will fail.
Servers will fail.
Networks will fail.
Racks will fail.
Availability Zones may be impaired.
Bits may corrupt.
Customers will create unpredictable workloads.
Access patterns will change.
Costs must remain low.
Performance must scale horizontally.
Security must be configurable but safe by default.
And still, the system provides a simple API:
\nPUT object
GET object
DELETE object
LIST objects
That simplicity hides enormous complexity.
\nThe true genius of S3 is this:
\n\n\nIt gives developers a simple object abstraction while hiding one of the most sophisticated distributed storage systems ever built.
\n
S3’s story is not:
\nAWS has big discs.
S3’s story is:
\nCheap hardware
+ extreme parallelism
+ erasure coding
+ checksums
+ metadata systems
+ repair loops
+ lifecycle automation
+ operational discipline
= planet-scale durable storage
That is the lesson worth remembering.
\nThe future of storage is not about one infinitely fast disk.
\nIt is about fleets.
\nIt is about failure-aware software.
\nIt is about mathematical redundancy.
\nIt is about background repair.
\nIt is about turning unreliable parts into a reliable whole.
\nThat is what S3 does.
\nAnd that is why S3 is one of the most important distributed systems in the history of cloud computing.
\n![\"Caching](\"https://blogs.pranjalkumar.in/assets/images/caching-vs-cdn-explained%20by%20Pranjal%20Kumar%20-%20software%20engineer-48f9f95788cfd3f6afd1a14e95ba449b.png\")
\n\nThere are only two hard things in Computer Science: cache invalidation and naming things.\n— Phil Karlton
\n
Here’s a conversation I’ve had more times than I can count: someone on the team says \"let’s add a CDN\" when what they actually mean is \"we need caching.\" Or worse, someone implements Redis caching for static assets that should just live on a CDN edge node. Both are about serving content faster, but they solve fundamentally different problems, at different layers, with different tradeoffs.
\nThis post breaks down what each actually does under the hood, when to reach for one vs. the other, and how they work together in production systems.
\nBefore we compare them, let’s zoom out. When a user requests a resource, the request passes through multiple layers where speed-ups can happen:
\n\n\nBrowser cache → CDN edge → Reverse proxy cache → Application cache → Database cache → Disk
\n
Caching is a technique — storing computed results closer to where they’re needed. It can happen at any layer: browser, server memory, Redis, database query cache, even CPU L1/L2/L3.
\nCDN is infrastructure — a geographically distributed network of servers that moves content physically closer to users. A CDN uses caching as its mechanism, but its value proposition is about geography and network topology, not just storage.
\nThink of it this way: all CDNs cache, but not all caches are CDNs.
\nCaching is storing the result of an expensive computation or data retrieval so that future requests for the same thing can be served faster. The \"expensive\" part might be:
\nThe key insight: caching trades freshness for speed. Every cache is a bet that the data won’t change before someone asks for it again.
\nThere are several patterns for how reads and writes interact with the cache:
\nRead-through (Cache-aside): The most common pattern. Application checks cache first; on miss, fetches from source, writes to cache, returns result.
\nasync function getUser(userId) {
// Check cache first
const cached = await redis.get(`user:${userId}`);
if (cached) {
return JSON.parse(cached); // Cache hit: ~1ms
}
// Cache miss: fetch from DB (~50ms)
const user = await db.query('SELECT * FROM users WHERE id = $1', [userId]);
// Populate cache for next time (TTL: 1 hour)
await redis.setex(`user:${userId}`, 3600, JSON.stringify(user));
return user;
}
![\"How](\"https://www.imperva.com/wp-content/uploads/sites/13/2020/03/diagram-31@3x.png\")
Write-through: Every write goes to both cache and backing store synchronously. Guarantees consistency but adds write latency.
\nWrite-behind (Write-back): Writes go to cache immediately; cache asynchronously flushes to backing store. Fast writes, but you risk data loss if the cache crashes before flushing.
\nWrite-around: Writes go directly to the backing store, bypassing cache. The cache only gets populated on reads. Good when writes rarely get re-read.
\nThis is where most developers first encounter caching, and where most mistakes happen. The HTTP spec gives us powerful cache control:
\nHTTP/1.1 200 OK
Cache-Control: public, max-age=31536000, immutable
ETag: \"a1b2c3d4\"
Last-Modified: Wed, 15 Jan 2025 08:00:00 GMT
Vary: Accept-Encoding
Let me break these down because they’re more nuanced than people realize:
\nCache-Control directives:
| Directive | What it means | When to use |
|---|---|---|
public | Any cache (CDN, proxy, browser) can store this | Static assets, public pages |
private | Only the user’s browser can cache this | User-specific data (dashboards, profiles) |
no-cache | Cache it, but revalidate with origin before serving | Content that changes unpredictably |
no-store | Don’t cache at all. Period. | Sensitive data (banking, health records) |
max-age=N | Fresh for N seconds | Everything with a known TTL |
s-maxage=N | Like max-age, but only for shared caches (CDNs) | When CDN TTL should differ from browser TTL |
stale-while-revalidate=N | Serve stale content while fetching fresh in background | High-traffic pages where slight staleness is OK |
immutable | This will never change (don’t even revalidate) | Fingerprinted assets (app.a3b4c5.js) |
ETag and conditional requests: Instead of re-downloading a full response, the browser sends If-None-Match: \"a1b2c3d4\". If the content hasn’t changed, the server responds with 304 Not Modified — zero bytes transferred for the body.
stale-while-revalidate: This is the underrated gem. It lets you serve a stale cached response immediately while fetching a fresh one in the background. Users always get instant responses, and the cache stays reasonably fresh. Cloudflare, Fastly, and modern browsers all support it.
Cache-Control: public, max-age=60, stale-while-revalidate=300
This says: \"fresh for 60 seconds; after that, serve stale for up to 300 more seconds while revalidating.\" User never waits for origin.
\nWhen the cache is full, something has to go. The eviction policy determines what:
\nallkeys-random.In practice, LRU with TTLs is the sweet spot for most applications. Set a TTL as a safety net (data can’t be stale forever), and let LRU handle capacity management within that window.
\nPhil Karlton wasn’t joking. Cache invalidation is genuinely hard because you’re dealing with distributed state:
\nThe stale data problem: User updates their profile. The database has the new data. But the cache still has the old data. For the next N seconds (until TTL expires), other users see stale information.
\nStrategies to handle this:
\nuser:123:v7). On write, increment version. Old versions naturally become unreachable.// Event-driven invalidation example
async function updateUser(userId, data) {
await db.query('UPDATE users SET data = $1 WHERE id = $2', [data, userId]);
// Publish invalidation event
await eventBus.publish('cache.invalidate', {
key: `user:${userId}`,
reason: 'user_update',
});
}
node-cache, lru-cache (Node.js), Guava Cache, Caffeine (Java). Zero network overhead, but not shared across instances.proxy_cache, Varnish. Caches full HTTP responses. Great for APIs and rendered pages.![\"Content](\"https://doimages.nyc3.cdn.digitaloceanspaces.com/CDN.png\")
A CDN is a network of servers (called edge nodes or Points of Presence / PoPs) deployed in data centers worldwide. When a user requests content, the CDN routes them to the nearest edge node using anycast routing or DNS-based routing, so the data travels the shortest physical distance.
\nThe key insight: CDNs solve the speed-of-light problem. Light in fiber travels at ~200,000 km/s. A request from Mumbai to a server in Virginia is ~14,000 km — that’s 70ms just for the photons, one way. A CDN PoP in Mumbai makes that ~2ms.
\nWithout CDN: User (Mumbai) ─── 14,000 km ───> Origin (Virginia)
RTT: ~140ms + server processing
With CDN: User (Mumbai) ─── 50 km ───> CDN PoP (Mumbai)
RTT: ~4ms + edge processing (if cached)
That’s a 35x improvement in network latency alone. For a page that makes 20 requests for CSS, JS, images, and fonts, this adds up fast.
\ncdn.example.com. DNS resolves to the nearest PoP’s IP (via anycast or geo-DNS).Cache-Control headers, then serves to the user.![\"CDN](\"https://media.geeksforgeeks.org/wp-content/uploads/20231222162112/Usecase-of-CDN.jpg\")
Pull-based vs Push-based:
\nAnycast routing: All CDN PoPs advertise the same IP address via BGP. The internet’s routing protocol naturally sends packets to the nearest PoP. This is how Cloudflare handles DDoS — attack traffic gets absorbed by the nearest PoP rather than concentrating on one location.
\nTiered caching: Not every PoP goes directly to origin on a miss. CDNs use a hierarchy:
\nEdge PoP (city-level) → Regional shield (continent-level) → Origin
This reduces origin load. If the Mumbai PoP has a miss, it asks the Singapore regional shield first. Only if Singapore also misses does the request go to origin. This is called origin shielding and can reduce origin traffic by 90%+.
\nModern CDNs are more than just cache-and-serve. They’re now full edge compute platforms:
\nCDN invalidation is trickier than application cache invalidation because you’re purging content across potentially hundreds of global nodes:
\nPurge by URL: Invalidate a specific resource. Propagates to all PoPs (usually takes 1-30 seconds depending on provider).
\nPurge by tag/surrogate key: Tag responses with categories (product:123, category:electronics). Purge all resources with a specific tag at once. Fastly does this in ~150ms globally.
Purge everything: Nuclear option. Clears all cached content. Use with caution on high-traffic sites; origin will get hammered.
\nThe safer approach — versioned URLs:
\n<!-- Instead of purging, just change the URL -->
<link rel=\"stylesheet\" href=\"/css/app.css?v=a3b4c5\">
<!-- Or better, use content-hash in filename -->
<link rel=\"stylesheet\" href=\"/css/app.a3b4c5.css\">
With content-hashed filenames, you never need to invalidate. Old versions age out naturally. New deploys reference new filenames. This is what Webpack, Vite, and every modern bundler do.
\n| Caching | CDN | |
|---|---|---|
| Primary problem | Computation/retrieval is expensive | Users are physically far from servers |
| Latency saved | DB query time (50-500ms) | Network round-trip time (50-200ms) |
| Layer | Application logic layer | Network/transport layer |
| Scope | Single datacenter (usually) | Global |
| Content type | Any computed result (DB rows, API responses, rendered HTML) | Primarily HTTP responses (static and dynamic) |
| Invalidation | Programmatic, precise, fast | Global propagation, eventually consistent |
| Cost model | RAM/infrastructure you manage | Per-GB egress + request count |
Use caching when:
\nUse a CDN when:
\nUse both when (most production systems):
\nHere’s how I’d configure caching for a typical web application with both a CDN and application caching:
\n# Nginx configuration (behind CDN)
# Static assets with content-hash in filename: cache forever
location ~* \\.(js|css)$ {
# Fingerprinted files never change
add_header Cache-Control \"public, max-age=31536000, immutable\";
}
# Images: cache for 1 year on CDN, 1 week in browser
location ~* \\.(png|jpg|webp|svg|gif|ico)$ {
add_header Cache-Control \"public, max-age=604800, s-maxage=31536000\";
}
# HTML pages: CDN caches for 60s, browser always revalidates
location ~* \\.html$ {
add_header Cache-Control \"public, max-age=0, s-maxage=60, stale-while-revalidate=300\";
add_header Surrogate-Key \"pages\";
}
# API responses: private, short TTL, revalidate
location /api/ {
add_header Cache-Control \"private, max-age=0, must-revalidate\";
add_header ETag $request_id;
}
# User-specific pages: never cache on shared caches
location /dashboard {
add_header Cache-Control \"private, no-store\";
}
The key patterns:
\ns-maxage for CDN caching with stale-while-revalidate for seamless updatesIf your CDN caches a page that shows \"Hello, Pranjal\" and serves it to every user... you’ve got a privacy incident. Always use Cache-Control: private or no-store for authenticated responses. If you do cache personalized pages on a CDN, use Vary: Cookie (but this usually kills your cache hit rate anyway).
Vary headers correctlyIf your server returns different content based on Accept-Encoding (gzip vs brotli) or Accept (JSON vs HTML), you must include a Vary header. Otherwise the CDN might serve a gzip response to a client that sent Accept-Encoding: br.
Vary: Accept-Encoding, Accept
<!-- Fragile: some CDNs ignore query strings by default -->
<script src=\"/app.js?v=123\"></script>
<!-- Robust: always treated as a unique resource -->
<script src=\"/app.abc123.js\"></script>
If you set max-age=31536000 without immutable, browsers will still occasionally revalidate (especially on page reload). Adding immutable tells the browser \"don’t even bother checking, this will never change.\"
When a popular cached item expires, hundreds of requests simultaneously hit origin. Solutions:
\nWhen you’re designing a system and thinking about performance:
\nMeasure first. Where is the actual latency? Network? Database? Computation? Don’t add caching for a problem that’s actually a slow query needing an index.
\nIf the bottleneck is network latency to users → CDN.
\nIf the bottleneck is repeated expensive computation → Application cache.
\nIf you need both speed and freshness → stale-while-revalidate + background refresh.
If consistency matters more than speed → Cache conservatively with short TTLs and active invalidation.
\nIf you’re serving static assets → CDN with immutable caching. This is non-negotiable for any production site.
\nCaching and CDNs are complementary, not competing. Caching is the technique of storing results to avoid re-computation. CDNs are the infrastructure that moves content geographically closer to users. Almost every production system needs both:
\nThe art isn’t in choosing one over the other. It’s in understanding which cache layer is responsible for which content, setting appropriate TTLs for your consistency requirements, and having a clear invalidation strategy for when things change.
\nGet the headers right, version your static assets, and use stale-while-revalidate. That alone puts you ahead of 90% of web applications.
![\"Multi-process](\"https://miro.medium.com/v2/1*meHpNW0zTlYpyxNa1aqTFw.png\")
\n\nThe front end may be a rapidly changing field, and it is difficult for us to understand all aspects of it. However, front-end systems generally run in browsers, so a better understanding of browsers can help us develop front-end applications more effectively. This is also one of the reasons for this article and serves as an instance analysis of runtime.
\n
The browser architecture has undergone a transition from single-process browsers to multi-process browsers. Emphasizing stability, smoothness, and security, processes have been decomposed into rendering, GPU, network, and plugins, improving the cleanliness of the architecture. To review the browser's architecture, it is necessary to further understand the process of opening pages, the page rendering process, and the browser plugin mechanism. By organizing the timeline of Chrome extension versions, especially the transition from Manifest V1 to Manifest V3, one can gain a relatively comprehensive understanding of how browsers have evolved.
\nBefore 2007, the typical browser architecture was like this:
\nSingle-process browser architecture runs the entire web browser in a single operating system process, consolidating tasks such as network processing, plugins, JavaScript runtime, rendering engine, page management, and user interface elements into one execution space. While simplifying resource management, this architectural design has the following main issues:
\nThe advantage of a single-process browser is that all browser components operated within one process simplify resource management and coordination. Single-process browsers typically exhibit lower memory usage rates which benefit resource efficiency improvement. Tasks run sequentially within a unified process.
\nThe Chrome process architecture released in 2008 is an example of a multi-process browser as shown in the diagram below:
\nEarly browser architectures divided functionality into three main processes: browser, plugins, and rendering. Each page and its plugins run independently in dedicated rendering and plugin processes, communicating through IPC.
\nInter-process communication (IPC) is a mechanism that allows processes to communicate and synchronize operations on a computer. It facilitates efficient data exchange and coordination between different programs. Key IPC mechanisms include shared memory, allowing processes to synchronize access to shared public memory areas using semaphores. Named and unnamed pipes provide one-way communication; IPC in Linux typically involves sharing files or shared memory storage with semaphores. Message queues support asynchronous communication, helping separate sender and receiver processes. Additionally, processes can communicate through signals to notify specific events or requests. Sockets extend IPC across different machines using network protocols.
\nMulti-process browsers enhance stability; isolated processes prevent crashes from affecting the entire browser. Crashes of pages or plugins only affect their specific process, ensuring the stability of other pages and the browser itself. Running JavaScript in a separate rendering process also isolates its impact; if a script blocks the rendering process, it only affects the current page while leaving the rest of the browser unaffected since each page runs scripts in its dedicated rendering process. Furthermore, Chrome places plugins and rendering processes in sandbox environments limiting read/write access to data - even if malicious software executes within these processes it cannot breach the sandbox for system permissions gain - this is an example of a compartmentalized architecture pattern.
\nA sandbox is a testing environment that allows users to run programs or open files without impacting the entire system's operation. In cybersecurity fields, sandboxes analyze potential malicious code execution detecting threats for mitigation purposes.
\nIn recent versions of Chrome browsers have key components as shown below:
\nThe browser process manages the display interface, user interaction, and subprocess coordination, and provides storage functions. It acts as a \"scheduler\" to coordinate other processes, for example calling the network process when entering a URL. The rendering process converts HTML, CSS, and JavaScript into interactive web pages and runs the V8 engine. For security reasons, Chrome creates a separate rendering process for each tab in sandbox mode.
\nThe GPU process was initially designed for 3D CSS effects but later expanded to drawing web pages and the Chrome UI interface. Introduced in Chrome's multi-process architecture to meet common browser needs. The network process independently loads page network resources; originally a module within the browser process, it now runs as a separate process. The plugin process manages plugins to prevent crashes affecting browsers and pages due to the inherent instability of plugins.
\nThe modern browser architecture is shown in the diagram below:
\nModern browsers use strategies such as lazy loading and caching to prioritize performance. Browsers display web content through rendering processes. Key stages include HTML parsing, CSS styling, layout creation, and drawing, with specific steps as follows:
\nThe browser reads HTML character by character, identifying elements, attributes, and text, then constructs the DOM tree representing the structure of the webpage to ensure the correct display of HTML code.
\nThe CSS Object Model expresses the styles applied to HTML elements, similar to the structured hierarchy of the DOM tree, taking into account the specificity and cascading nature of styles, allowing access, manipulation, and computation of styles.
\nLayout managers combine the DOM and CSS object model to form a rendering tree, determine the size of the Box based on content, padding, etc., and use various methods to build specific positions. At the same time, stacking contexts and Z-index are used to handle overlapping elements and batch processing techniques are used to optimize layout changes. Finally, elements are drawn on the screen and continuously updated during user interaction.
\n\n\nWhen using plugins, the browser's operation is simpler than regular web pages. The rendering process is responsible for running the webpage. When a page is opened, contentscript.js is loaded and injected into the webpage environment, operating similar to JavaScript by manipulating the DOM tree and altering display. The GPU process supports hardware functions for rendering plugin interfaces, while the network process manages external resource requests within plugins, such as dependencies on external JS resources. Simultaneously, the storage process provides local storage functionality for plugins using chrome.storage.local to store and retrieve data in Chrome extensions. The browser process acts as a bridge facilitating communication between Extension Page and contentscript.js.
\n
The development process of the plugin mechanism is as follows:
\n| Year | Version | Description |
|---|---|---|
| August 2012 | Manifest V1 | Chrome plugins were initially based on Manifest V1, defining basic functions and permissions. |
| April 2013 | Chrome26 Stable | Including support for Manifest V1 plugins. |
| May 2014 | Chrome35 Stable | Incremental updates of the browser continue to support Manifest V1 plugins. |
| September 2014 | Chrome 37 Stable | Further improvements and bug fixes for Manifest V1 plugins. |
| May 2015 | Chrome43 Stable | Continued support for Manifest V1. |
| December 2015 | Manifest V2 | Introduced Manifest V2, bringing better security and additional features. |
| June 2016 | Chrome51 Stable | Manifest V2 became the standard for plugin development. |
| September 2016 | Chrome53 Stable | Continuous improvement and optimization of Manifest V2. |
| January 2019 | Manifest V3 Birthed | Focuses on security, performance, and development flexibility. |
| March 2020 | Chrome80 Stable | While manifest v2 remains the standard for plugins. manifest v3 is available for testing but not mandatory yet. |
| March 2021 | Chrome89 stable | Manifest v2 still default but attention shifting towards manifest v3. |
| October 2021 | Chrome94 stable | Manifestv3 starts enforcing some features with migration guides & tools provided to developers. |
| March 2022 | Chrome98 stable | Manifestv3 becomes the default new plugin version while continuing to support manifestv2. |
| August 2022 | Chrome104 stable | All plugins fully transitioned to manifestv3 officially announcing no more support for manifestv2. |
| March 2023 | Chrome108 stable | Maintained full support for manifestv3 ensuring smooth transition for all plugins. |
| July 2023 | ManifestV3 Preview | Allowing developers to explore upcoming changes and provide valuable feedback. |
| October 2023 | Chrome version | Further improved support for ManifestV3 in Chrome Stable Version addressing all issues reported during the preview stage. |
| January 2024 | Pre-stable release | Encouraging developers to migrate their extensions to Manifest V3 and providing comprehensive documentation and migration guidance. |
| January 2024 | Manifest V3 pre-stable version released | Manifest V3 has reached a stable state, encouraging developers to migrate their extensions to V3 and providing comprehensive documentation and migration guides. |
| March 2024 | Chrome116 stable | Fully supports Manifest V3, developers update their plugins to ensure compatibility with the latest standards. |
Overall, Chrome extensions (also known as add-ons) have gone through the development of 3 major versions: Manifest V1, Manifest V2, and Manifest V3.
\nManifest V1 (MV1) was the initial version of the Chrome extension manifest and has been deprecated. Manifest V2 (MV2) is the mainstream version widely used in current Chrome extensions, providing a robust framework for building extensions with enhanced browser functionality features. Manifest V3 is the latest version gradually replacing MV2. The introduction of MV3 aims to address security and performance issues by strengthening security measures and promoting better performance in extension development. Starting from Chrome 127 (June 2024), Google will begin disabling Manifest V2 extensions in pre-stable versions of Chrome, encouraging developers to transition to MV3.
\nFeatures of Manifest V2:
\nscript-src 'self'; object-src 'self'; Content-Security-Policy (CSP) to set the default content security policy.web_access_resources property in the manifest.chrome.extension replaces chrome.self pointing to the plugin itself, chrome.tension.getTabContentses is replaced by tension.getView, Port.tab is replaced by runtime.Port, etc.Features of Manifest V3:
\nManifest V3 introduces Service worker, replacing background pages.Manifest V3 represents a significant shift from V1 and V2, driven by Chrome's commitment to improving privacy, security, and overall extension performance. Unlike previous versions, Manifest V3 prioritizes resource efficiency, addressing concerns about Chrome's historically high resource usage. Its core goal is to optimize browser performance by limiting system resource consumption through extensions. While imposing additional restrictions, Manifest V3 introduces significant benefits. The ServiceWorker feature allows extensions to operate without constantly residing in the background, enabling the recycling of extension resources and effectively reducing overall browser overhead. Restrictions on rule calculations serve as a control mechanism to ensure that individual extensions do not excessively consume resources. These changes collectively contribute to a smoother browsing experience in Chrome, meeting users' expectations for improved browser efficiency.
\nWhen migrating from V2 to V3, due to the absence of background.html for configuring page backgrounds, XMLHttpRequest on the Windows object is no longer applicable for constructing AJAX requests in background.html as it was in the V2 version. Instead, data must be retrieved using extraction methods.
\nFurthermore, the short lifespan of service workers and their termination during periods of inactivity leading to occasional startups, runs, and terminations throughout the plugin lifecycle introduce instability. In MV2 global variables were used directly for storing data; therefore modifications are required in backound.js logic to enhance stability and functionality under these circumstances. Additionally transitioning from webRequest API to statativeNetRequest API necessitates substantial code refactoring.
\n4.3.1 manifest file
\nThe manifest.json file is very important for Chrome extensions located in the root directory. It is used to configure all plugin settings, with basic parameters such as Manif_version, name, and version.
\nAn example of Manifest V2 is as follows:
\n<button class=\"hljs-copy p-1 bg-gray-300 dark:bg-black\">Copy</button><merlin-component id=\"merlin-code-summarizer\" class=\"merlin-code-summarizer\"></merlin-component><code class=\"overflow-x-auto mt-1 language-json bg-gray-100 dark:bg-gray-900 hljs\"><span class=\"hljs-punctuation\">{</span>
<span class=\"hljs-attr\">\"manifest_version\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-number\">2</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-comment\">// Plugin name</span>
<span class=\"hljs-attr\">\"name\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"...\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-comment\">// Plugin version</span>
<span class=\"hljs-attr\">\"version\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"1.0.0\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-comment\">// Plugin description</span>
<span class=\"hljs-attr\">\"description\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"...\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"icons\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-punctuation\">{</span>
<span class=\"hljs-attr\">\"16\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"img/icon16.png\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"48\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"img/icon48.png\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"128\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"img/icon128.png\"</span>
<span class=\"hljs-punctuation\">}</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-comment\">// Persistent background JS or background page</span>
<span class=\"hljs-attr\">\"background\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-punctuation\">{</span>
<span class=\"hljs-attr\">\"scripts\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-punctuation\">[</span><span class=\"hljs-string\">\"js/background.js\"</span><span class=\"hljs-punctuation\">]</span>
<span class=\"hljs-punctuation\">}</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-comment\">// Browser icon settings ：browser_action, page_action, app</span>
<span class=\"hljs-attr\">\"browser_action\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-punctuation\">{</span>
<span class=\"hljs-attr\">\"default_icon\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"img/icon.png\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"default_title\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"...\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"default_popup\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"popup.html\"</span>
<span class=\"hljs-punctuation\">}</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-comment\">// Icon displayed only when specific pages are open</span>
<span class=\"hljs-attr\">\"page_action\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-punctuation\">{</span>
<span class=\"hljs-attr\">\"default_icon\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"img/icon.png\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"default_title\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"...\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"default_popup\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"popup.html\"</span>
<span class=\"hljs-punctuation\">}</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-comment\">// JS directly injected into pages</span>
<span class=\"hljs-attr\">\"content_scripts\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-punctuation\">[</span><span class=\"hljs-punctuation\">{</span>
<span class=\"hljs-attr\">\"matches\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-punctuation\">[</span><span class=\"hljs-string\">\"<all_urls>\"</span><span class=\"hljs-punctuation\">]</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"js\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-punctuation\">[</span><span class=\"hljs-string\">\"js/content-script.js\"</span><span class=\"hljs-punctuation\">]</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"css\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-punctuation\">[</span><span class=\"hljs-string\">\"css/custom.css\"</span><span class=\"hljs-punctuation\">]</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-comment\">// Code injection timing, default is document_idle</span>
<span class=\"hljs-attr\">\"run_at\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"document_start\"</span>
<span class=\"hljs-punctuation\">}</span><span class=\"hljs-punctuation\">]</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-comment\">// Permissions requested</span>
<span class=\"hljs-attr\">\"permissions\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-punctuation\">[</span>
<span class=\"hljs-string\">\"contextMenus\"</span><span class=\"hljs-punctuation\">,</span> <span class=\"hljs-comment\">// Right-click menu</span>
<span class=\"hljs-string\">\"tabs\"</span><span class=\"hljs-punctuation\">,</span> <span class=\"hljs-comment\">// Tabs</span>
<span class=\"hljs-string\">\"notifications\"</span><span class=\"hljs-punctuation\">,</span> <span class=\"hljs-comment\">// Notifications</span>
<span class=\"hljs-string\">\"webRequest\"</span><span class=\"hljs-punctuation\">,</span> <span class=\"hljs-comment\">// Web requests</span>
<span class=\"hljs-string\">\"webRequestBlocking\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-string\">\"storage\"</span><span class=\"hljs-punctuation\">,</span> <span class=\"hljs-comment\">// Plugin local storage</span>
<span class=\"hljs-string\">\"https://*/*\"</span> <span class=\"hljs-comment\">// Websites accessible via executeScript or insertCSS</span>
<span class=\"hljs-punctuation\">]</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-comment\">// List of plugin resources directly accessible by normal pages \"web_accessible_resources\": [\"js/inject.js\"],</span>
<span class=\"hljs-attr\">\"homepage_url\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"...\"</span><span class=\"hljs-punctuation\">,</span> <span class=\"hljs-comment\">// Plugin homepage</span>
<span class=\"hljs-attr\">\"chrome_url_overrides\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-punctuation\">{</span> <span class=\"hljs-comment\">// Override browser default pages</span>
<span class=\"hljs-attr\">\"newtab\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"newtab.html\"</span>
<span class=\"hljs-punctuation\">}</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"options_ui\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-punctuation\">{</span> <span class=\"hljs-comment\">// Plugin options page</span>
<span class=\"hljs-attr\">\"page\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"options.html\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"chrome_style\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-literal\"><span class=\"hljs-keyword\">true</span></span>
<span class=\"hljs-punctuation\">}</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"omnibox\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-punctuation\">{</span> <span class=\"hljs-attr\">\"keyword\"</span> <span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"...\"</span> <span class=\"hljs-punctuation\">}</span><span class=\"hljs-punctuation\">,</span> <span class=\"hljs-comment\">// Register a keyword in the address bar for search suggestions, only one keyword can be set</span>
<span class=\"hljs-attr\">\"default_locale\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"en\"</span><span class=\"hljs-punctuation\">,</span> <span class=\"hljs-comment\">// Default language</span>
<span class=\"hljs-attr\">\"devtools_page\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"devtools.html\"</span><span class=\"hljs-punctuation\">,</span> <span class=\"hljs-comment\">// Devtools page entry, can only point to an HTML file \"content_security_policy\": \"...\", // Security policy</span>
<span class=\"hljs-attr\">\"web_accessible_resources\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-punctuation\">[</span> <span class=\"hljs-comment\">// Loadable resources</span>
<span class=\"hljs-string\">\"RESOURCE_PATHS\"</span>
<span class=\"hljs-punctuation\">]</span>
<span class=\"hljs-punctuation\">}</span></code>
An example of Manifest V3 is as follows:
\n<button class=\"hljs-copy p-1 bg-gray-300 dark:bg-black\">Copy</button><merlin-component id=\"merlin-code-summarizer\" class=\"merlin-code-summarizer\"></merlin-component><code class=\"overflow-x-auto mt-1 language-json bg-gray-100 dark:bg-gray-900 hljs\"><span class=\"hljs-punctuation\">{</span>
<span class=\"hljs-attr\">\"manifest_version\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-number\">3</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"name\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"...\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"version\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"1.0.0\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"description\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"...\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"icons\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-punctuation\">{</span>
<span class=\"hljs-attr\">\"16\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"img/icon16.png\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"48\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"img/icon48.png\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"128\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"img/icon128.png\"</span>
<span class=\"hljs-punctuation\">}</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"background\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-punctuation\">{</span>
<span class=\"hljs-attr\">\"service_worker\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"js/background.js\"</span>
<span class=\"hljs-punctuation\">}</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"action\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-punctuation\">{</span>
<span class=\"hljs-attr\">\"default_icon\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"img/icon.png\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"default_title\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"...\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"default_popup\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"popup.html\"</span>
<span class=\"hljs-punctuation\">}</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"content_security_policy\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-punctuation\">{</span>
<span class=\"hljs-attr\">\"extension_pages\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"...\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"sandbox\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"...\"</span>
<span class=\"hljs-punctuation\">}</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"web_accessible_resources\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-punctuation\">[</span>
<span class=\"hljs-punctuation\">{</span>
<span class=\"hljs-attr\">\"resources\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-punctuation\">[</span><span class=\"hljs-string\">\"RESOURCE_PATHS\"</span><span class=\"hljs-punctuation\">]</span>
<span class=\"hljs-punctuation\">}</span>
<span class=\"hljs-punctuation\">]</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"permissions\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-punctuation\">[</span>
<span class=\"hljs-string\">\"contextMenus\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-string\">\"tabs\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-string\">\"notifications\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-string\">\"webRequest\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-string\">\"webRequestBlocking\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-string\">\"storage\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-string\">\"https://*/*\"</span>
<span class=\"hljs-punctuation\">]</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"web_accessible_resources\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-punctuation\">[</span><span class=\"hljs-string\">\"js/inject.js\"</span><span class=\"hljs-punctuation\">]</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"homepage_url\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"...\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"chrome_url_overrides\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-punctuation\">{</span>
<span class=\"hljs-attr\">\"newtab\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"newtab.html\"</span>
<span class=\"hljs-punctuation\">}</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"options_ui\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-punctuation\">{</span>
<span class=\"hljs-attr\">\"page\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"options.html\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"chrome_style\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-literal\"><span class=\"hljs-keyword\">true</span></span>
<span class=\"hljs-punctuation\">}</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"omnibox\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-punctuation\">{</span>
<span class=\"hljs-attr\">\"keyword\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"...\"</span>
<span class=\"hljs-punctuation\">}</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"default_locale\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"zh_CN\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"devtools_page\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"devtools.html\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"content_security_policy\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-string\">\"...\"</span><span class=\"hljs-punctuation\">,</span>
<span class=\"hljs-attr\">\"web_accessible_resources\"</span><span class=\"hljs-punctuation\">:</span> <span class=\"hljs-punctuation\">[</span><span class=\"hljs-string\">\"RESOURCE_PATHS\"</span><span class=\"hljs-punctuation\">]</span>
<span class=\"hljs-punctuation\">}</span></code>
4.3.2 Content Script
\nContent scripts in Chrome extensions inject JS and CSS into specified pages through configuration. They share the DOM with the original page but not JavaScript. Accessing page JS variables requires injecting JS. Content scripts cannot access most Chrome APIs, except for:
\nFor other APIs, communication needs to be done with the background or service worker.
\n4.3.3 Background Script
\nThe background scripts in Chrome extensions have the longest lifespan and run continuously when the browser is open. They have extensive permissions, allowing access to most Chrome extension APIs and cross-origin requests without CORS restrictions. In Manifest V3, background pages are replaced by service workers with shorter lifespans and event-based execution, making them unsuitable for storing global variables.
\n4.3.4 Pop-up window
\nA pop-up window is a small window on a webpage that appears when clicking on an icon in the top right corner. It quickly closes when the user interacts outside of the webpage. Typically used for temporary interactions, its permission level is similar to the background but with a shorter lifespan.
\n4.3.5 Inject Script
\nDevelopers coined the term \"injection script\" during Chrome extension development. It refers to JavaScript injected into a page through DOM manipulation. Although content scripts can manipulate the DOM, they cannot be directly accessed due to access restrictions. This limitation is particularly evident in event binding. To meet the common requirement of adding a button to trigger the plugin on a web page, developers have adopted script insertion.
\nIn Chrome extensions, communication relies on five types of scripts:
\nwindow.postMessage.window.postMessage, chrome.runtime.sendMessage and chrome.runtime.connect for inter-script communication.chrome.tabs.sendMessage and chrome.tabs.connect for communication.chrome.tabs.sendMessage\", chrome.tabs.connect, chrome.extension.getBackgroundPage and chrome.extension.getViews.chrome.devtools.inspectedWindow.eval and chrome.runtime.sendMessage.Each script has different permissions emphasizing the importance of their communication. This interaction is crucial for enabling extensive plugin functionality.
\nReviewing the past to learn new things, browser architecture, as the cornerstone of the modern Internet, has gone through multiple iterations and innovations, always serving as a bridge between users and web content. Looking back at its development history, from the early days of single rendering engines to today's multi-process, multi-threaded architecture, each change has brought about a smoother and more secure browsing experience. Looking ahead, browser architecture will continue to deepen its performance optimization and security enhancement to bring users an even better web browsing experience.
", "url": "https://blogs.pranjalkumar.in/blog/past-and-present-browser-architecture", "title": "The past and present of browser architecture", "summary": "How browser architecture evolved from single-process to multi-process - covering Chrome's rendering, GPU, network, and plugin process decomposition, IPC, sandboxing, and the Manifest V1 to V3 transition.", "date_modified": "2024-03-20T00:00:00.000Z", "author": { "name": "Pranjal Kumar", "url": "https://www.linkedin.com/in/pranjal--kumar" }, "tags": [ "browser", "programming", "javascript" ] }, { "id": "https://blogs.pranjalkumar.in/blog/docker-image-guide", "content_html": "![\"Dockfolio](\"https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o2jy675uu0n48fpq4fx1.png\")
Prepare for an exhilarating journey into the world of \"dockfolio,\" a comprehensive Docker image created to simplify Portfolio creation for developers. This article is your personalized roadmap, guiding you through every step, command, and thrill involved in curating and distributing a Docker image.
The Birth of Dockfolio: The genesis of \"dockfolio\" emerged from a passionate pursuit to streamline the intricate process of web development deployment of a Developer Portfolio. It's a brainchild designed to be your all-in-one, hassle-free toolkit, blending a medley of essential tools to supercharge your development environment.
\nCrafting the Container:
\nmkdir dockfolio
cd dockfolio
tar & gzip\ntar czvf dockfolio.tar.gz dockfolio/*
touch Dockerfile
vim Dockerfile
![\"Making](\"https://cdn.hashnode.com/res/hashnode/image/upload/v1702374832676/8dd6fb2a-3d6e-458c-84f3-ebaa72f71cbe.png\")
docker build -t pranjalbarnwal/dockfolio .
![\"Build](\"https://cdn.hashnode.com/res/hashnode/image/upload/v1702374391910/56d50482-01da-4bb3-a513-78e6b3bcd4d7.png\")
Navigating Challenges, Embracing Victories:
\nPublishing on Docker Hub:
\ndocker login
docker tag dockfolio pranjalbarnwal/dockfolio:latest
docker push pranjalbarnwal/dockfolio:latest
![\"Push](\"https://cdn.hashnode.com/res/hashnode/image/upload/v1702374187200/34d55858-5e74-4cdb-8f55-ec6e3544aa96.png\")
With this, our image is live and public at Docker Hub.
\n![\"Image](\"https://cdn.hashnode.com/res/hashnode/image/upload/v1702367467166/4fdf8bcc-c657-4902-982e-d2e04f243720.png\")
Now anyone will be able to fetch the image and run the same image using below commands
\ndocker pull pranjalbarnwal/dockfolio
docker run -d -p 8080:80 pranjalbarnwal/dockfolio
\n\nOnce done, website will be live at: localhost:8080
\nNOTE: if you're running Docker on server, then replace localhost above with your ip address ex:
\nhttp://54.80.107.150:8080
![\"Deployment](\"https://cdn.hashnode.com/res/hashnode/image/upload/v1702367398185/496e4ee7-b075-4a80-8a65-53ce6fef8898.png\")
dockfolio based on community suggestions, evolving it into a tool that resonates with developers worldwide.\nFeel free to bring improvements and dive into the sea of Open-source: RepoConclusion: \"Dockfolio\" encapsulates a voyage-beyond just an image-of persistence, learning, and community camaraderie. Dive into its world, contribute, and immerse yourself in the transformative domain of containerization and efficient development workflows.
\nFinal Thoughts: As this chapter concludes, heartfelt gratitude is extended to the vibrant Docker community for their unwavering support and collective wisdom. Remember, every Docker command, every interaction, fosters collaboration and fuels innovation.
\n![\"Line](\"https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1q91urxsd6tfphio44tc.jpg\")
Line breaks play an essential role in text formatting, yet they are often overlooked in the programming world. Operating systems and text editors have their conventions for line breaks, leading to variations in line break types. This article will explore the differences between three common line break types: CR LF, LF, and CR.
\nThe CR LF line break type is commonly used in Windows operating systems and DOS-based text files. It combines two characters: Carriage Return (CR) and Line Feed (LF). Let’s delve into the significance of each of these characters:
\nCarriage Return (CR): Represented as \\r, this character moves the cursor to the beginning of the current line without advancing to the next line. It's like pushing the typewriter carriage back to the left.
\nLine Feed (LF): Represented as \\n, this character moves the cursor down to the next line without affecting the horizontal position. It's like moving the typewriter carriage down to the next line.
\nIn Windows and DOS text files, the combination of CR and LF characters indicates a line break. The CR LF sequence \\r\\n is commonly used to represent line breaks in these systems.
\n![\"Line](\"https://miro.medium.com/v2/resize:fit:482/1*_3VfbLPRpQpaTKgNtwdPMA.png\")
The LF line break type is predominantly used in Unix, Linux, macOS, and modern text editors, including those for web development. In this convention, a single Line Feed character (\\n) represents a line break. Unlike CR LF, there is no preceding Carriage Return character. The LF line break type solely relies on the line feed character to move to the next line.
\n![\"End](\"https://miro.medium.com/v2/resize:fit:191/1*yOUPzH_4vVs6oxnAF9PfWA.png\")
The CR line break type is rarely used in modern systems but holds historical significance in older Macintosh operating systems. In this convention, a single Carriage Return character (\\r) represents a line break. Similar to LF, there is no subsequent Line Feed character in the CR line break type.
\n![\"Have](\"https://miro.medium.com/v2/resize:fit:608/1*J_MSqJ_zRPOIjAXqBWDH-Q.gif\")
Understanding the different line break types is crucial when working with text files intended for sharing across multiple platforms. It is essential to consider compatibility issues and ensure the intended systems correctly interpret the line break types.
\nWindows applications and text editors generally handle CR LF line breaks correctly.
\nUnix, Linux, and macOS systems expect LF line breaks, and modern text editors seamlessly handle them.
\nCross-platform projects or collaborations between different systems may require graceful handling of different line break types to avoid formatting issues.
\nWhile line breaks may appear insignificant, understanding the differences between CR LF, LF, and CR line break types is crucial for achieving seamless cross-platform compatibility. Whether you’re working on Windows, Unix, Linux, macOS, or modern text editors, being aware of line break conventions ensures consistent and accurate rendering of text files.
\nBy adopting best practices and selecting the appropriate line break type for your target platform, you can ensure optimal readability and portability of your text files.
\nNow equipped with the knowledge of line break types, you can confidently handle text formatting and avoid any surprises in your coding and development endeavors.
\n\n\nHappy coding!
\n
![\"\"](\"https://miro.medium.com/v2/resize:fit:788/1*HfiLf5jlA2QuMX6N7mpKqw.gif\")
![\"Flutter](\"https://dev-to-uploads.s3.amazonaws.com/uploads/articles/88u4offseko5xx3gmwi9.jpg\")
In Flutter, widgets have a lifecycle that determines how they are created, updated, and disposed of. Understanding the widget lifecycle is essential for managing state, performing side effects, and optimizing your Flutter application. Let’s dive into an overview of the widget lifecycle in Flutter:
\n![\"Widget](\"https://dev-to-uploads.s3.amazonaws.com/uploads/articles/al3bq7afxyvkblzyeby3.png\")
When you instantiate a widget by calling its constructor, the widget is created. During construction, you typically provide any required parameters and initialize the widget’s state. The constructor is called once for each instance of the widget.
\nAfter construction, the framework calls the initState() method of the widget's corresponding State object. This is where you can perform any initialization tasks that require the widget to have access to its BuildContext. You can initialize variables, subscribe to streams, set up listeners, and perform other setup tasks.
\nThe build() method is called whenever the widget needs to be rendered or updated. This method returns a widget tree, describing how the UI should look based on the current state of the widget. The framework invokes build() initially and subsequently whenever the widget's state changes or when its parent requests a rebuild.
\nWhen the widget’s state changes, either through user interaction, data changes, or other events, the framework calls the setState() method of the widget's State object. This triggers a rebuild of the widget, causing the framework to call build() again. By calling setState(), you signal that the widget's state has changed and should be reflected in the UI.
\nWhen the framework determines that a widget needs to be rebuilt, it calls the build() method to obtain a new widget tree. The new tree is compared to the previous tree, and the framework applies the necessary updates to the UI to reflect the changes. The rebuild process is efficient because Flutter uses a diffing algorithm to determine the minimal changes required to update the UI.
\nIf a widget is removed from the widget tree but might be added back later, the framework calls the deactivate() method of the widget's State object. This method allows the widget to clean up any resources it no longer needs while preserving its state. For example, you can pause animations, cancel subscriptions, or release resources.
\nWhen a widget is removed from the widget tree permanently, the framework calls the dispose() method of the widget's State object. This is the final opportunity for the widget to release resources, cancel subscriptions, or perform any cleanup tasks. After dispose() is called, the widget and its state are no longer usable.
\nIt’s important to note that widgets in Flutter are immutable. When a widget’s state changes, Flutter rebuilds the widget tree, creating a new set of widgets with the updated state. This approach allows for efficient UI updates and helps maintain a declarative programming style.
\nUnderstanding the widget lifecycle enables you to manage state appropriately, handle asynchronous operations, and optimize your Flutter app’s performance by minimizing unnecessary rebuilds and resource usage.
\n![\"Flutter](\"https://miro.medium.com/v2/resize:fit:788/1*gZraN2IEy8I1VqxbZF_zyA.png\")
The lifecycle of a stateful widget in Flutter consists of seven cycles. Understanding these cycles is essential for managing the state and controlling the behavior of the widget. Let’s explore each cycle:
\ncreateState(): This method is required and creates a State object for the widget. It holds all the mutable state for that widget. The State object is associated with the BuildContext by setting the mounted property to true.
\ninitState(): This method is automatically called after the widget is inserted into the tree. It is executed only once when the state object is created for the first time. Use this method for initializing variables and subscribing to data sources.
\ndidChangeDependencies(): The framework calls this method immediately after initState(). It is also called when an object that the widget depends on changes. Use this method to handle changes in dependencies, but it is rarely needed as the build method is always called after this.
\nbuild(): This method is required and is called many times during the lifecycle. It is called after didChangeDependencies() and whenever the widget needs to be rebuilt. Update the UI of the widget in this method.
\ndidUpdateWidget(): This method is called when the parent widget changes its configuration and requires the widget to rebuild. It receives the old widget as an argument, allowing you to compare it with the new widget. Use this method to handle changes in the widget's configuration.
\nsetState(): The setState() method notifies the framework that the internal state of the widget has changed and needs to be updated. Whenever you modify the state, use this method to trigger a rebuild of the widget's UI.
\ndeactivate(): This method is called when the widget is removed from the widget tree but can be reinserted before the current frame changes are finished. Use this method for any cleanup or pausing ongoing operations.
\ndispose(): This method is called when the State object is permanently removed from the widget tree. Use this method for cleaning up resources, such as data listeners or closing connections.
\n![\"Order](\"https://miro.medium.com/v2/resize:fit:539/1*2JKqkXKuRptjg3zm6X0r4Q.png\")
After the dispose() method, the State object is no longer in the tree, and the mounted property is set to false. The state object cannot be remounted.
Understanding these lifecycle cycles helps in managing state changes, handling dependencies, and updating the UI effectively in stateful widgets.
\nNote: The constructor function is not part of the lifecycle as the state of the widget property is empty during that time.
\nNow that you have a comprehensive understanding of the widget lifecycle in Flutter, you can harness its power to create robust and efficient Flutter applications.
", "url": "https://blogs.pranjalkumar.in/blog/widget-lifecycle-flutter", "title": "The journey of a widget: understanding the lifecycle in Flutter", "summary": "How Flutter widgets are created, updated, and disposed - a walkthrough of the stateful widget lifecycle, state management hooks, and performance implications.", "date_modified": "2023-05-29T00:00:00.000Z", "author": { "name": "Pranjal Kumar", "url": "https://www.linkedin.com/in/pranjal--kumar" }, "tags": [ "flutter", "android" ] }, { "id": "https://blogs.pranjalkumar.in/blog/deploy-firestore-in-react-on-firestore", "content_html": "![\"React](\"https://cdn.hashnode.com/res/hashnode/image/upload/v1671905247518/9c2bcdf1-2f6c-40fc-acc5-82640c2a599b.jpeg\")
Unlock the power of Firebase and Firestore integration to seamlessly deploy your React projects. Streamline your development workflow and harness the scalability of Firebase for efficient deployment.
\nBefore starting with anything make sure you have created your project in FireBase & you have used the same Firebase Config in your `firebase.js` file in your local react project folder
\nnpm run build
![\"Building](\"https://cdn.hashnode.com/res/hashnode/image/upload/v1671902084976/77c99ea9-f4dd-4c8e-acbd-d9c6bcd79b9d.png\")
This in turn will create a build folder in the root of your project folder.
\n![\"Folder](\"https://cdn.hashnode.com/res/hashnode/image/upload/v1671902782212/f3c67aee-7837-48c2-99b9-ef8b70ca294c.png\")
Basically the build files are optimized for production, where there are no ongoing changes in code every now & then.
\nnpm install -g firebase-tools
![\"Installing](\"https://cdn.hashnode.com/res/hashnode/image/upload/v1671902182782/63e161e6-ad4f-4fa3-826d-efa8ebfc7ed7.png\")
`-g` flag is used here to install it globally, so it can be accessed easily from any project.
\nfirebase login
![\"Logged](\"https://cdn.hashnode.com/res/hashnode/image/upload/v1671902368435/dc6c5d46-028b-4ce7-bc59-036343589ed5.png\")
For logging in, you will be redirected to browser window, there you simply have to login using your FireBase credentials.
\nfirebase init
You will be asked loads & loads of questions while in `firebase init` but don't worry, I will guide you here:
\n![\"Firebase](\"https://cdn.hashnode.com/res/hashnode/image/upload/v1671902954507/03d58c8d-edb4-466c-8a01-8f5e9c60a031.png\")
Are you ready to proceed?
\nType `y`, since we want to continue.
\nWhich Firebase features do you want to set up for this directory? Press Space to select features, then Enter to confirm your choices.
\nHere, you need to select all features which you included in your project, it may be, you needed nothing & you are just using it for free Hosting. :) In my case, I used Firestore & Storage both in my project & also needed Hosting so, I'm selecting these 3 using space & then press enter.
\n![\"Select](\"https://cdn.hashnode.com/res/hashnode/image/upload/v1671905433004/1563a5cf-7170-4492-ba47-1f11e0cc1b98.png\")
Make sure to choose the Firebase Hosting, where Github actions are optional.(Refer green stars on above picture).
\nWhen prompted for Firebase rules and other similar stuff, then go with default options.
\n![\"Continue](\"https://cdn.hashnode.com/res/hashnode/image/upload/v1671904330866/041b8619-bb08-498e-b2d8-2a1f8298ad29.png\")
For choosing public directory, make sure you choose the build folder
![\"Crosscheck](\"https://cdn.hashnode.com/res/hashnode/image/upload/v1671904645045/722ca2fe-8586-4fd8-ae5c-708de194bca7.png\")
Go with these options. Make sure you don't overwrite buid/index.html.
![\"Select](\"https://cdn.hashnode.com/res/hashnode/image/upload/v1671904709223/bfc85f34-558f-4ee3-aefe-815e0d4ac003.png\")
firebase deploy
![\"Building](\"https://cdn.hashnode.com/res/hashnode/image/upload/v1671904878075/6eddb40a-14a6-41dd-a966-f7c970571e8b.png\")
![\"Build](\"https://cdn.hashnode.com/res/hashnode/image/upload/v1671904911421/12d9c020-4d17-453a-b6a6-840ac597f45b.png\")
That's it we're done
\nLink for the Deployed application will be in the Terminal
\n