The SQL You Actually Need for Data Science (and the 20% That's 80% of the Job)

# The SQL You Actually Need for Data Science (and the 20% That's 80% of the Job) Let's get the obvious thing out of the way: I tutor Python and data science for a living, so you'd expect me to tell you to go learn the whole of SQL, every clause, every dialect quirk, all of it. I'm not going to do that, because it isn't true and it would waste your time. So here's the actual answer: **for data science, about 20% of SQL covers roughly 80% of what you'll do day-to-day and in interviews. That 20% is SELECT/WHERE/GROUP BY/HAVING, the JOIN types (and knowing why LEFT vs INNER changes your row counts), aggregation, CASE, CTEs, and window functions. Window functions are the real dividing line between someone who studied SQL and someone who uses it.** That's the whole thesis. Everything below is me showing you each piece with SQL you can actually run, plus an honest note at the end about what you can safely ignore for now. Let's go through them. ## The foundation: SELECT, WHERE, GROUP BY, HAVING Almost every query you'll write starts here. You pick columns, filter rows, group them, and aggregate. The thing most beginners get wrong is the difference between `WHERE` and `HAVING`, so let's nail that early. `WHERE` filters individual rows *before* grouping. `HAVING` filters *after* grouping, on the aggregated result. They are not interchangeable. ```sql SELECT country, COUNT(*) AS num_orders, SUM(amount) AS total_revenue FROM orders WHERE status = 'completed' -- filters rows first GROUP BY country HAVING SUM(amount) > 10000 -- filters groups after aggregating ORDER BY total_revenue DESC; ``` Read that out loud: take completed orders, group them by country, count and sum within each country, then keep only the countries whose total revenue cleared 10,000. If you tried to put `SUM(amount) > 10000` in the `WHERE` clause, the database would reject it, because the sum doesn't exist yet at the row-filtering stage. That single distinction comes up constantly, and interviewers love it precisely because it tells them whether you actually understand the order things happen in. While we're here, the logical order a query executes in is roughly: `FROM` -> `WHERE` -> `GROUP BY` -> `HAVING` -> `SELECT` -> `ORDER BY`. You don't need to memorise this like a poem, but it explains a lot of "why can't I use my column alias in WHERE?" confusion. The alias is created in `SELECT`, which happens after `WHERE`, so it isn't available yet. ## JOINs, and the LEFT-vs-INNER gotcha that bites everyone In any real dataset your information lives in more than one table. Customers in one, orders in another, products in a third. Joining is how you stitch them back together, and it's where I see the most expensive mistakes, because a bad join doesn't error out. It just quietly gives you the wrong numbers. The four you actually use: - `INNER JOIN` keeps only rows that match in both tables. - `LEFT JOIN` keeps every row from the left table, filling in `NULL` where there's no match on the right. - `RIGHT JOIN` is the mirror of LEFT, and honestly I almost never use it, you can rewrite any RIGHT join as a LEFT join by swapping the table order, which reads better. - `FULL OUTER JOIN` keeps everything from both sides. Less common, but worth knowing. Here's the gotcha that I make every student trip over once so they never forget it. Suppose you have 1,000 customers, and 300 of them have placed at least one order. ```sql -- INNER JOIN: only customers who have orders SELECT c.customer_id, o.order_id FROM customers c INNER JOIN orders o ON o.customer_id = c.customer_id; ``` ```sql -- LEFT JOIN: ALL customers, orders attached where they exist SELECT c.customer_id, o.order_id FROM customers c LEFT JOIN orders o ON o.customer_id = c.customer_id; ``` The `INNER JOIN` gives you one row per order, and only for the 300 customers who ordered. The `LEFT JOIN` gives you those same order rows *plus* one row for each of the 700 customers who never ordered, with `NULL` in the order columns. Different row counts, different answer to "how many customers do we have." The classic real-world version: someone wants "total revenue per customer, including customers who spent nothing." If you `INNER JOIN` and then `SUM`, the zero-spend customers vanish entirely and your averages are silently wrong. You need a `LEFT JOIN` so they show up as zero. ```sql SELECT c.customer_id, COALESCE(SUM(o.amount), 0) AS lifetime_value FROM customers c LEFT JOIN orders o ON o.customer_id = c.customer_id GROUP BY c.customer_id; ``` Note the `COALESCE` — without it, customers with no orders get `NULL`, not `0`, because `SUM` of nothing is `NULL`. Small detail, real bug. My rule of thumb: whenever you join, ask yourself "did my row count change, and did I expect it to?" If you can answer that honestly every time, you're ahead of most people. ## Aggregation and CASE: counting things the way you actually mean `COUNT`, `SUM`, `AVG`, `MIN`, `MAX` are the bread and butter. Two things worth saying out loud, because both come up in interviews. First, `COUNT(*)` counts rows, `COUNT(column)` counts non-`NULL` values in that column, and `COUNT(DISTINCT column)` counts unique non-`NULL` values. Those are three different numbers and mixing them up is a common quiet error. Second, `CASE` is how you do "if/else" inside SQL, and combining it with aggregation is one of the most useful tricks you'll learn. It lets you pivot and bucket without leaving the query. ```sql SELECT customer_id, COUNT(*) AS total_orders, COUNT(DISTINCT product_id) AS distinct_products, SUM(CASE WHEN amount > 100 THEN 1 ELSE 0 END) AS big_orders, AVG(CASE WHEN status = 'completed' THEN amount END) AS avg_completed_value FROM orders GROUP BY customer_id; ``` That `SUM(CASE WHEN ... THEN 1 ELSE 0 END)` pattern is a conditional count, count the rows that meet a condition, and you'll use it constantly. The last line is sneakier: by returning `NULL` for non-completed orders (there's no `ELSE`), those rows simply don't contribute to the average, because `AVG` ignores `NULL`. That's conditional aggregation, and once it clicks you'll reach for it all the time. `CASE` also handles plain bucketing, the kind of thing you'd otherwise do in pandas after the fact: ```sql SELECT CASE WHEN age < 18 THEN 'minor' WHEN age BETWEEN 18 AND 64 THEN 'adult' ELSE 'senior' END AS age_group, COUNT(*) AS n FROM users GROUP BY 1; -- group by the first SELECT expression ``` ## Subqueries vs CTEs: same power, very different readability A subquery is a query nested inside another query. A CTE (Common Table Expression, the `WITH` clause) is a named, temporary result you define up front and then reference. Functionally they overlap a lot. The difference that matters in practice is that CTEs read like a sequence of steps, and nested subqueries read like a riddle. Here's the same logic both ways. Find customers whose total spend is above the overall average. The subquery version: ```sql SELECT customer_id, total_spend FROM ( SELECT customer_id, SUM(amount) AS total_spend FROM orders GROUP BY customer_id ) AS per_customer WHERE total_spend > ( SELECT AVG(total_spend) FROM ( SELECT SUM(amount) AS total_spend FROM orders GROUP BY customer_id ) AS t ); ``` Notice I had to compute per-customer totals *twice* and you have to read it inside-out. Now the CTE version: ```sql WITH per_customer AS ( SELECT customer_id, SUM(amount) AS total_spend FROM orders GROUP BY customer_id ) SELECT customer_id, total_spend FROM per_customer WHERE total_spend > (SELECT AVG(total_spend) FROM per_customer); ``` Same result, but you define `per_customer` once, name it something meaningful, and read top to bottom like a recipe. You can chain several CTEs, each building on the last, which is how you keep a genuinely complex transformation legible. This is why I push students toward CTEs early. They're not more powerful, they're more *humane*, and on a team that matters because someone (often future you) has to read this in six months. The one honest caveat: there are still cases where a correlated subquery or an `EXISTS` check is the natural tool, especially for "does any matching row exist" questions: ```sql SELECT c.customer_id FROM customers c WHERE EXISTS ( SELECT 1 FROM orders o WHERE o.customer_id = c.customer_id AND o.amount > 500 ); ``` So learn both. Reach for CTEs by default, keep subqueries and `EXISTS` for when they genuinely read better. ## Window functions: the line between "studied SQL" and "uses SQL" Here's the part I care most about. If you can write window functions comfortably, you are doing something the majority of people who "know SQL" cannot, and it shows immediately in interviews and on the job. A window function performs a calculation across a set of rows *related to the current row*, without collapsing them into one row the way `GROUP BY` does. That's the key insight: with `GROUP BY`, ten rows become one. With a window function, you keep all ten rows and attach a computed value to each. The `OVER (...)` clause defines the "window" of rows to look at. The three ranking functions everyone should know, and the difference between them, are a classic interview question: ```sql SELECT product_id, category, revenue, ROW_NUMBER() OVER (PARTITION BY category ORDER BY revenue DESC) AS rn, RANK() OVER (PARTITION BY category ORDER BY revenue DESC) AS rnk, DENSE_RANK() OVER (PARTITION BY category ORDER BY revenue DESC) AS dense_rnk FROM products; ``` `PARTITION BY category` means "restart the numbering for each category." Then, on ties: - `ROW_NUMBER` gives every row a distinct number, even on ties (1, 2, 3, 4), breaking ties arbitrarily. - `RANK` gives ties the same number, then skips (1, 1, 3, 4). - `DENSE_RANK` gives ties the same number but doesn't skip (1, 1, 2, 3). Knowing those three apart is genuinely one of the most common SQL interview questions there is, so it's worth being able to explain it in a sentence. The single most useful pattern built on this is **"the top N per group."** "Give me the top-selling product in each category." With `GROUP BY` alone this is awkward. With a window function it's clean: rank inside a CTE, then filter. ```sql WITH ranked AS ( SELECT product_id, category, revenue, ROW_NUMBER() OVER (PARTITION BY category ORDER BY revenue DESC) AS rn FROM products ) SELECT product_id, category, revenue FROM ranked WHERE rn = 1; -- top product per category; use rn <= 3 for top 3 ``` You can't put a window function directly in a `WHERE` clause (it's computed after `WHERE`), which is exactly why the CTE-then-filter structure is the standard idiom. Worth remembering. The other everyday pattern is the **running total**, where you add a frame to the window: ```sql SELECT order_date, amount, SUM(amount) OVER ( ORDER BY order_date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ) AS running_total FROM orders; ``` And `LAG` / `LEAD`, which let each row see the previous or next row, perfect for month-over-month change or measuring gaps between events: ```sql SELECT order_date, amount, LAG(amount) OVER (ORDER BY order_date) AS prev_amount, amount - LAG(amount) OVER (ORDER BY order_date) AS change_vs_prev FROM orders; ``` If you only deeply learn one "advanced" thing from this whole post, make it window functions. They're the highest-leverage chunk of SQL for data science, full stop. So much of analytics is "rank these, compare each row to its neighbour, accumulate over time," and these are exactly the tools for it. ## What you honestly don't need yet Now the honest part, the stuff people stress about that doesn't earn its keep early on. You don't need query-planner internals. Reading `EXPLAIN` output, understanding hash joins vs merge joins, reasoning about the optimiser, that's real and useful *eventually*, but it's not what's blocking you from doing data science today. You don't need to hand-tune indexes on day one. Knowing roughly what an index is helps, but designing index strategy is a job that arrives when you own a production database, not when you're querying one to pull a dataset. You don't need exotic, database-specific features, recursive CTEs, fancy JSON operators, full-text search, every dialect's special functions. Wonderful when you need them, a distraction when you don't. And honestly, the differences between PostgreSQL, MySQL, SQLite and the cloud warehouses matter far less at the start than tutorials imply. The core 20% above is portable across basically all of them. You don't need to choose between SQL and pandas, either. They're complements. A common, sane real-world pattern is to do the heavy filtering, joining and aggregating in SQL (close to the data, where it's fast) and pull a smaller, focused result into Python and pandas for modelling and visualisation. Knowing where that line sits is itself a skill, and it's a much better use of your attention than memorising obscure syntax. ## How to practise (the part that actually makes it stick) Reading SQL is not learning SQL. You have to write it, get it wrong, and see the row counts move. Here's what I'd actually do: 1. **Get a real database in front of you.** Install SQLite (it's a single file, zero setup) or PostgreSQL, and load a dataset you find a little interesting, your own data, a Kaggle CSV, anything with at least two tables you can join. Caring even slightly about the data makes a real difference. 2. **Re-derive every pattern above from scratch.** Don't copy mine. Write your own top-N-per-group, your own running total, your own conditional count. If you can reproduce them without looking, you own them. 3. **Grind window-function and JOIN questions specifically.** Sites like StrataScratch, DataLemur and LeetCode's database section are full of real interview questions, and they're disproportionately about exactly the 20% in this post. That's not a coincidence. 4. **Always sanity-check your row counts.** After every join, ask whether the count changed and whether you expected it to. This one habit catches more real bugs than any amount of theory. That's the honest list. Master that core and you'll out-SQL plenty of people with fancier-sounding resumes, because they studied the whole language and never got fluent in the part that does the work. If you'd like a hand getting fluent, the window-function patterns especially, this is exactly the kind of thing I tutor one-on-one, and the first session is free so we can see whether it clicks for you. Either way: install something, load a dataset, and start writing queries today. That's the whole secret.