Pandas in 8 Slides

This cover sets up the mistakes post with the most important warning about Pandas: the dangerous errors don't raise exceptions, they hand you a confidently wrong answer. A loud crash is easy to fix; a silent corruption of your results can ship undetected.

The post is a field guide. For each common trap we name the symptom, explain the underlying cause, and give the habit that prevents it. These are the errors that catch beginners and experienced analysts alike, precisely because Pandas is permissive enough to let them happen quietly.

The first trap is the SettingWithCopy problem. Writing something like df[df.x > 0]["y"] = 1 performs chained indexing: the first part produces a temporary intermediate object, and the assignment modifies that temporary rather than df itself. Pandas often can't guarantee which, so it emits the SettingWithCopyWarning and your change may silently fail to stick.

The cause is the ambiguity between a view and a copy when you chain two indexing operations. The cure is to collapse the selection and assignment into a single .loc call that Pandas can unambiguously resolve to the original frame. Once you adopt single-step .loc assignment as a habit, this entire class of bug disappears.

This code slide gives the concrete fix for the SettingWithCopy trap. The wrong version, df[df["x"] > 0]["y"] = 1, chains a boolean filter and a column selection before assigning, which may modify a throwaway copy. The right version, df.loc[df["x"] > 0, "y"] = 1, expresses the row condition and the target column inside a single .loc indexer.

The single .loc form is unambiguous: it tells Pandas exactly which rows and which column of the original frame to write to, so the assignment lands where you intend. Make this your default pattern for conditional assignment and you'll never see the warning again.

The second trap is reaching for a loop when a vectorized operation exists. Using iterrows() or an explicit for-loop to build a column is both the most common performance mistake and one of the most common readability problems in beginner Pandas code.

The fix is almost always available: a vectorized expression, a built-in method, or a helper like np.where does the same job in one line and runs orders of magnitude faster by pushing the work into compiled code. The habit to build is, before writing any loop over rows, to pause and ask whether a whole-column operation could do it instead.

This code slide shows the vectorized cure for looping. The slow version uses a list comprehension over itertuples() to assign a tier label row by row. The fast version, np.where(df["spend"] > 100, "hi", "lo"), evaluates the condition across the entire column at once and selects between the two labels element-wise.

np.where is the vectorized equivalent of an if/else applied to a column, and it's the go-to tool for conditional column derivation. Beyond speed, the vectorized line is also shorter and reads as a direct statement of intent, reinforcing that in Pandas the fast path and the clear path usually coincide.

The third trap is misunderstanding NaN. By IEEE floating-point rules, NaN is not equal to anything, including itself — NaN == NaN evaluates to False. As a direct consequence, filtering with df[df.x == np.nan] returns no rows, even when the column is full of missing values.

The correct tools are the dedicated missing-value methods: df.x.isna() to detect gaps and fillna to replace them. A related subtlety is that many aggregations skip NaN by default, which can quietly change the denominator of a mean or the count behind a sum. Knowing how NaN behaves prevents both empty filters and silently shifted statistics.

This trace diagram contrasts the broken and correct ways to handle missing values. Comparing with == np.nan returns all False because NaN never equals anything — the diagram's comment line drives that home. Calling .isna() correctly marks the gaps, and .fillna(0) replaces them with a chosen value.

Reading the trace top to bottom shows the progression from the wrong instinct to the right tools. The visual is meant to cement a reflex: whenever you need to find or handle missing data, reach for .isna() and .fillna(), never for an equality comparison against NaN.

The fourth trap concerns the inplace parameter. Writing df = df.dropna(inplace=True) is a double mistake: inplace methods mutate the frame and return None, so the assignment overwrites df with None and you lose your data entirely.

There are two correct forms. You can use inplace=True with no assignment, letting the method mutate df directly, or — the clearer and increasingly recommended style — drop inplace altogether and assign the returned frame with df = df.dropna(). The assign-the-result style is easier to read, composes well into method chains, and avoids the None pitfall completely.

This code slide shows the fix for the inplace mistake. The wrong line, df = df.dropna(inplace=True), assigns None to df. The right line, df = df.dropna(), captures the cleaned frame that dropna returns. The slide also notes that df.dropna(inplace=True) without assignment is acceptable, just harder to chain.

The broader guidance is to prefer assignment over inplace across Pandas. It produces uniform, predictable code where every operation returns a new value you explicitly keep, sidesteps the None trap, and reads naturally as a pipeline of transformations — which is how idiomatic modern Pandas is written.

The fifth trap is confusing loc and iloc. loc indexes by label while iloc indexes by integer position, and when your index happens to be the default integers they appear interchangeable — until the index is filtered, sorted, or otherwise reordered, at which point label and position no longer coincide.

A sharp-edged detail is slicing behavior: df.loc[0:2] is label-based and inclusive of 2, returning three rows, while df.iloc[0:2] is position-based and excludes 2, returning two. Choosing deliberately between label and position addressing, and remembering the slice-endpoint difference, prevents subtle off-by-one and wrong-row errors.

This decision tree is a quick triage for the most common Pandas confusion. If you're getting a wrong result or a warning, first ask whether you're assigning into a filtered slice — if so, switch to a single df.loc[mask, col] = v. If the trouble isn't assignment, the next most likely culprit is missing-value handling, so check NaNs with .isna() rather than an equality comparison. If neither applies, nothing is wrong and you can continue.

Keeping this short flow in mind turns most everyday Pandas incidents into a fast, deterministic fix instead of an open-ended debugging session, especially the two failure modes that account for the majority of silent wrong answers.

These bullets collect the preventive habits in one place: assign with a single .loc, vectorize instead of using iterrows, find NaN with .isna() rather than ==, assign results instead of trusting inplace, and remember that loc is label-based while iloc is position-based.

None of these habits is difficult, and adopting them as defaults is what separates analysts who occasionally ship wrong numbers from those who essentially never do. The mistakes are predictable and repetitive, which is exactly what makes them preventable once you've seen them named.

This closes Day 25 and the topic of Pandas. With the core objects, the rationale, the internal mechanics, a full worked example, and the common mistakes all covered, you have a complete working command of Pandas for real data tasks. The CTA points to Day 26, which begins a new topic, so following keeps the series going.