How pandas explode list to rows works when there are both lists and non-lists

pandas explode turns each list in a chosen column into separate rows when both lists and non-lists are present. Non-list values become a single row unchanged, and NaN or None remain a single NaN in the output. That means mixed columns do not raise an error; they silently produce a mix of expanded and unexpanded rows unless you drop them first.

The operation is a row-wise broadcast: for one list, explode preserves that list's order. It also resets the index by repeating each prior index label across its expanded rows, which is why a reset_index call usually follows. The result is convenient, but it hides how types get coerced and which rows disappear.

Mechanism: how explode handles mixed types

pandas implements explode by treating each element of the target column as either an iterable or a scalar. A list, tuple, set, or similar sequence gets unrolled. A scalar, meaning a bare string, number, or timestamp, is treated as a length-one sequence. The function walks the column once, builds an index with the original row labels repeated per sequence length, and constructs a new Series. Non-iterables are simply wrapped rather than rejected.

The coercion happens quietly. A string is technically iterable in Python, so older pandas versions had to guard against exploding a string into characters. Current pandas checks for list-likes specifically and treats strings and dicts as scalars. The result is that a column holding Python lists alongside dicts or strings expands only the lists, and the dicts and strings appear as single rows.

Index behavior is the part most people miss. explode keeps the original index label on every expanded row and does not renumber. If row 7 holds a three-element list, the output has three rows all labeled 7. Call reset_index(drop=True) afterward to get a clean RangeIndex. Without it, downstream groupby and merge operations may double-count or fail.

A key edge is that NaN and None are not treated as empty lists. A true empty list produces zero rows, so the original index label vanishes from the output entirely. A NaN produces exactly one row with NaN. If your workflow assumes "every input row yields at least one output row," the empty list is the silent exception.

Quantified context: sizes, counts, and time costs

The practical effect is best seen in figures. A column with 10 rows where 6 contain lists of length 3 and 4 are scalars yields 6×3 + 4×1 = 22 output rows. The row count is the sum of lengths where lists count their length and scalars count 1. This is a straightforward arithmetic check you can run before or after.

Memory scales with the expanded row count, not the input row count. If your input frame is 500,000 rows and the average list length is 4, you can expect roughly 2 million rows in memory. At pandas defaults, a numeric column costs about 8 bytes per value for int64 or float64, so 2 million rows of a single numeric column is roughly 16 megabytes. Multiplied across 20 columns with object dtype, memory can climb into hundreds of megabytes.

For a typical 100,000-row frame with an average of 5 items per list, explode commonly runs in a fraction of a second on modern hardware, while the same operation in pure Python loops takes seconds to tens of seconds. The speedup comes from the vectorized index construction. That gain evaporates if the column holds deeply nested structures, since each element still needs a Python-level interpretation.

Edge cases: NaN, empty lists, and odd shapes

Three inputs cause more bugs than any others: NaN, empty lists, and inconsistent element types.

An empty list produces no output row. If that column is a join key or an ID carrier, you lose the record without a warning. If you need a placeholder, replace empty lists with [None] before exploding, or filter and handle them separately.

NaN and None survive as one NaN row. A common trap is replacing NaN with an empty list via fillna([]), which then deletes the row. Instead, replace NaN with [np.nan] or [None] so the row is preserved as an explicit missing entry.

Nested lists do not explode recursively. A list-of-lists yields one row per outer element, each still a list. To flatten, explode repeatedly until the column contains scalars, but each pass multiplies row counts, so track totals. Sets and tuples expand in arbitrary or ordinal order respectively; sets have no reliable order, so do not depend on it.

A subtle type issue: mixed numeric and string elements inside the same list produce object dtype output. That blocks numeric aggregation until you convert, and a single null or mixed value is often enough to prevent a conversion error you would rather catch early.

Misconceptions: what explode does not do

The most common belief is that explode drops NaN by default. It does not. NaN becomes a single NaN row, and the dropna call is separately required if you want to remove those rows.

Another misconception is that explode reindexes for you. It does not; index labels repeat and must be reset explicitly when consumers expect a unique index.

A third is that strings expand into characters. They do not in modern pandas; strings are scalars. However, a column of list-like strings such as "[1, 2, 3]" will not be parsed automatically, so you need to convert with literal evaluation or JSON parsing first. If you skip that, you get a single row containing the raw string.

The professional caveat: explode does not validate that the list contains homogeneous, serializable values. It copies references. If your lists contain mutable objects, modifying a row in the exploded result may affect the source object in memory, which is a silent aliasing bug in long pipelines. Copy or deep-copy when that matters.

Finally, the operation is not idempotent on an already exploded column. Running explode twice on a column that no longer contains lists yields the same rows, but running it on a column that still has lists splits further. Track state explicitly in reusable functions.

Practical actions: steps and thresholds

Before exploding, run these checks in order.

Inspect dtype with df['col'].apply(type).value_counts() to confirm which rows are list-like versus scalar. Count expected output rows as sum(map(len, lists)) + count(scalars), and compare to memory headroom. Decide a policy for empty lists: drop, replace with [None], or fail loudly. Replace NaN with [None] rather than an empty list if the row must survive. Reset index immediately after explode with reset_index(drop=True). Convert mixed object columns to a declared dtype before any aggregation.

Order matters. If you plan to merge on an ID column, reset the index first so you can group back to the original row. If you plan to aggregate, verify dtype after explode and before groupby. A threshold worth remembering: if the exploded frame exceeds roughly half your available memory, switch to a chunked approach (explode in slices of 100,000 rows) or use a library like polars whose list.explode handles the same operation with lower overhead.

For empty-list policy, a safe default is to replace with [None], apply explode, then apply dropna(how='all') only on the columns you intentionally allow to vanish. This makes disappearance a decision, not a side effect.

Limitations and reconciliation

explode's treated-scalar behavior and its empty-list row deletion are two behaviors that appear contradictory until explained. A scalar becomes one row because pandas wraps it as a length-one sequence; an empty list becomes zero rows because its length is zero. Both follow from the same length arithmetic, but only the first feels obvious. When people say "explode drops some rows," they are almost always describing the empty-list case, not scalars.

The practical reconciliation is to normalize your column before exploding. Replace empty lists with [None] if preservation is required, replace NaN with [None] rather than [], and convert anything list-like from strings to actual lists first. After explode, reset the index and check dtype. That sequence removes every silent disagreement between list rows, scalar rows, and missing rows.

One final limit: explode performance and memory depend on total element count, not on row count or list length alone. A frame with many short lists and a frame with fewer long lists can cost the same to expand. Profile on real data before assuming a plan scales. And because explode copies element references, avoid mutating the exploded output in place when the source lists came from another frame you still need.

For most pipelines, the safest pattern is four lines: normalize the column, explode, reset index, cast. Those four steps are the difference between a clean long-format frame and one that silently loses rows or holds dtype surprises. It is worth writing them into a helper function and testing it against a fixture that mixes lists, scalars, NaN, and empty lists deliberately, because that fixture will catch the empty-list deletion, the NaN retention, and the index duplication in one run.