Python Data Analysis and Visualization: A Beginner's Guide to Pandas and NumPy

What is Python data analysis, and what do Pandas and NumPy actually do?

Python data analysis means using Python code to load, clean, summarize and chart raw tables. NumPy supplies fast numeric arrays and vectorized math. Pandas builds on those arrays and adds labeled rows and columns, so you can handle spreadsheet-style tables in code. Matplotlib then turns the results into charts.

I came to Python data analysis from the marketing side. I have worked with ad and SEO reports since 2012, and at some point my spreadsheets hit 300,000 rows. That is when I started writing code. This guide collects the practical notes I wish I had back then. Every section includes a short example you can run.

I will not map the whole library ecosystem here or teach machine learning. The goal is narrower: finish your first real analysis with Pandas, NumPy and Matplotlib. You can find related posts in the software category.

What do you need to install before you start?

First, install a current Python release. According to the pandas 3.0 release notes, the new version requires at least Python 3.11 and NumPy 1.26. So an old interpreter will simply fail to install it. Next, create a separate virtual environment for each project.

python3 -m venv .venv
source .venv/bin/activate
pip install pandas numpy matplotlib jupyterlab

JupyterLab works well for learning because you see each cell's output right away. That said, the notebook support inside VS Code does the same job. Whichever you pick, keep files inside a project folder and put raw data in its own data folder.

• Python 3.11 or newer, the floor for pandas 3.
• A virtual environment, so projects never fight over versions.
• JupyterLab or VS Code for step by step experiments.
• One real CSV file. Your own ad or sales export teaches the most.

Why is a NumPy array faster than a plain Python list?

A NumPy array (ndarray) stores every element with the same type in one contiguous block of memory. As a result, loops written in C do the work, and Python skips its per element type checks. We call this vectorization: you apply one operation to a whole array without writing a loop.

import numpy as np

clicks = np.array([120, 340, 95, 410])
spend = np.array([300.0, 720.0, 250.0, 900.0])

cpc = spend / clicks
print(cpc.round(2))  # cost per click

That single line computes cost per click for four campaigns. With a list you would need a for loop and a temporary list. Moreover, once the data reaches millions of rows, the gap grows from milliseconds to minutes.

The concepts you will use most are shape, dtype, slicing and broadcasting. Broadcasting lets arrays of different sizes combine under clear rules. For example, you can multiply every price by one tax rate in a single step.

Which NumPy calculations will you use most often?

In daily work you mostly use NumPy indirectly, because Pandas calls it behind the scenes. Still, a few functions are worth knowing directly. For instance, np.where assigns values by condition, and np.percentile finds quantiles.

import numpy as np

orders = np.array([150, 90, 1200, 340, 75, 610])
print(orders.mean(), np.median(orders))
print(np.percentile(orders, [25, 75]))
label = np.where(orders >= 500, "large", "small")

Watch the gap between mean and median. One large order pulls the mean up, while the median barely moves. Therefore I always add the median to reports on skewed metrics like basket size.

To generate random sample data, use np.random.default_rng(). The NumPy docs recommend this generator over the older np.random.seed pattern. The NumPy absolute beginners guide is short and reliable if you want more depth.

What exactly are a Pandas DataFrame and a Series?

A Series is a labeled one dimensional column. A DataFrame is a two dimensional table made of Series that share one index. In short, think of a DataFrame as a spreadsheet you control with code. The difference is that every step stays recorded, so you can rerun tomorrow's report with one click.

import pandas as pd

df = pd.DataFrame({
    "campaign": ["Brand", "Generic", "Competitor"],
    "spend": [1200.0, 3400.0, 900.0],
    "conversions": [48, 61, 9],
})
print(df.dtypes)

pandas 3.0 changed how text columns behave. According to its release notes, text now defaults to a dedicated str dtype instead of the old object dtype. So when an older tutorial shows object, do not panic. The same code prints str on the new version.

The index deserves its own attention. By default it counts from zero. However, setting a meaningful column such as a date or product ID as the index makes time series work much easier.

How do you load CSV and Excel files for Python data analysis?

In practice, almost every analysis starts by reading a file. Pandas offers read_csv, read_excel and read_json for this. Exports from European systems often use a semicolon as separator and a comma as decimal mark. So set those parameters explicitly from the start.

df = pd.read_csv(
    "data/ads.csv",
    sep=",",
    encoding="utf-8",
    parse_dates=["date"],
    usecols=["date", "campaign", "clicks", "spend"],
)
df.head()

Reading Excel files also requires the openpyxl package. In addition, usecols keeps memory low on large files because Pandas skips columns you do not need. If you pass parse_dates, you avoid a separate conversion step later.

When your source is an ad platform, consistent campaign names save hours down the line. For that reason, I tag links with a standard naming scheme using a UTM builder before any data ever lands in a CSV.

Which commands should you run on your first look at the data?

Do not jump straight into calculations after loading a file. First, understand what the data looks like. I spend five minutes on this step every time, and it catches most of my errors.

1. df.shape shows row and column counts.
2. df.info() lists column types and non null counts.
3. df.describe() gives summary statistics for numeric columns.
4. df.head() and df.sample(5) let you eyeball real rows.
5. df["col"].value_counts() reveals category distributions.

For example, if the spend column shows as str, it probably contains text like "$1,250.00". In other words, you must clean it before summing. Likewise, a date column that is not a datetime type will break monthly grouping.

Also check the min and max values in the describe() output. A negative spend or a single ten million dollar order signals a data entry problem.

How do you clean missing and messy data?

Cleaning is the longest part of any analysis, and the most valuable. Use isna().sum() to find missing values and duplicated() to spot repeated rows. Then make a deliberate decision for each case: drop, fill or flag.

print(df.isna().sum())
df = df.drop_duplicates()

df["spend"] = (
    df["spend"].astype(str)
      .str.replace("$", "", regex=False)
      .str.replace(",", "", regex=False)
      .astype(float)
)
df["conversions"] = df["conversions"].fillna(0)

Filling missing conversions with 0 makes sense, because no record means no conversion. On the other hand, setting a missing price to 0 would wreck your averages. In short, the right fill depends on what the column means. No single rule covers every case.

Fix whitespace and letter case in text columns too. With str.strip() and str.lower(), "Brand " and "brand" become the same campaign. Otherwise they show up as two separate rows after grouping.

How do you filter and select rows?

You have two main selection tools. loc works by label, and iloc works by position. For conditional filters, put a boolean expression inside square brackets. Combine conditions with & and |, and wrap each one in parentheses.

costly = df[(df["spend"] > 1000) & (df["conversions"] < 10)]

# Always assign through loc
df.loc[df["conversions"] == 0, "status"] = "review"

pandas 3.0 introduced a critical rule here. According to the release notes, Copy-on-Write is now the default. Every object returned by indexing behaves like a copy from your point of view. Therefore chained assignment such as df[mask]["col"] = value no longer changes the table at all. The old SettingWithCopyWarning is gone too.

The fix is simple: always assign with df.loc[mask, "col"]. For text matching, use str.contains(), and for list matching, use isin(). The pandas 3.0 release notes explain the change in detail.

How do you create new columns and ratios?

Derived metrics are where analysis starts to pay off. Arithmetic between Pandas columns uses NumPy's vectorized engine, so you never need a loop. Handle division by zero up front, though.

df["cpc"] = df["spend"] / df["clicks"]
df["conv_rate"] = df["conversions"] / df["clicks"]
df["cpa"] = df["spend"] / df["conversions"].replace(0, np.nan)

In the last line I swapped zeros for NaN. As a result, campaigns without conversions show an empty cell instead of infinity, and averages stay intact. I cover what these metrics mean for the business in my post on digital marketing KPIs.

For banding, pd.cut is very handy. For example, you can split basket values into three bands and then compute each band's share. To sanity check a percentage change by hand, the percentage calculator is a quick second opinion.

How do you summarize data with groupby?

groupby is the most powerful function in Pandas. It follows a split, apply, combine pattern. First it splits rows by a column, then applies an aggregation to each group, and finally combines the results into one table. It is the code version of a pivot table.

summary = (
    df.groupby("campaign")
      .agg(spend=("spend", "sum"),
           conversions=("conversions", "sum"),
           days=("date", "nunique"))
      .assign(cpa=lambda t: t["spend"] / t["conversions"])
      .sort_values("cpa")
)

I used named aggregation here, so the output columns carry clear names from the start. I also computed ratios from group totals. Averaging row level ratios is a common mistake, because small campaigns get the same weight as large ones.

For time based summaries, use resample. After you set the date column as the index, df.resample("W").sum() returns weekly totals, and "MS" returns totals per month start.

How do merge and pivot operations work?

Real projects rarely use a single file. Ad spend lives in one export, and orders live in another. To join two tables on a shared key, use merge. It follows the same logic as a SQL JOIN.

joined = ads.merge(orders, on=["date", "campaign"], how="left", validate="one_to_one")

pivot = joined.pivot_table(
    index="campaign", columns="device",
    values="revenue", aggfunc="sum", fill_value=0,
)

I strongly recommend the validate argument. If a key repeats, Pandas raises an error instead of silently multiplying rows. In my experience, inflated revenue after a join is the most frequent analysis bug I see.

• how="left" keeps every row from the left table.
• how="inner" keeps only rows that match on both sides.
• how="outer" keeps everything and leaves gaps where nothing matches.
• pd.concat stacks files with the same structure.

How do you visualize Python data analysis results with Matplotlib?

Numbers in a table are rarely enough. Decision makers want to see the trend at a glance. The .plot() method on Pandas objects calls Matplotlib directly, so quick charts need no extra code. For charts that go into a presentation, switch to Matplotlib's object oriented interface.

import matplotlib.pyplot as plt

weekly = df.set_index("date").resample("W")[["spend", "revenue"]].sum()

fig, ax = plt.subplots(figsize=(9, 4))
weekly.plot(ax=ax, marker="o")
ax.set_title("Weekly spend and revenue")
ax.set_ylabel("USD")
fig.tight_layout()
fig.savefig("weekly.png", dpi=150)

Once you save the chart as a file, you can drop it straight into reports and emails. Also, never send a chart without a title and axis labels. Three weeks later, even you will not remember what it shows. The Matplotlib quick start explains the two interfaces well.

Which chart type fits which question?

Chart choice depends on the question first and style second. The wrong chart can make correct data look misleading. The table below covers the pairings you will need most as a beginner.

I left the pie chart out on purpose. Once you pass three slices, the eye struggles to compare areas, and a horizontal bar says the same thing more clearly. On the other hand, if you plot two metrics on one chart, be careful with a secondary y axis. Different scales can suggest a correlation that does not exist.

Keep colors simple as well. One accent color plus grays reads much faster than a rainbow.

Worked example: how do you analyze ad and sales data end to end?

Now let us put the pieces together. This is an illustrative scenario, not real client data. Assume you have a daily ad export and an order list. The goal is to see which campaign returns more revenue than it spends.

1. Load both files with the right separator and date settings.
2. Standardize campaign names with str.strip().str.lower().
3. Join on date and campaign with merge, and use validate.
4. Sum spend and revenue per campaign with groupby.
5. Compute ROAS from the group totals.
6. Plot a horizontal bar chart and save the table as CSV.

summary = joined.groupby("campaign")[["spend", "revenue"]].sum()
summary["roas"] = summary["revenue"] / summary["spend"]
summary.sort_values("roas").plot.barh(y="roas", legend=False)
summary.to_csv("campaign_summary.csv")

To double check the output, verify a few rows by hand with the ROAS calculator. That way you catch a formula error before it reaches a slide. When you interpret the final report, ask yourself the questions from my guide on how to read a digital marketing report.

How do you inspect Search Console data with Pandas?

I use this method every week on the SEO side too. Query and page tables exported from Search Console reveal patterns in Pandas that the interface hides. For example, one filter lists queries with high impressions but a low click through rate.

gsc = pd.read_csv("data/queries.csv")
gsc["CTR"] = gsc["CTR"].str.rstrip("%").astype(float) / 100
opps = gsc[(gsc["Impressions"] > 500) & (gsc["CTR"] < 0.02)]
opps.sort_values("Impressions", ascending=False).head(20)

Column names depend on your export language, so check gsc.columns first. The CTR column usually arrives as text with a percent sign, which is why the second line strips it. I walk through the export steps in my Google Search Console guide.

If you lack the time for this kind of weekly review, I can interpret the same data for you as part of SEO consulting.

What mistakes do beginners make most often?

Everyone falls into similar traps while learning. I did too, and this list comes from my own notes. If your code runs but the result looks odd, check these items first.

• Chained assignment. In pandas 3 it has no effect at all, so use loc.
• Averaging row level ratios. Compute ratios from totals instead.
• Misreading decimal marks. Remember decimal="," for European exports.
• Duplicate keys in a merge. Catch them with validate.
• Row by row for loops. Prefer vectorized math or groupby.
• Overwriting the raw file. Always save clean data to a new file.

The last point is a lifesaver. If the raw data stays untouched, you can rerun the whole analysis the moment you spot an error. So structure your notebook to run top to bottom in one pass.

Dates changed as well. According to the pandas 3.0 notes, dates parsed from strings now default to microsecond resolution instead of nanoseconds. If you convert dates to integers for math, recheck those results.

How do you speed things up on large files?

Pandas handles a few hundred thousand rows comfortably on a laptop. As files grow, memory becomes the bottleneck. Your first move should be to skip unneeded columns and shrink data types.

• Convert repetitive text columns to the category dtype.
• Use pd.to_numeric(downcast="integer") on numeric columns.
• Prefer Parquet over CSV. It is smaller and much faster to read.
• Process huge files in pieces with chunksize.
• Use built in vectorized functions instead of apply where you can.

Run df.memory_usage(deep=True).sum() to see memory use. In addition, the new str dtype in pandas 3.0 uses PyArrow under the hood when it is installed. Per the release notes, it falls back to NumPy object without it. So installing PyArrow makes sense if you work with large text columns.

Once data passes millions of rows, you can look at Polars or DuckDB. However, beginners do not need them yet. A solid Pandas foundation makes that switch easy later.

How do you keep a notebook clean and share results?

As an analysis grows, a notebook turns into a pile of cells. To prevent that, split it into sections from the start: imports, loading, cleaning, analysis and charts. Put a short Markdown heading above each one. Then, three months later, you will find everything instantly.

Turn repeated steps into functions. For example, write one clean_amount() function instead of copying the same cleanup into three columns. Also keep constants such as file paths and thresholds in a single cell at the top. Finally, run Restart and Run All regularly, because cells executed out of order create hidden dependencies.

For sharing, export tables with to_excel() and charts with savefig(). Lead with a one sentence conclusion, then show the table and chart that support it. Managers rarely ask about method. Still, when they do, your notebook shows where every number came from.

What study plan helps you lock in these skills?

The fastest way to learn is to work with your own data. Public datasets are a good start. However, chasing real questions in your own sales or traffic export keeps motivation much higher. The plan below is a suggestion based on my field experience, not a guarantee.

1. First week: NumPy arrays, slicing and basic statistics.
2. Second week: loading CSVs, first look commands and cleaning.
3. Third week: filtering, new columns and groupby.
4. Fourth week: merge, pivot_table and a first report with Matplotlib.

At the end of each week, produce a one page mini report with one table, one chart and three sentences of interpretation. That way you build the skill of drawing conclusions, not just writing code. If you work with store data, you can adapt the question list I use in ecommerce consulting projects to these reports.

Is Python data analysis worth learning?

My short answer is yes. Spreadsheets remain great for small tables. However, when reports repeat, files multiply and rows reach the hundreds of thousands, Python data analysis saves you time and improves accuracy. Moreover, your code becomes a record of exactly how you reached each number.

To sum up, learn to think in vectors with NumPy, clean and summarize tables with Pandas, and tell the story with Matplotlib. Adopt the pandas 3.0 changes, such as Copy-on-Write and the str dtype, from day one, and you will avoid the traps in older tutorials. Then pick one small question from your own data and ship your first report this week.