Data and analysis

A spam filter is a machine learning model: it learns from the labels (spam or not spam) of past emails and uses the learned keywords to send similar new emails to the spam folder. It does not delete or mark all emails, and it has nothing to do with storage.

An overfitted model memorises the training data, including its noise, so it scores well on training data but poorly on new, unseen data. Poor performance on training data (A) is underfitting.

Feature engineering is the process of selecting and transforming the most relevant data features (such as size, colour and ear shape of animals) to improve a model's performance.

Precision = TP / (TP + FP): of the predictions made for the positive class, how many are correct. Option A is accuracy and option C is recall.

Continuous learning keeps the model learning from new data, so it stays accurate and improves its accuracy over time.

Data storytelling combines charts and graphs with a narrative, which makes data relatable, engaging and memorable. It uses more visuals, not fewer.

The null hypothesis states there is no effect or relationship and assumes any observed difference is due to random chance. Option A describes the alternative hypothesis. Option C is tempting (researchers often hope to reject H_0), but it describes the researcher's aim, not what the null hypothesis states.

The book defines the p-value as a measure of how likely the results are due to chance. Option A is a common misconception: the p-value is calculated assuming H_0 is true, so it cannot be the probability that H_0 is true. The significance level (\alpha) is chosen before the test.

Visuals such as distribution graphs, test statistics and p-value charts help present the results of hypothesis testing clearly. They support the statistics; they do not replace them.

Credibility depends on the source of the data, its accuracy, clear presentation and relevance. The time of day the data was collected usually has nothing to do with whether the story can be trusted.

When one class is very common (for example 98% of cases are negative), a model can get high accuracy by mostly predicting the common class while doing badly on the rare class, so precision and recall are low. This is the sign of imbalanced data.

With no labels, the task is unsupervised learning. Clustering (such as K-Means) groups similar customers together; the book gives customer segmentation as its example. The other three are supervised methods that need labels.

Feature engineering is the process of selecting and transforming the most relevant features (measurable properties) of the data so that a machine learning model performs better. Good features make the pattern easier for the model to learn; irrelevant features confuse it.

Example (image classification): to classify images of dogs, cats and horses, we list the common and distinct features of each animal, such as body size, colour, shape of the eyes, ears, nose and hair (fur). The model keeps data about the possible variations of these features, so that a sitting horse is not confused with a big standing dog.

Test data is a small portion of the dataset kept aside and not shown to the model during training. After training, the model is run on the test data and its answers are compared with the correct labels.

Its purpose is to check how well the model works on new, unseen data, which is what it will face in real use. It gives a fair measure of accuracy, precision and recall, and it shows problems such as overfitting (high training accuracy but low test accuracy). For example, from 100,000 animal images, 30,000 unseen images can be used as test data.

Model assessment is the evaluation of a trained machine learning model's performance, using measures such as accuracy, precision and recall, on data it has not seen before. It tells us whether the model is good enough to deploy.

• Accuracy: the percentage of all predictions that are correct, \dfrac{\text{correct predictions}}{\text{total predictions}}. If 80 of 100 new images are labelled correctly, accuracy is 80%.
• Precision: of the predictions made for a class, how many are correct, \dfrac{TP}{TP+FP}. (Recall, \dfrac{TP}{TP+FN}, and the F1 score are also used.)

The main goal of data visualization is to represent data graphically (charts, graphs, maps, dashboards) so that trends, patterns and outliers can be seen and understood quickly.

It turns large tables of numbers into pictures the brain reads at a glance, helps the analyst explore the data, and communicates findings clearly to others, including non-technical audiences, so that better decisions can be made. For example, a line chart of monthly sales instantly shows in which month sales dropped.

Data storytelling is the use of visuals (charts and graphs) together with a narrative to communicate insights from data. It usually has a relatable character, context data, an insight and a call to action.

Importance: it makes data relatable, engaging and memorable; it simplifies complex data; it highlights the key points for data-driven decisions; it shows the impact of data on real issues; and it bridges the gap between data and decision-making. Example: the story of Jameel, a 12th-grader with 78% class attendance, compared with 9th-graders at 90%, persuades the school to act.

Continuous learning means the model keeps learning from new data after it is first trained, instead of staying fixed.

• It keeps the model accurate when the real world changes (new trends, new behaviour).
• It improves accuracy over time, because every new example adds to what the model knows.
• It prevents the model from becoming out of date.

Example: a spam filter learns new spam words each time users mark new emails as spam, so it can catch new kinds of spam that did not exist at training time.

A test-train split divides the dataset into two parts: training data (the main, larger portion used to teach the model) and test data (the smaller portion kept aside to evaluate it). A common ratio is 70:30 or 80:20.

Importance: it lets us measure the model on data it has never seen, which gives an honest estimate of real-world performance and exposes overfitting. The ratio must be chosen wisely: too little training data makes the model underfit, and (as the book states) too much training data with too small a test set leads to an overfitted model going unnoticed. Example: 70,000 of 100,000 animal images for training, 30,000 for testing.

The significance level (\alpha) is the threshold, chosen before the test, for deciding whether a result is significant. Usually \alpha=0.05. If the p-value \le\alpha, we reject H_0; otherwise we fail to reject it.

Effect: \alpha is also the risk of a Type I error (rejecting a true H_0). A smaller \alpha (such as 0.01) makes the test stricter: fewer false alarms but a higher chance of missing a real effect (Type II error). A larger \alpha (such as 0.10) does the opposite. Example: a p-value of 0.03 is significant at \alpha=0.05 but not at \alpha=0.01.

Identify bias by asking: Is the source named and credible? Is the sample large and representative? Are any groups or time periods left out (cherry-picking)? Does a chart use a truncated axis or loaded colours? Is a correlation presented as a cause?

Mitigate bias by using data from reliable sources, using a larger and representative sample, showing the complete data with clear labelled axes starting from zero, stating limitations, separating facts from opinion, and getting feedback from others (peer review) before publishing. Example: a claim based on 20 students of one city should be checked against national data.

1. Formulate H_0 and H_1.
2. Collect data and analyse it.
3. Calculate the test statistic.
4. Compare it with the critical value and find the p-value.
5. Decide to reject or not reject H_0.
6. Interpret the findings and report them with visuals.

Importance: the steps make the investigation objective and repeatable. Clear hypotheses make the question testable; the test statistic and p-value give a fixed rule instead of opinion; and interpreting and reporting connect the result to the real question so that data-driven decisions can be made and checked by others.

Standard deviation (\sigma) measures how spread out the values are around the mean. It is the square root of the variance: \sigma=\sqrt{\dfrac{\sum (x-\bar x)^2}{n}}.

A small standard deviation means the values are close to the mean (consistent data); a large one means they are widely spread. It has the same unit as the data, so it is easy to interpret. Example: two classes both average 65 marks, but class A has \sigma=3 (all students similar) and class B has \sigma=20 (very mixed). The mean alone would hide this difference.

Descriptive statistics reduce thousands of values to a few meaningful numbers and charts:

• Central tendency (mean, median, mode) gives the typical value.
• Dispersion (range, variance, standard deviation) shows how spread out the values are.
• Frequency distribution (histograms, percentiles) shows the shape of the data and where most values lie.

Example: for the marks of 50,000 FBISE students, the mean, median, standard deviation and a histogram describe overall performance on one page. Such summaries give quick insight and help make informed decisions, including in machine learning models.

A result is statistically significant when it is unlikely to be due to chance (p-value below \alpha). Significance supports a conclusion by showing the effect is probably real; a non-significant result challenges it by showing the difference could be luck.

Example: people who exercised for six weeks showed an 8 mmHg drop in blood pressure. The p-value was 0.005, below 0.05, so H_0 ("exercise has no effect") is rejected: the conclusion that exercise lowers blood pressure is supported. If the p-value had been 0.4, the claim would be challenged, since such a drop could happen by chance.

Technical analysis produces tables, statistics and model outputs that non-technical people find hard to read. Data visualization turns these into simple charts, maps and dashboards, where a trend or comparison is visible at a glance, so the message is understood without statistics knowledge.

Scenario: a hospital analyst finds, using statistics, that dengue cases rise sharply two weeks after heavy rain. Instead of showing regression output, she shows the hospital board a line chart of rainfall and dengue cases over the year, with the peaks marked. The board immediately sees the pattern and approves extra beds and spraying before the monsoon.

Machine learning models

A machine learning model learns from data to make decisions. It adapts and improves over time as it processes more data. Example: a spam filter that learns spam words from emails users have marked as spam.

Rule-based algorithms

A rule-based algorithm follows fixed instructions written by a person (if condition, then action). It does not change unless someone deliberately updates the rules. Example: "if the subject contains the word lottery, move the email to spam".

Differences

Advantages of machine learning

• Handles complex and unstructured data such as images and text.
• Learns patterns automatically without explicit programming of every rule.
• Does not need expert knowledge or human intervention for every decision.
• Becomes more accurate as the dataset grows and keeps improving through continuous learning.

Limitations of machine learning

• Performs poorly if data is too little.
• Its decision-making is complex and cannot be easily interpreted.
• Needs significant computational resources.
• A too-complex model can overfit.
• Biased training data gives wrong results.

Advantages of rule-based algorithms

• Do not need large datasets; work with limited data.
• Consistent performance with the same set of rules.
• Need fewer resources and are easy to implement.

Limitations of rule-based algorithms

• Limited flexibility: cannot adapt to new data the rules did not expect.
• Give wrong results for unstructured or ambiguous data.
• Cannot learn from data.
• Many accumulated rules slow the system down.

Conclusion

Use a rule-based algorithm when the rules are clear, data is limited and decisions must be explained (for example calculating income tax). Use machine learning when patterns are complex and plenty of data is available (for example recognising faces or filtering spam).

Meaning

Model building is the process of creating a mathematical or computational model that learns relationships in data so that it can make predictions or decisions. It is done in steps, and the steps are repeated until the model is good enough. Example used throughout: a model that identifies images of dogs, cats and horses.

1. Data collection

Data is collected from various sources: databases, files, sensors, the internet. For our example we gather a large set of animal images (say 100,000), each labelled cat, dog or horse. More and better-quality data lets the model learn better.

2. Data preprocessing and feature engineering

Raw data is usually messy, so it is cleaned to make it suitable for analysis:

• Handling missing data: fill or remove missing values.
• Normalising values: bring numbers to a similar scale, for example pixel values from 0 to 255 to the range 0 to 1.
• Encoding categorical variables: convert labels such as cat, dog and horse into numbers.
• Feature engineering: select and transform the most relevant features, such as size, colour, ear and eye shape.

Then the data is split into training data and test data (for example 70:30), with a validation set used during training.

3. Choosing an algorithm and training

A suitable algorithm is chosen for the problem: classification (our animal example), regression, clustering or another type. The algorithm is trained on the training data. During training it adjusts its parameters to reduce error and improve accuracy. Training may be supervised (labelled data), unsupervised or reinforcement learning.

4. Evaluation

During training, performance is checked on a separate validation set to be sure the model works on unseen data. After optimisation, it is tested on the final test dataset using accuracy, precision, recall and F1 score. For example, if it labels 80 of 100 new images correctly, its accuracy is 80%. Evaluation also detects overfitting (good on training, poor on test) and underfitting (poor on both).

5. Deployment and refinement

If results are satisfactory, the model is deployed for real-world use. If not, we return to earlier steps: collect more data, improve features or choose another algorithm. Every time the model reads data and calculates results, it keeps learning; repeated refinement and evaluation give a model that meets the needs of the application.

Predictive models

A predictive model forecasts future events or unknown values from patterns in existing data. It answers "what will happen?" and does not need to know why.

• Uses: forecasting sales, predicting exam marks from study hours (linear regression), spam detection, weather forecasts, fraud alerts, film and song recommendations.
• Limitations: it relies on correlation, not causation; it can fail when conditions change (for example a pandemic changes buying habits); it can copy bias present in the training data; complex models are hard to interpret.

Causal models

A causal model explains cause-and-effect relationships: whether changing one factor (the cause) actually changes another (the effect). It answers "why does it happen?" and "what will happen if we change X?"

• Uses: deciding policies and actions, for example whether a breakfast programme causes higher attendance, or whether exercise lowers blood pressure; testing medicines; judging the effect of a price change.
• Limitations: needs carefully designed experiments or controlled data and hypothesis testing; can be costly and slow; hidden (confounding) factors can give a false cause; ethical limits on experiments with people.

Comparison

Conclusion

Both are needed. A predictive model tells a shop that sales will rise next week; a causal model tells it whether a discount will be the reason. In the book's example, "increased study hours lead to better exam scores" is a causal hypothesis, while predicting marks from hours with linear regression is a predictive model.

Role of research questions

Formulating questions guides data exploration and analysis. A good research question decides what data to collect, which variables to measure, which analysis to use and how to judge the answer. Vague questions produce vague results; specific questions give accurate, useful insights.

How good questions improve quality

• Focus: the question points the analysis at specific insights and trends, so time is not wasted on irrelevant data.
• Choosing data: it identifies which datasets are relevant.
• Evaluation: we can check how well the data answers the question and compare new data with previous findings.
• Refinement: questions are refined after evaluation, which improves data quality.
• Testability: a precise question can be turned into null and alternative hypotheses.

Example: school breakfast programme

Topic: the impact of school breakfast programmes on student attendance.

Conclusion

Effective question formulation drives meaningful analysis: good questions lead to better data, better evaluation and better decisions.

Hypothesis formulation

Hypothesis formulation is making an educated guess about a relationship between variables, based on existing data or prior knowledge. It names an independent variable (the cause) and a dependent variable (the effect) and proposes how they are related. It is the starting point of an investigation.

Null and alternative hypotheses

• **Null hypothesis (H_0):** there is no effect or relationship; any difference is due to random chance. It is the default assumption that must be challenged.
• **Alternative hypothesis (H_1):** there is a significant effect or relationship. It is what the researcher wants to show.

How they guide research and decisions

• They turn a vague idea into a precise, testable statement.
• They decide what data must be collected and what counts as evidence.
• The test aims to reject H_0. If the p-value is below the significance level (\alpha=0.05), H_0 is rejected and H_1 is supported; otherwise we fail to reject H_0.
• The conclusion leads to a decision: adopt, change or reject a policy or product.

Real-world example: exercise and blood pressure

Conclusion

Hypothesis formulation gives the question; H_0 and H_1 give the two possible answers; testing with data decides between them, so decisions are based on evidence rather than opinion.

A published data story to critique

Suppose a news website publishes: "Online classes destroyed results! Pass rate crashed." It shows a bar chart with two bars, pass rate 78% in 2019 and 72% in 2021, with the y-axis starting at 70%, so the 2021 bar looks one third as tall as the 2019 bar. The data is from "a survey" of 50 students of one college. No source link is given.

Checking clarity

• Purpose is clear (online classes and results), but the message is exaggerated.
• The chart has no axis titles or units, and it does not say which exam or which students.

Checking accuracy and credibility

• Source is not named, so it cannot be checked.
• Sample of 50 students from one college is small and not representative of the country.
• Only two years are shown; 2020 and 2022 are missing (cherry-picking).

Misleading or biased elements

• Truncated y-axis makes a 6-point fall look like a crash.
• Causation claimed from correlation: many other factors changed in 2021 (illness, exam pattern, grading policy).
• Loaded words ("destroyed", "crashed") push the reader's emotions rather than the data.

Suggested improvements

• Name the source (for example the board's official results) and give the sample size.
• Use data from several colleges and all years 2018 to 2023 as a line chart.
• Start the y-axis at 0 and label axes with units.
• Use neutral wording: "Pass rate fell from 78% to 72%".
• Mention other possible causes and limitations; say that the data shows a link, not proof of cause.
• Ask peers to review the story before publishing.

After these changes the story scores well on the five criteria: clarity of purpose, accuracy and source credibility, relevance, insight and fair persuasion.

Meaning of p-value

A p-value measures how likely the observed results are due to chance. It is the probability of getting a result at least as extreme as the observed one if the null hypothesis (H_0) were true.

Role in hypothesis testing

• It measures the strength of evidence against H_0. A low p-value (such as 0.02) is strong evidence against H_0; a high p-value (such as 0.5) is weak evidence.
• It is compared with the significance level \alpha (usually 0.05). If p-value \le\alpha, reject H_0: the result is statistically significant. If p-value >\alpha, fail to reject H_0.
• It makes the decision objective, so conclusions rest on reliable evidence rather than opinion.

Examples

• Fertilizer: fertilized plants grew taller. A p-value of 0.02 means taller growth by chance is very unlikely, so the fertilizer probably helped. A p-value of 0.5 means the difference could easily be random.
• Coin: 7 heads in 10 tosses gives a p-value of about 0.17 for "7 or more heads", well above 0.05, so we cannot call the coin unfair.
• Exercise: an 8 mmHg drop in blood pressure with p-value 0.005 is significant, so exercise likely lowers blood pressure.

Limitations of p-values

• A p-value is not the probability that H_0 is true, and 1-p is not the probability that H_1 is true.
• It does not show the size or importance of an effect: with a huge sample, a tiny, useless effect can be significant.
• The cut-off 0.05 is a convention; 0.049 and 0.051 are almost the same evidence but give opposite decisions.
• It depends on sample size: small samples may miss real effects (Type II error).
• Testing many things and reporting only the significant ones (p-hacking) gives false findings.
• It says nothing about bias or poor data collection.

Conclusion

P-values are a useful tool to judge whether findings are significant, but they must be reported with the effect size, confidence interval, sample size and a clear description of the data.

Introduction

Simple hypothesis testing checks a claim with data in a fixed sequence of steps. Each step has a job, and skipping one can lead to a wrong conclusion. Example throughout: does drinking coffee improve productivity?

Importance in validating research

Together the steps protect against the two errors: rejecting a true H_0 (Type I) and missing a real effect (Type II). They make research objective, repeatable and transparent, so findings can be trusted and used for decisions.

Why accuracy and relevance matter

A data story is only as credible as its data. Accuracy means the numbers are correct and come from credible sources. Relevance means the data and narrative are directly connected to the main topic. If either fails, the audience loses trust and may make wrong decisions.

Challenges

• Unreliable sources: data from unknown websites or social media may be wrong.
• Errors in data: missing values, duplicates, typing mistakes, outdated records.
• Sampling bias: a small or one-sided sample that does not represent the population.
• Misleading visuals: truncated axes, wrong chart type, 3-D effects that distort size.
• Cherry-picking and irrelevant data: choosing only data that supports the message, or adding impressive but unrelated numbers.
• Confusing correlation with causation.
• Storyteller bias: the writer's opinion shapes the story.

Solutions

• Use official and credible sources and cite them with dates.
• Clean and verify the data: check for missing or inconsistent values; cross-check with a second source.
• Use large, representative samples and state the sample size.
• Choose the right chart, label axes, start bar charts at zero.
• Keep every chart linked to the main question; remove unrelated data.
• State limitations and uncertainty honestly; say "linked to" rather than "caused" unless tested.
• Get peer review and audience feedback before publishing.

Example

A story on school attendance that uses current attendance registers (accurate, relevant), states the average 85%, compares grades 9 (90%) and 12 (78%) on a correctly scaled bar chart, and names the school records as its source is far more credible than one using a rumour or old data.

Role of feedback

Feedback is the response of the audience, teachers or peers to a data story. Constructive criticism points out specific weaknesses and suggests improvements. Using it, the storyteller refines the visuals, narrative and data until the story is clear, accurate, relevant, insightful and persuasive (the critique criteria of Activity 2).

How feedback is used

• Collect feedback through presentations, peer review, short surveys or comments.
• Analyse it: group comments under clarity, accuracy, relevance, insight and persuasion.
• Revise charts, wording and data according to the comments.
• Test again with a new audience and repeat.

Examples of constructive criticism leading to improvement

Conclusion

Feedback turns a first draft into an effective data presentation. Stories improved this way are easier to understand, more trustworthy and more likely to lead to action.

Why accuracy and relevance matter

A data story is only as credible as its data. Accuracy means the numbers are correct and come from credible sources. Relevance means the data and narrative are directly connected to the main topic. If either fails, the audience loses trust and may make wrong decisions.

Challenges

• Unreliable sources: data from unknown websites or social media may be wrong.
• Errors in data: missing values, duplicates, typing mistakes, outdated records.
• Sampling bias: a small or one-sided sample that does not represent the population.
• Misleading visuals: truncated axes, wrong chart type, 3-D effects that distort size.
• Cherry-picking and irrelevant data: choosing only data that supports the message, or adding impressive but unrelated numbers.
• Confusing correlation with causation.
• Storyteller bias: the writer's opinion shapes the story.

Solutions

• Use official and credible sources and cite them with dates.
• Clean and verify the data: check for missing or inconsistent values; cross-check with a second source.
• Use large, representative samples and state the sample size.
• Choose the right chart, label axes, start bar charts at zero.
• Keep every chart linked to the main question; remove unrelated data.
• State limitations and uncertainty honestly; say "linked to" rather than "caused" unless tested.
• Get peer review and audience feedback before publishing.

Example

A story on school attendance that uses current attendance registers (accurate, relevant), states the average 85%, compares grades 9 (90%) and 12 (78%) on a correctly scaled bar chart, and names the school records as its source is far more credible than one using a rumour or old data.

Role of feedback

Feedback is the response of the audience, teachers or peers to a data story. Constructive criticism points out specific weaknesses and suggests improvements. Using it, the storyteller refines the visuals, narrative and data until the story is clear, accurate, relevant, insightful and persuasive (the critique criteria of Activity 2).

How feedback is used

• Collect feedback through presentations, peer review, short surveys or comments.
• Analyse it: group comments under clarity, accuracy, relevance, insight and persuasion.
• Revise charts, wording and data according to the comments.
• Test again with a new audience and repeat.

Examples of constructive criticism leading to improvement

Conclusion

Feedback turns a first draft into an effective data presentation. Stories improved this way are easier to understand, more trustworthy and more likely to lead to action.

Data is a collection of raw facts and figures collected from different sources.

Both are types of categorical data (data with discrete values or labels).

• Nominal data has categories with names only and no natural order. Examples: gender (male, female), colour (red, green, blue), city. We cannot say one category is greater than another.
• Ordinal data has categories with a natural order or rank, although the gaps between levels are not fixed numbers. Examples: small, medium, large; low, high; class position 1st, 2nd, 3rd.

So we can sort ordinal data, but sorting nominal data has no meaning.

Reinforcement learning is a type of machine learning in which a model (an agent) learns by interacting with an environment. After each action it receives feedback as a reward (for a good action) or a penalty (for a bad one), and it gradually learns the actions that give the most reward.

Algorithm: Q-Learning. Applications: game playing, where a program learns to win by playing many games, and robotic navigation, where a robot learns to reach a goal without hitting obstacles.

The model is right 70% of the time; when it says Cat it is right 80% of the time, but it finds only two thirds of the real cats.

The F1 score is the harmonic mean of precision and recall: F1=2\cdot\dfrac{\text{Precision}\cdot\text{Recall}}{\text{Precision}+\text{Recall}}. It gives one number (0 to 1) that is high only when both precision and recall are high.

In imbalanced data one class is much larger. If only 20 of 1000 patients have a disease, a model that always says "healthy" is 98% accurate but finds no patient, so its recall and F1 are 0. Accuracy hides the failure; F1 exposes it.

A confidence interval is the range of values in which the true value (for example the true average effect) is likely to fall, together with a confidence level such as 95%. It estimates the true value instead of giving only one number.

Example: in the coffee study, the 95% confidence interval for the average productivity increase is (5, 15). This means we are 95% confident that the true increase lies between 5 and 15 units. Because the interval does not include 0, the data suggests coffee does improve productivity. A narrower interval (from a larger sample) gives a more precise estimate.

The power of a test is its ability to detect an effect if one really exists, that is, the probability of correctly rejecting a false H_0. High power means a low chance of a Type II error.

A larger sample size increases power, because random variation averages out and a real effect stands out more clearly. For example, testing coffee on 1000 people is much more likely to detect a real productivity increase than testing it on 10 people, and gives more reliable results.

A test statistic is a number calculated from the sample data that measures how much the data deviates from the null hypothesis (for example a t-statistic). A critical value is the boundary value, set by the significance level \alpha, that separates the \"reject H_0\" region from the \"fail to reject\" region.

Decision: if the test statistic falls beyond the critical value (in the rejection region), the result is significant and H_0 is rejected. If it lies between the critical values, we fail to reject H_0. For a two-sided test at \alpha=0.05 with a normal distribution, the critical values are \pm1.96.

• Independent variable: the factor that is changed or chosen by the researcher; it is the possible cause.
• Dependent variable: the factor that is measured; it is the possible effect, and its value depends on the independent variable.

Example hypothesis: "Increased study hours lead to better exam scores for students." Here study hours is the independent variable and exam score is the dependent variable. The hypothesis proposes a cause-and-effect relationship between them, which can be checked with hypothesis testing.

• Matplotlib (Python library): draws charts such as line, bar and scatter plots from code; used in the study hours vs marks program.
• Seaborn (Python library): makes attractive statistical charts easily on top of Matplotlib.
• Plotly and Dash (Python): Plotly creates interactive charts; Dash builds them into web dashboards.
• R: a language that is especially good at statistical graphics.
• SQL: not a charting tool itself, but used to get the required data from databases before visualising it.

The right tool depends on the task and the skill level of the user.

A dashboard is a visual tool that displays data on one screen as charts, graphs, tables and key numbers (KPIs), so that the current situation and trends can be understood at a glance and decisions can be made quickly. Example: a school dashboard showing marks and attendance of each class.

In Python, dashboards are made with Dash, a framework developed by Plotly. Plotly creates the interactive charts, Dash creates the web page layout (\texttt{html.Div}, \texttt{html.H1}, \texttt{dcc.Graph}), and \texttt{app.run()} starts the dashboard in a browser.

Unstructured data (images, audio, video, text) has no rows and columns, and its patterns are very complex. In a cat image the model must deal with colours, shapes, textures, the position of the cat and the angle of the picture.

Neural networks can learn such complex patterns directly from large amounts of data, without a person listing every rule. They can also be trained on huge datasets (for example millions of social media posts on COVID-19) in parallel using powerful computers and GPUs. Traditional algorithms such as regression suit structured data instead.

A percentile is a value below which a given percentage of the data falls. If a student's mark is at the 90th percentile, 90% of students scored below it. Percentiles show a value's position within the whole data (the median is the 50th percentile).

A histogram is a bar chart of the frequency distribution of numerical data: the x-axis is divided into classes (such as 60 to 70 marks) and each bar's height shows how many values fall in that class. It shows the shape of the data, where most values lie, the spread and any outliers.

Both learn from experience, but in different ways (Fig. 4.5):

• Machine learning learns from a large number of samples with labels (this image: dog; this image: cat). It needs thousands of examples and finds numerical patterns in them.
• Human learning uses a few examples, descriptions and diagrams (a dog has long ears and a black nose) and comparisons (a dog next to a cat). Humans can learn a new concept from very few examples and use common sense.

A machine can, however, process far more data, much faster and without getting tired.

A machine learning model can be trained in three common ways. They differ in the kind of data and feedback the model receives.

1. Supervised learning

The model is trained on labelled data: every input comes with the correct output. The model learns the mapping from inputs to outputs and then predicts outputs for new inputs.

• Algorithm: Linear Regression (predicts continuous values); classification algorithms for categories.
• Example: predicting marks from study hours; classifying images as cat, dog or horse; spam filtering from emails labelled spam or not spam.

2. Unsupervised learning

The data is unlabelled. The model tries to find patterns, structure or groups in the data on its own.

• Algorithm: K-Means Clustering, which groups similar data points together.
• Example: customer segmentation: a shop groups customers by buying habits (budget buyers, weekend shoppers, big spenders) without being told the groups.

3. Reinforcement learning

The model (agent) learns by interacting with an environment. It tries actions and receives rewards for good actions and penalties for bad ones, and learns the strategy that maximises its total reward.

• Algorithm: Q-Learning.
• Example: a program learning to play a game by playing it many times; a robot learning to move through a room without hitting obstacles.

Comparison

The choice depends on the problem and the available data: labels available means supervised; no labels means unsupervised; learning by trial and error means reinforcement learning.

Four outcomes

For a positive class such as Cat: TP (predicted Cat, is a cat), FP (predicted Cat, is not a cat), TN (predicted Not Cat, is not a cat), FN (predicted Not Cat, is a cat).

Measures

• Accuracy =\dfrac{TP+TN}{TP+TN+FP+FN}: the percentage of all predictions that are correct; how often the model is right overall.
• Precision =\dfrac{TP}{TP+FP}: of the predictions made for the class, how many are correct (quality of positive predictions).
• Recall =\dfrac{TP}{TP+FN}: of the actual positive cases, how many the model found (ability to find all relevant instances).
• F1 score =2\cdot\dfrac{\text{Precision}\cdot\text{Recall}}{\text{Precision}+\text{Recall}}: harmonic mean of the two; useful for imbalanced data.

Program

The program compares 10 actual labels with the model's predictions, counts the four outcomes for Cat with an if-elif chain, and applies the formulas.

# Accuracy, precision, recall and F1 score for the class "Cat"
actual    = ["Cat", "Cat", "Dog", "Cat", "Horse", "Dog", "Cat", "Horse", "Dog", "Cat"]
predicted = ["Cat", "Dog", "Cat", "Cat", "Horse", "Dog", "Cat", "Cat", "Dog", "Cat"]

tp = fp = fn = tn = 0
for a, p in zip(actual, predicted):
    if p == "Cat" and a == "Cat":
        tp += 1          # said Cat, really Cat
    elif p == "Cat" and a != "Cat":
        fp += 1          # said Cat, but not a Cat
    elif p != "Cat" and a == "Cat":
        fn += 1          # missed a real Cat
    else:
        tn += 1          # said Not Cat, really Not Cat

correct = sum(1 for a, p in zip(actual, predicted) if a == p)
accuracy = correct / len(actual)
precision = tp / (tp + fp)
recall = tp / (tp + fn)
f1 = 2 * precision * recall / (precision + recall)

print("TP =", tp, " FP =", fp, " FN =", fn, " TN =", tn)
print("Accuracy  =", round(accuracy, 2))
print("Precision =", round(precision, 2))
print("Recall    =", round(recall, 2))
print("F1 score  =", round(f1, 2))

TP = 4  FP = 2  FN = 1  TN = 3
Accuracy  = 0.7
Precision = 0.67
Recall    = 0.8
F1 score  = 0.73

Interpretation

Accuracy 0.7 means 7 of 10 images were labelled correctly. Precision 0.67 means 2 of the 6 "Cat" predictions were wrong. Recall 0.8 means the model found 4 of the 5 real cats. F1 of 0.73 summarises both. Because accuracy can mislead on imbalanced data, all four measures are reported together during model assessment.

Program

# Descriptive statistics of the marks of 10 students
import statistics as st

marks = [45, 50, 55, 60, 60, 65, 70, 75, 80, 90]

print("Mean     =", st.mean(marks))
print("Median   =", st.median(marks))
print("Mode     =", st.mode(marks))
print("Range    =", max(marks) - min(marks))
print("Variance =", st.pvariance(marks))
print("Std dev  =", round(st.pstdev(marks), 2))

Mean     = 65
Median   = 62.5
Mode     = 60
Range    = 45
Variance = 175
Std dev  = 13.23

Explanation of each measure

• Mean (average): sum of values divided by their number, \bar x=\dfrac{650}{10}=65. It uses every value but is pulled by extreme values.
• Median: the middle value of the ordered data. With 10 values it is the average of the 5th and 6th values, (60+65)/2=62.5. It is not affected by extreme values.
• Mode: the most frequent value, 60 (it occurs twice). Useful for categorical data too.
• Range: highest minus lowest, 90-45=45. Simple, but depends only on two values.
• Variance: average of squared differences from the mean, \sigma^2=\dfrac{1750}{10}=175. \texttt{pvariance} divides by n, as in the book's definition.
• Standard deviation: square root of the variance, \sqrt{175}=13.23 marks. It shows that marks typically lie about 13 marks from the mean.

Notes on the code

• \texttt{import statistics as st} loads Python's built-in statistics module.
• \texttt{max} and \texttt{min} give the highest and lowest values for the range.
• \texttt{round(..., 2)} keeps two decimal places.
• \texttt{st.variance} and \texttt{st.stdev} (sample versions) divide by n-1 and would give 194.44 and 13.94; the population versions match the definitions in this unit.

Together, central tendency (mean, median, mode) and dispersion (range, variance, standard deviation) summarise the whole class in six numbers.

Code

# Import numpy library and give it a short name 'np'
# We use numpy to create and manage arrays (tables of numbers)
import numpy as np

# Import the 'pyplot' part of matplotlib and give it a short name 'plt'
# We use 'pyplot' to make graphs like scatter plots and lines
import matplotlib.pyplot as plt

# Import the LinearRegression class from sklearn's linear_model module
# We use it to create a machine learning model that can draw a straight line through points
from sklearn.linear_model import LinearRegression

# Step 1: Prepare the data (study hours and marks)
X = np.array([[1], [2], [3], [4], [5]])  # Input: hours studied
y = np.array([10, 20, 30, 40, 50])       # Output: marks obtained

# Step 2: Create and train the Linear Regression model
model = LinearRegression()
model.fit(X, y)

# Step 3: Predict the marks based on study hours
y_pred = model.predict(X)

# Step 4: Plot the actual points and the prediction line
plt.scatter(X, y, color='blue')          # Plot real data points in blue
plt.plot(X, y_pred, color='red')         # Plot the prediction line in red
plt.title('Study Hours vs Marks')        # Title of the graph
plt.xlabel('Hours Studied')              # X-axis label
plt.ylabel('Marks Obtained')             # Y-axis label
plt.show()                               # Show the final graph

Explanation

• Libraries: NumPy (\texttt{np}) creates arrays; Matplotlib's pyplot (\texttt{plt}) draws graphs; \texttt{LinearRegression} from scikit-learn is a machine learning model that fits a straight line through points.
• Step 1, data: \texttt{X} holds hours studied (1 to 5) as a column (2-D array, as scikit-learn needs); \texttt{y} holds marks (10 to 50).
• Step 2, training: \texttt{model = LinearRegression()} creates the model and \texttt{model.fit(X, y)} trains it. This is supervised learning because each input has its correct output.
• Step 3, prediction: \texttt{model.predict(X)} returns predicted marks for each input.
• Step 4, plotting: \texttt{scatter} plots actual points (blue), \texttt{plot} draws the prediction line (red), \texttt{title}, \texttt{xlabel} and \texttt{ylabel} label the graph, \texttt{show} displays it.

Output

A graph titled "Study Hours vs Marks" with Hours Studied (1 to 5) on the x-axis and Marks Obtained (10 to 50) on the y-axis. The five points lie on a straight rising line: every extra hour adds 10 marks.

Printing the model's learned values confirms this:

# Train a Linear Regression model on study hours and marks, then print its predictions
import numpy as np
from sklearn.linear_model import LinearRegression

X = np.array([[1], [2], [3], [4], [5]])   # input: hours studied
y = np.array([10, 20, 30, 40, 50])        # output: marks obtained

model = LinearRegression()
model.fit(X, y)                           # the model learns the pattern from the data

print("Slope:", round(model.coef_[0], 2))
print("Intercept:", round(model.intercept_, 2))
y_pred = model.predict(X)
for hours, marks in zip(X.ravel(), y_pred):
    print(hours, "hours ->", round(marks, 1), "marks")
print("Predicted marks for 6 hours:", round(model.predict([[6]])[0], 1))

Slope: 10.0
Intercept: 0.0
1 hours -> 10.0 marks
2 hours -> 20.0 marks
3 hours -> 30.0 marks
4 hours -> 40.0 marks
5 hours -> 50.0 marks
Predicted marks for 6 hours: 60.0

The model has learned marks =10\times hours, so it predicts 60 marks for 6 hours. This shows how machine learning and visualization work together: the model finds the pattern and the graph shows it.

Dashboard

A dashboard is a visual tool that shows data as charts, graphs and tables on one screen, so that users can monitor information and spot trends quickly. Dash is a Python framework developed by Plotly for building interactive web dashboards; Plotly makes the interactive charts.

Code

# Step 1: Import necessary libraries
# (in Google Colab first run: !pip install dash plotly)
import dash
from dash import dcc, html
import plotly.express as px
import pandas as pd

# Step 2: Create sample data (e.g., sales data over months)
data = pd.DataFrame({
    "Month": ["January", "February", "March", "April", "May"],
    "Sales": [100, 200, 150, 300, 250]
})

# Step 3: Create a simple line plot with Plotly
fig = px.line(data, x="Month", y="Sales", title="Sales Over Time")

# Step 4: Initialize the Dash app
app = dash.Dash()

# Step 5: Create the layout for the dashboard
app.layout = html.Div(children=[
    html.H1("Simple Sales Dashboard"),   # Title of the dashboard
    dcc.Graph(
        id="sales-graph",
        figure=fig                       # Display the line plot
    )
])

# Step 6: Run the app
if __name__ == "__main__":
    app.run(debug=True)

Explanation of the steps

• Step 1: import \texttt{dash}, the components \texttt{dcc} and \texttt{html}, \texttt{plotly.express} as \texttt{px} and \texttt{pandas} as \texttt{pd}. In Colab, install them first with \texttt{!pip install dash plotly}.
• Step 2: create sample data in a pandas DataFrame: months January to May and sales 100, 200, 150, 300, 250.
• Step 3: \texttt{px.line} creates a line chart of Sales against Month titled "Sales Over Time".
• Step 4: \texttt{dash.Dash()} initialises the app.
• Step 5: \texttt{app.layout} is an \texttt{html.Div} containing an \texttt{H1} heading "Simple Sales Dashboard" and a \texttt{dcc.Graph} that displays the chart.
• Step 6: \texttt{app.run(debug=True)} starts a web server; the dashboard appears in the browser. The \texttt{if \_\_name\_\_ == "\_\_main\_\_"} check runs the app only when the file is run directly.

Output

A web page headed "Simple Sales Dashboard" with an interactive line chart: sales rise from 100 in January to 200 in February, dip to 150 in March, peak at 300 in April and fall to 250 in May. Hovering shows exact values.

Example

Question: Does drinking coffee improve productivity? H_0: coffee has no effect on productivity. H_1: coffee improves productivity.

Terms

• Test statistic: a number that measures how much the data deviates from H_0. We measure productivity of people who drink coffee and those who do not, and calculate a statistic such as the t-statistic. The larger it is, the stronger the evidence against H_0.
• Confidence interval: the range where the true value likely falls. A 95% confidence interval of (5, 15) for the productivity increase means we are 95% confident the true increase is between 5 and 15.
• Type I error: rejecting H_0 when it is actually true, that is concluding coffee improves productivity when it does not.
• Type II error: failing to reject H_0 when it is actually false, that is concluding coffee has no effect when it actually does.
• Power: the ability to detect a real productivity increase if coffee really improves it. A larger sample increases power.
• **Significance level (\alpha):** the threshold for significance, set at 0.05. If p-value <0.05, reject H_0. It is also the probability of a Type I error we accept.
• Sample size: the number of people studied. 1000 people give more reliable results than 10 people.

Summary

The test statistic measures deviation from H_0; the confidence interval estimates the true value; Type I and II errors come from wrong conclusions; power detects real effects; the significance level sets the threshold; and sample size affects reliability. Understanding these terms leads to accurate hypothesis tests and valid conclusions.

Data is a collection of raw facts and figures collected from different sources. The type of data decides how it is stored, visualised and which algorithm learns from it.

Why the type matters

• Visualization: numerical data suits histograms and scatter plots; categorical data suits bar and donut charts; time series suits line charts.
• Algorithms: structured data suits traditional algorithms like regression and classification; unstructured data (images, text, audio) needs advanced algorithms such as neural networks.
• Preprocessing: categorical data must be encoded as numbers, numerical data may need normalising, and text must be broken into words before a model can learn from it.