Difficulty: Intermediate | Prerequisites: Familiarity with predictive modelling concepts (Modules 1-16)
This material sits at the tail end of the course, covering what happens after you have built a model: keeping it useful over time, understanding that other analysts may reach different conclusions from the same data, and navigating the ethical responsibilities that come with working in analytics. If you have skipped the earlier modules on model building and evaluation, go back to those first. This section ties the technical work to the professional judgement you will need in practice.
TL;DR: Models degrade over time and need ongoing monitoring and maintenance. Different analysts can legitimately reach different conclusions from the same data, because modelling involves judgement. Analysts have an ethical duty to avoid harm, balance competing interests, and remain open to the possibility that someone else's analysis is better than theirs.
Model maintenance
The process of keeping a deployed model useful over time. Because the world changes, a model that performed well at launch can quietly become unreliable. In simple terms, think of it as servicing a car: the engine was fine when you bought it, but it needs regular checks to stay that way.
Model degradation
The decline in a model's predictive accuracy or usefulness over time. This happens when the data the model was trained on no longer reflects reality. In simple terms, the model's "map" no longer matches the "territory."
Change detection
Techniques for spotting when something meaningful has shifted, either in the model's output quality or in the incoming data itself. Think of it as an early warning system that tells you when maintenance is due.
Data drift / data stream change
A shift in the statistical properties of the input data over time. The data source may change format, coverage, or distribution. In simple terms, if your model learned from last year's customers and this year's customers behave differently, that is data drift.
Analyst bias
The tendency for an analyst's expectations, assumptions, or preferences to influence their modelling choices and conclusions. This is not necessarily intentional. In simple terms, you tend to find what you are looking for, which is why testing the opposite of what you expect matters.
Ethical tradeoff (in data analytics)
A situation where the benefits of a particular analysis or data use conflict with potential harm to individuals, groups, or society. There is no formula for resolving these; they require professional judgement and honest engagement with differing opinions.
Why models degrade
The data source changes (new fields, different collection methods, altered coverage)
The situation generating the data changes (customer behaviour shifts, market conditions move, regulations update)
The relationship between inputs and outputs that the model learned no longer holds
When to do maintenance
Track the model's performance continuously, not just at launch
Monitor model performance metrics over time (accuracy, precision, recall, error rates)
Monitor the incoming data streams for distributional shifts
Use change detection methods to flag when performance or data properties move outside expected bounds
When a meaningful change is detected, that is the trigger for maintenance work: retraining, recalibrating, or rebuilding the model
The core principle
Building a model is not a one-off task. A model is a living system that needs ongoing attention. The monitoring strategy should be decided during the build, not bolted on afterwards.
Modelling is an art, not a recipe
Given the same data and the same question, different analysts will make different modelling choices and reach different (sometimes opposite) conclusions
This is normal and expected. There is rarely one "correct" approach to a problem
The course teaches techniques, but applying them involves judgement. Usage is not cookie-cutter
Bias shapes results
Your own expectations and assumptions influence which variables you select, how you clean the data, which model you choose, and how you interpret the output
Testing the opposite of what you expect is a practical safeguard against confirmation bias
Keeping an open mind is not a platitude here; it is a specific analytical discipline
What this means in practice
Two well-executed analyses of the same dataset can legitimately disagree
The goal is not to eliminate subjectivity (you cannot) but to be transparent about the choices you made and why
"Do good, not harm" as a framework
If there is no competing benefit, do not use data or analysis in ways that harm individuals, groups, or society. Full stop
If there is a competing benefit (e.g. public health surveillance vs. individual privacy), balance the tradeoff honestly and recognise that ethical dilemmas involve opinions, not just facts
Professional responsibility
Good analyses can differ and reach opposite conclusions. This is a feature of the discipline, not a flaw
It is your professional and ethical responsibility to:
Listen to other analysts' work with genuine openness
Consider that their analysis may be right and yours may be wrong
"Freedom to speak, duty to listen" captures the reciprocal obligation: you are entitled to publish your findings, and you are obligated to engage seriously with others' findings
The practical takeaway
Ethics in analytics is not a checkbox exercise. It requires ongoing judgement about who benefits, who is harmed, and whether you have engaged honestly with alternative interpretations of the data.
Model maintenance: Credit scoring models are a classic example. A model trained on pre-pandemic spending data would have performed poorly in 2020 when consumer behaviour changed sharply. Banks monitor these models continuously and retrain them when drift is detected.
Many analysts, one data set: In clinical trials, independent statisticians routinely analyse the same trial data and reach different conclusions about efficacy. Regulatory bodies expect this and weigh multiple analyses before making decisions.
Ethics: Predictive policing algorithms that direct officers to particular neighbourhoods raise the "do good, not harm" question directly. The data may be accurate, but the deployment can reinforce existing biases and cause measurable harm to the communities being policed.
"Once a model is deployed, the work is done." It is not. Deployment is the beginning of the maintenance phase, not the end of the project. Models degrade silently; without monitoring, you will not know until the damage is done.
"If two analysts disagree, one of them is wrong." Both can be right, or both can be partially right. Different modelling choices (variable selection, algorithm, preprocessing) can produce legitimately different results from the same data.
"Bias only matters if you are trying to be biased." Analyst bias is largely unconscious. You do not need intent for your expectations to shape your results. The antidote is testing the opposite of your hypothesis, not trusting your own objectivity.
"Ethics is a separate topic from analytics." The course treats ethics as embedded in analytical practice, not bolted on. Every modelling decision, from what data to collect to how to present findings, has ethical dimensions.
Know the definition of model maintenance and be able to explain why models degrade over time
Be prepared to name at least two triggers for maintenance (change in model performance, change in data streams)
Understand that "modelling is an art" is a course theme, not a throwaway line. Expect questions testing whether you grasp that different valid approaches exist
The ethical framework ("do good, not harm" plus balancing competing benefits) is likely to appear as a scenario question. You will need to identify the tradeoff and explain how you would approach it
"Freedom to speak, duty to listen" captures a testable concept: the professional obligation to engage with analyses that contradict your own
True or false: A model that performed well at deployment will continue to perform well indefinitely without intervention.
False. Models degrade over time as data and conditions change.
Fill in the blank: The process of keeping a model useful over time is called ________.
Model maintenance.
True or false: If two analysts reach opposite conclusions from the same data, one of them must have made an error.
False. Different valid approaches can produce different conclusions.
Fill in the blank: When there is a competing benefit in data use, analysts should balance the ________ and recognise that opinions are involved.
Tradeoff.
True or false: It is your professional responsibility to consider that another analyst's work may be correct and yours may be wrong.
True.
Q: Define model maintenance and explain why it is necessary.
A: Model maintenance is the process of keeping a deployed model useful over time. It is necessary because models can degrade as the data sources change or the conditions that generated the training data shift. Without maintenance, a model's predictions become unreliable.
Q: Name two methods for determining when model maintenance is needed.
A: (1) Change detection on model performance metrics over time. (2) Change detection on incoming data streams to identify distributional drift.
Q: Why might two analysts reach different conclusions from the same dataset? Does this indicate a problem?
A: Different analysts bring different modelling choices: variable selection, algorithm choice, data cleaning approaches, and interpretive frameworks. This is expected and does not indicate an error. Modelling is an art, and legitimate variation in approach is a normal feature of analytical work.
Q: Explain the "do good, not harm" ethical framework as applied to data analytics.
A: If there is no competing benefit, avoid using data or analysis in ways that harm individuals, groups, or society. If there is a competing benefit, balance the tradeoff as honestly as possible, recognising that ethical dilemmas involve opinions and professional judgement, not purely technical solutions.
Q: What does "freedom to speak, duty to listen" mean in the context of analytics ethics?
A: It means that analysts have the right to present their findings, but they also have a professional obligation to engage seriously with other analysts' work, to consider that alternative analyses may be correct, and to remain open to the possibility that their own conclusions are wrong.
Model maintenance connects directly to the model evaluation techniques covered in earlier modules. The metrics you used to assess a model at build time (accuracy, error rates, confusion matrices) are the same metrics you monitor during maintenance. If you are comfortable with evaluation, maintenance is the ongoing application of those same tools.
The "many analysts" theme reinforces the earlier material on model selection. When you learned that different algorithms suit different problems, the underlying lesson was the same: analytical judgement matters, and there is rarely a single correct path.
The ethics material connects to broader discussions in data science about algorithmic fairness, bias in training data, and the social impact of automated decision-making. These topics extend well beyond this course but start here.
Model maintenance, model degradation, concept drift, data drift, model monitoring, change detection, model retraining, model lifecycle, predictive model upkeep, analyst bias, confirmation bias in analytics, modelling as art, subjective modelling choices, analytical judgement, multiple valid analyses, data ethics, do good not harm, ethical tradeoffs in data science, freedom to speak duty to listen, professional responsibility in analytics, algorithmic fairness, responsible data use, W15, Modules 17-21, Introduction to Computer Programming, UT Austin