Analyzing and Securing AI System Performance
Keep adding new skills with 10,000+ programs for $239 (usually $399). Save now.
Analyzing and Securing AI System Performance
This course is part of Master Agentic AI: Core Principles & Real-World PC Professional Certificate
Included with
Learn more
Ask Coursera
Recommended experience
Recommended experience
What you'll learn
Use data aggregation and A/B testing to analyze metrics, create clear visualizations, and build automated KPI alerts.
Clean raw data, evaluate quality trade-offs, and create reproducible, versioned notebooks for peer replication.
Secure APIs using OWASP guidelines, analyze vulnerability scans, and evaluate secret management solutions.
Create structured threat models to analyze, document, and prioritize system security risks and vulnerabilities.
Skills you'll gain
- Threat Management
- Data Processing
- Data Governance
- Data Presentation
- Data Management
- Application Security
- MLOps (Machine Learning Operations)
- Threat Modeling
- A/B Testing
- Cyber Governance
- Data Visualization
- AI Security
- Vulnerability Assessments
- Analytics
- Secure Coding
- System Monitoring
- Responsible AI
- Interactive Data Visualization
Details to know
March 2026
See how employees at top companies are mastering in-demand skills
Build your Software Development expertise
- Learn new concepts from industry experts
- Gain a foundational understanding of a subject or tool
- Develop job-relevant skills with hands-on projects
- Earn a shareable career certificate from Coursera
There are 7 modules in this course
This long course develops skills for operational analytics, secure data practices, and governance essential to building trustworthy, auditable agentic systems. You will aggregate and analyze operational metrics, design A/B experiments and statistical tests to validate agent improvements, and craft clear visualizations and alerting rules for stakeholders. The course covers end-to-end data hygiene: cleaning, schema validation, reproducible notebooks with data versioning, and trade-offs between sample size and noise for experimental design. It also addresses security and governance: securing API endpoints per OWASP ASVS, dependency vulnerability analysis, secret-management trade-offs (on-prem vs managed), and threat modeling (STRIDE). Practical tasks include building DBT models for telemetry, configuring alerts, producing reproducible analytic notebooks, and creating STRIDE diagrams with documented mitigations to reduce operational and supply-chain risk.
This module trains data analysts, ML engineers, and developers to optimize AI agents built with frameworks like LangChain and Autogen and learn to prove the effectiveness of the agents. You will transform raw logs into actionable KPIs using SQL and dbt, design and execute A/B tests to compare agent versions, and apply statistical methods like the Chi-square test to validate your results. This course equips you to make objective, evidence-based recommendations for deploying agent enhancements, moving from correlation to causation and ensuring your improvements are statistically significant.
What's included
5 videos2 readings4 assignments1 ungraded lab
5 videosβ’Total 27 minutes
- Defining Agent Success: From Vanity Metrics to Actionable KPIsβ’6 minutes
- The Modern Data Stack for AIβ’6 minutes
- Correlation is not Causationβ’5 minutes
- Running a Chi-square Testβ’5 minutes
- Non-Parametric Testsβ’6 minutes
2 readingsβ’Total 15 minutes
- Advanced Time-Series Aggregation: Windows, Bucketing, and Operational Definitionsβ’7 minutes
- Principles of A/B Testingβ’8 minutes
4 assignmentsβ’Total 70 minutes
- Agent Performance Analysis Reportβ’30 minutes
- Build an Agent Performance Data Modelβ’20 minutes
- Knowledge Check: Data Transformation for Business Intelligenceβ’10 minutes
- Knowledge Check: Statistical Significance in Agent Experimentsβ’10 minutes
1 ungraded labβ’Total 25 minutes
- Analyze a Controlled Experimentβ’25 minutes
This module is for training data analysts, ML engineers, and product managers to monitor the operational health of AI systems by focusing on cost, latency, and impact. You will master data storytelling, transforming complex performance data into clear, compelling visualizations that drive decisions. Through hands-on labs, you will learn to build proactive monitoring systems by defining critical KPIs, setting precise thresholds, and configuring automated alerts. By the end, you can create dashboards that empower leadership and build automated defenses to protect your AI systems from budget overruns and performance degradation, ensuring real-world success.
What's included
4 videos4 readings4 assignments1 ungraded lab
4 videosβ’Total 23 minutes
- Dashboard Failure: The Cost of Clutterβ’6 minutes
- Choosing the Right Visualization Typeβ’5 minutes
- The High Cost of Unmonitored AIβ’8 minutes
- How to Configure an Alert in a BI Toolβ’4 minutes
4 readingsβ’Total 29 minutes
- What Makes a Visualization Effective?β’10 minutes
- How to Redesign a Cluttered Chartβ’7 minutes
- What is an Effective Alerting System?β’6 minutes
- Best Practices for Alertingβ’6 minutes
4 assignmentsβ’Total 68 minutes
- Visualizing and Alerting on AI KPIsβ’30 minutes
- Knowledge Check: Data Visualization Best Practicesβ’10 minutes
- Hands-On Learning: Designing a Cost Management Alerting Planβ’18 minutes
- Knowledge Check: Proactive Alerting for AI Cost and Performance Management β’10 minutes
1 ungraded labβ’Total 25 minutes
- Redesigning a Performance Visualizationβ’25 minutes
This module, designed for aspiring AI and data professionals, provides hands-on experience in data preparation and exploration. You will learn to build world-class models on high-quality data by implementing systematic cleaning and validation routines with tools like Pandera. In guided Jupyter labs, you will master statistical visualization and dimensionality reduction techniques, such as t-SNE, to transform complex data into clear, interpretable plots. Uncover hidden patterns, diagnose issues, and derive key insights. You'll move beyond just cleaning data to truly understanding it, ensuring your AI development is built on a solid foundation.
What's included
3 videos2 readings3 assignments2 ungraded labs
3 videosβ’Total 13 minutes
- How to Build a Validation Schema with Panderaβ’4 minutes
- Seeing the Unseen: Finding a Hidden Error Clusterβ’5 minutes
- How to Create and Interpret a t-SNE Plotβ’5 minutes
2 readingsβ’Total 18 minutes
- The Data Wrangler's Toolkit: Core Cleaning Conceptsβ’8 minutes
- Taming the Dimensions: An Introduction to t-SNE and PCAβ’10 minutes
3 assignmentsβ’Total 55 minutes
- Report: From Data Cleaning to Visual Insightβ’30 minutes
- Data Validation and Imputation: Quiz β’15 minutes
- Analyzing a New Visualization β’10 minutes
2 ungraded labsβ’Total 40 minutes
- Cleaning a Raw Customer Datasetβ’20 minutes
- Visualizing Message Embeddings to Find Errorsβ’20 minutes
This module helps data scientists and analysts deliver efficient, trustworthy results. Tackle critical questions like, "Is our data sufficient?" and "Are our findings replicable?" Learn statistical power analysis to optimize sample sizes, preventing wasted resources. You will master reproducible workflows by parameterizing Jupyter notebooks with Papermill and versioning data with DVC. Move beyond simple scripts to build robust, automated analytical projects that accelerate innovation and foster a culture of trust, ensuring your findings can be validated by peers and stakeholders.
What's included
3 videos2 readings4 assignments1 ungraded lab
3 videosβ’Total 17 minutes
- The Trade-Off Triangle: Sample Size, Noise, and Confidenceβ’6 minutes
- Why Reproducibility Matters?β’4 minutes
- How to Build a Reproducible Notebook?β’7 minutes
2 readingsβ’Total 14 minutes
- The Point of Diminishing Returnsβ’7 minutes
- The Reproducibility Toolkit: Papermill and DVCβ’7 minutes
4 assignmentsβ’Total 85 minutes
- Reproducible Data Analysis Projectβ’30 minutes
- Hands-On Learning: Analyzing Sample Size and Diminishing Returnsβ’25 minutes
- Knowledge Check: Sampling Strategy Conceptsβ’10 minutes
- Knowledge Checkβ’20 minutes
1 ungraded labβ’Total 60 minutes
- Creating a Reproducible Workflowβ’60 minutes
This module transforms developers into defenders, teaching you to build secure, production-grade AI. Learn to harden API endpoints using OWASP guidelines by implementing JWT authentication, input validation, and rate limiting. Adopt an attackerβs mindset, using DAST tools like OWASP ZAP to verify your defenses. You'll master software supply chain security by analyzing vulnerabilities, prioritizing threats with the CVSS framework, and creating hotfix and rollback plans. Through hands-on labs simulating real security incidents, you will be prepared to build and deploy resilient AI services against modern threats.
What's included
4 videos4 readings5 assignments
4 videosβ’Total 17 minutes
- JWT: Authentication and Access Control in AI Servicesβ’4 minutes
- The Tester's Mindset: From Coder to Attackerβ’4 minutes
- CVSS Explained: Technical Severity vs. Contextual Riskβ’5 minutes
- Hotfix Strategy: Compatibility and Rollback Planningβ’4 minutes
4 readingsβ’Total 27 minutes
- Securing the Gates: The OWASP API Security Top 10β’5 minutes
- Input Validation: The Primary Defense Against Injectionβ’7 minutes
- The Log4j Case Study: Anatomy of a Supply Chain Crisisβ’7 minutes
- The CVSS Framework: A Deeper Diveβ’8 minutes
5 assignmentsβ’Total 60 minutes
- Security Portfolio and SecOps Defenseβ’15 minutes
- Hands-On Learning: Implement Authentication and Validation Guardsβ’10 minutes
- Hands-On Learning: Verification with Dynamic Security Testing (DAST)β’15 minutes
- Response: Defending Against the Next Attackβ’10 minutes
- Hands-On Learning: Scan Report Analysis: Spotting the Critical CVE in urllib3β’10 minutes
This module teaches architects and engineers to build resilience directly into AI system designs. You'll master secret management by comparing self-hosted (Vault) and cloud (AWS Secrets Manager) solutions, using Total Cost of Ownership (TCO) analysis to make a justifiable recommendation. Learn to proactively hunt for vulnerabilities by deconstructing architecture with Data Flow Diagrams and applying the STRIDE framework to mitigate threats. Through hands-on projects, you will draft professional security documents, defend your decisions, and gain the skills to design, build, and maintain secure AI systems from the ground up.
What's included
4 videos5 readings6 assignments
4 videosβ’Total 19 minutes
- TCO and Compliance: A Cost-Benefit Deep Diveβ’5 minutes
- Architect's Choice: Documenting Your Recommendationβ’6 minutes
- DFDs and Trust Boundaries: Decomposing AI Architectureβ’5 minutes
- STRIDE in Practice: Identifying Spoofing and Information Disclosureβ’3 minutes
5 readingsβ’Total 30 minutes
- Cloud vs. On-Prem: The Secret Management Trade-offβ’7 minutes
- Integration and Latency: Prototyping Your Connectionβ’6 minutes
- The Power of Proactivity: Threat Modeling in DevSecOpsβ’6 minutes
- STRIDE: Your Framework for Systematic Threat Identificationβ’6 minutes
- Targeted Mitigations: Countering Spoofing and Info Disclosureβ’5 minutes
6 assignmentsβ’Total 76 minutes
- Architectural Review and Mitigation Proposalβ’16 minutes
- Hands-On Learning: Prototype and Compare Solutionsβ’15 minutes
- Hands-On Learning: Draft the Technical Recommendationβ’10 minutes
- Justification of Secret Management Decisionβ’10 minutes
- Hands-On Learning: Scan Report Analysis: Diagramming the Chat-Agentβ’10 minutes
- Hands-On Learning: STRIDE Analysis and Mitigation Planβ’15 minutes
In this hands-on module, you'll master governance, alerting, and analytics by building a complete, reproducible telemetry-to-alert pipeline. Using automated notebooks, you will construct a workflow that ingests raw system data and generates critical, real-time alerts. To embed security directly into your design, you will apply the industry-standard STRIDE framework to develop a proactive threat model, identifying and mitigating vulnerabilities before they are exploited. This module will equip you with the skills to translate data into actionable intelligence, creating a robust, automated system for maintaining secure and reliable operations in a production environment.
What's included
2 readings1 assignment
2 readingsβ’Total 30 minutes
- Why This Project Matters: Building Trust in Automated Systems β’5 minutes
- Your Project Blueprint: Requirement and Evaluationβ’25 minutes
1 assignmentβ’Total 90 minutes
- Project: Governance, Alerts and Analyticsβ’90 minutes
Earn a career certificate
Add this credential to your LinkedIn profile, resume, or CV. Share it on social media and in your performance review.
Instructor
Offered by
Explore more from Software Development
Course
Course
Why people choose Coursera for their career
Frequently asked questions
This course assumes practical ML and engineering experience. Beginners should complete foundational ML and data-engineering courses first to gain the necessary background for the labs.
Labs include building telemetry-to-alert pipelines, creating DBT models and reproducible notebooks, configuring dashboards and alerts, and producing STRIDE threat models with mitigations suitable for a portfolio artifact.
The curriculum references telemetry tooling, DBT, reproducible notebooks, and dependency scanners. Exact tool choices and versions will be confirmed by instructors and may vary by offering.
More questions
Financial aid available,
ΒΉ Some assignments in this course are AI-graded. For these assignments, your data will be used in accordance with Coursera's Privacy Notice.
