Skip to content

Commit 3123445

Browse files
strickvlclaude
andauthored
Major UX improvements for credit-scorer: simplified workflows, unified visualizations, and user-friendly defaults (#236)
* Fix model directory creation in training step Create models directory before saving model to prevent FileNotFoundError when running training pipeline. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * Fix auto-approve flag functionality in deployment pipeline Connect the --auto-approve flag to the actual approval logic by checking DEPLOY_APPROVAL, APPROVER, and APPROVAL_RATIONALE environment variables in the approve_deployment step. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * Add Modal secrets setup instructions to README Document the required credit-scoring-secrets Modal secret for deployment pipeline, including Slack credentials needed for EU AI Act compliance incident reporting. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * Fix --all flag to run complete pipeline workflow Enable the --all flag to run feature engineering, training, and deployment pipelines sequentially with proper output chaining between steps. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * Make pipeline names less generic * Fix API documentation and risk management dashboard integration - Extract real deployment URLs from deployment_info parameter structure - Handle dictionary return from load_risk_register() by extracting 'Risks' sheet - Fix indentation and structure in risk management section - Resolve "Risk level information not found" issue by properly accessing risk data 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * Formatting * Fix --all flag functionality in pipeline execution - Move --all flag handling to beginning of main() function - Remove direct artifact passing between pipelines to avoid Pydantic validation errors - Let ZenML automatically fetch latest artifacts from artifact store - Enables successful execution of complete workflow: feature → training → deployment 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * Update README to highlight --all flag as recommended workflow - Add --all flag as the primary recommended command for running complete workflow - Show --all with --auto-approve for seamless execution - Reorganize commands to emphasize complete workflow over individual pipelines - Add --all --no-cache example for additional pipeline options 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * Add risk assessment HTML visualization as second artifact - Add generate_risk_visualization() function to create styled HTML report - Update risk_assessment step to return tuple: (risk_scores_dict, risk_visualization_html) - Add RISK_VISUALIZATION constant to annotations - Update training pipeline and run.py to handle new return signature - Visualization includes overall risk score, component risks, and detailed hazard breakdown - Color-coded risk levels (LOW/MEDIUM/HIGH) with severity badges for hazards - Professional styling with gradient header and responsive card layout 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * Unify HTMLString visualizations with shared CSS system Migrate all HTMLString artifacts to use centralized CSS styling for improved maintainability and visual consistency across EU AI Act compliance reports. Key changes: - Updated eval.py to use shared CSS classes for model evaluation dashboard - Migrated generate_sbom.py SBOM visualization to shared styling - Converted post_run_annex.py Annex IV documentation to use shared CSS - Updated dashboard.py compliance dashboard with unified styling - Enhanced risk_assessment.py with shared CSS components Benefits: - Consistent styling across all compliance visualizations - Reduced CSS duplication from ~500+ lines of inline styles - Improved maintainability with centralized style management - Enhanced visual consistency for EU AI Act reporting 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * Simplify deployment workflow with new flag defaults - Auto-approve is now the default behavior (use --manual-approve to disable) - Slack notifications are disabled by default (use --enable-slack to enable) - Modal secrets are now optional, only required when --enable-slack is used - Primary workflow is now just `python run.py --all` with no flags needed - Full EU AI Act compliance available via `python run.py --all --enable-slack` This makes the project much easier to test and get started with while keeping all compliance features available when explicitly requested. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * Add robust input validation for ENABLE_SLACK environment variable - Add strict validation to only accept "true" or "false" values for ENABLE_SLACK - In Modal app creation: raise ValueError and abort deployment for invalid values - In incident reporting: log error but gracefully continue with disabled Slack - Prevents silent failures from typos like "True", "1", "yes", or "tru" - Provides clear error messages showing expected vs actual values This improves user experience by catching configuration errors early rather than silently defaulting to disabled Slack notifications. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> --------- Co-authored-by: Claude <[email protected]>
1 parent 34c5c9a commit 3123445

File tree

13 files changed

+1546
-345
lines changed

13 files changed

+1546
-345
lines changed

credit-scorer/README.md

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -115,20 +115,37 @@ zenml alerter register slack_alerter \
115115
zenml stack update <STACK_NAME> -al slack_alerter
116116
```
117117

118+
5. Set up Modal secrets for deployment (optional, only needed with `--enable-slack` flag):
119+
120+
```bash
121+
# Create Modal secret with Slack credentials for incident reporting
122+
modal secret create credit-scoring-secrets \
123+
SLACK_BOT_TOKEN=<your_slack_bot_token> \
124+
SLACK_CHANNEL_ID=<your_slack_channel_id>
125+
```
126+
127+
> **Note:** The deployment pipeline uses Modal for cloud deployment. By default, Slack notifications are disabled for easier testing. The `credit-scoring-secrets` Modal secret stores the necessary Slack credentials for automated notifications when the deployed model API detects high or critical severity incidents.
128+
129+
> **Enabling full compliance features:** For complete EU AI Act compliance incident reporting (Article 18), use the `--enable-slack` flag (e.g., `python run.py --deploy --enable-slack`). This requires the Modal secret to be configured with your Slack credentials for automated incident notifications.
130+
118131
## 📊 Running Pipelines
119132

120133
### Basic Commands
121134

122135
```bash
136+
# Run complete workflow (recommended)
137+
python run.py --all # Feature → Training → Deployment (auto-approved, no Slack)
138+
123139
# Run individual pipelines
124140
python run.py --feature # Feature engineering (Articles 10, 12)
125141
python run.py --train # Model training (Articles 9, 11, 15)
126142
python run.py --deploy # Deployment (Articles 14, 17, 18)
127143

128144
# Pipeline options
129-
python run.py --train --auto-approve # Skip manual approval steps
130-
python run.py --feature --no-cache # Disable ZenML caching
145+
python run.py --all --no-cache # Complete workflow without caching
146+
python run.py --all --manual-approve # Complete workflow with manual approval steps
131147
python run.py --deploy --config-dir ./my-configs # Custom config directory
148+
python run.py --all --enable-slack # Complete workflow with Slack notifications (requires Modal secrets)
132149
```
133150

134151
### View Compliance Dashboard

credit-scorer/modal_app/modal_deployment.py

Lines changed: 40 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -81,9 +81,33 @@ def create_modal_app(python_version: str = "3.12.9"):
8181

8282
app_config = {
8383
"image": base_image,
84-
"secrets": [modal.Secret.from_name(SECRET_NAME)],
8584
}
8685

86+
# Only add secrets if Slack notifications are explicitly enabled
87+
enable_slack_raw = os.getenv("ENABLE_SLACK", "false").lower()
88+
if enable_slack_raw not in {"true", "false"}:
89+
logger.error(
90+
f"Invalid value for ENABLE_SLACK: '{enable_slack_raw}'. Expected 'true' or 'false'. Deployment aborted."
91+
)
92+
raise ValueError(
93+
f"Invalid ENABLE_SLACK value: '{enable_slack_raw}'. Deployment aborted."
94+
)
95+
96+
enable_slack = enable_slack_raw == "true"
97+
if enable_slack:
98+
try:
99+
app_config["secrets"] = [modal.Secret.from_name(SECRET_NAME)]
100+
logger.info(f"Added secret {SECRET_NAME} to Modal app")
101+
except Exception as e:
102+
logger.warning(f"Could not add secret {SECRET_NAME}: {e}")
103+
logger.info(
104+
"Continuing without secrets - Slack notifications will be disabled"
105+
)
106+
else:
107+
logger.info(
108+
"Slack notifications disabled by default - Modal app created without secrets"
109+
)
110+
87111
try:
88112
volume = modal.Volume.from_name(VOLUME_NAME)
89113
app_config["volumes"] = {"/mnt": volume}
@@ -167,7 +191,17 @@ def _report_incident(incident_data: dict, model_checksum: str) -> dict:
167191
logger.warning(f"Could not write to local incident log: {e}")
168192

169193
# 2. Direct Slack notification for high/critical severity (not using ZenML)
170-
if incident["severity"] in ("high", "critical"):
194+
enable_slack_raw = os.getenv("ENABLE_SLACK", "false").lower()
195+
if enable_slack_raw not in {"true", "false"}:
196+
logger.error(
197+
f"Invalid value for ENABLE_SLACK: '{enable_slack_raw}'. Expected 'true' or 'false'."
198+
)
199+
# Don't abort incident reporting, just skip Slack notification
200+
enable_slack = False
201+
else:
202+
enable_slack = enable_slack_raw == "true"
203+
204+
if incident["severity"] in ("high", "critical") and enable_slack:
171205
try:
172206
slack_token = os.getenv("SLACK_BOT_TOKEN")
173207
slack_channel = os.getenv("SLACK_CHANNEL_ID", SC.CHANNEL_ID)
@@ -209,6 +243,10 @@ def _report_incident(incident_data: dict, model_checksum: str) -> dict:
209243
)
210244
except Exception as e:
211245
logger.warning(f"Failed to send Slack notification: {e}")
246+
elif not enable_slack:
247+
logger.info(
248+
"Slack notifications disabled (use --enable-slack flag to enable)"
249+
)
212250

213251
return {
214252
"status": "reported",

credit-scorer/run.py

Lines changed: 34 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -81,25 +81,32 @@
8181
help="Directory containing configuration files.",
8282
)
8383
@click.option(
84-
"--auto-approve",
84+
"--manual-approve",
8585
is_flag=True,
8686
default=False,
87-
help="Auto-approve deployment (for CI/CD pipelines).",
87+
help="Require manual approval for deployment (disables auto-approve).",
8888
)
8989
@click.option(
9090
"--no-cache",
9191
is_flag=True,
9292
default=False,
9393
help="Disable caching for pipeline runs.",
9494
)
95+
@click.option(
96+
"--enable-slack",
97+
is_flag=True,
98+
default=False,
99+
help="Enable Slack notifications in deployment (requires Modal secrets setup).",
100+
)
95101
def main(
96102
feature: bool = False,
97103
train: bool = False,
98104
deploy: bool = False,
99105
all: bool = False,
100106
config_dir: str = "src/configs",
101-
auto_approve: bool = True,
107+
manual_approve: bool = False,
102108
no_cache: bool = False,
109+
enable_slack: bool = False,
103110
):
104111
"""Main entry point for EU AI Act compliance pipelines.
105112
@@ -115,19 +122,28 @@ def main(
115122
if not config_dir.exists():
116123
raise ValueError(f"Configuration directory {config_dir} not found")
117124

118-
# Handle auto-approve setting for deployment
125+
# Handle approval setting for deployment (auto-approve is now default)
126+
auto_approve = not manual_approve
119127
if auto_approve:
120128
os.environ["DEPLOY_APPROVAL"] = "y"
121129
os.environ["APPROVER"] = "automated_ci"
122130
os.environ["APPROVAL_RATIONALE"] = (
123-
"Automatic approval via --auto-approve flag"
131+
"Automatic approval (default behavior)"
124132
)
125133

134+
# Handle Slack setting for deployment (Slack disabled by default)
135+
if enable_slack:
136+
os.environ["ENABLE_SLACK"] = "true"
137+
126138
# Common pipeline options
127139
pipeline_args = {}
128140
if no_cache:
129141
pipeline_args["enable_cache"] = False
130142

143+
# Handle --all flag first
144+
if all:
145+
feature = train = deploy = True
146+
131147
# Track outputs for chaining pipelines
132148
outputs = {}
133149

@@ -162,15 +178,18 @@ def main(
162178

163179
train_args = {}
164180

165-
# Use outputs from previous pipeline if available
166-
if "train_df" in outputs and "test_df" in outputs:
167-
train_args["train_df"] = outputs["train_df"]
168-
train_args["test_df"] = outputs["test_df"]
181+
# Don't pass DataFrame artifacts directly - let training pipeline fetch them
182+
# from artifact store via Client.get_artifact_version() as designed
169183

170184
training_pipeline = training.with_options(**pipeline_args)
171-
model, eval_results, eval_visualization, risk_scores, *_ = (
172-
training_pipeline(**train_args)
173-
)
185+
(
186+
model,
187+
eval_results,
188+
eval_visualization,
189+
risk_scores,
190+
risk_visualization,
191+
*_,
192+
) = training_pipeline(**train_args)
174193

175194
# Store for potential chaining
176195
outputs["model"] = model
@@ -188,21 +207,15 @@ def main(
188207

189208
deploy_args = {}
190209

191-
if "model" in outputs:
192-
deploy_args["model"] = outputs["model"]
193-
if "evaluation_results" in outputs:
194-
deploy_args["evaluation_results"] = outputs["evaluation_results"]
195-
if "risk_scores" in outputs:
196-
deploy_args["risk_scores"] = outputs["risk_scores"]
197-
if "preprocess_pipeline" in outputs:
198-
deploy_args["preprocess_pipeline"] = outputs["preprocess_pipeline"]
210+
# Don't pass artifacts directly - let deployment pipeline fetch them
211+
# from artifact store via Client.get_artifact_version() as designed
199212

200213
deployment.with_options(**pipeline_args)(**deploy_args)
201214

202215
logger.info("✅ Deployment pipeline completed")
203216

204217
# If no pipeline specified, show help
205-
if not any([feature, train, deploy, all]):
218+
if not any([feature, train, deploy]):
206219
ctx = click.get_current_context()
207220
click.echo(ctx.get_help())
208221

credit-scorer/src/constants/annotations.py

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -32,9 +32,9 @@ class StrEnum(str, Enum):
3232
class Pipelines(StrEnum):
3333
"""Pipeline names used in ZenML."""
3434

35-
FEATURE_ENGINEERING = "feature_engineering"
36-
TRAINING = "training"
37-
DEPLOYMENT = "deployment"
35+
FEATURE_ENGINEERING = "credit_scoring_feature_engineering"
36+
TRAINING = "credit_scoring_training"
37+
DEPLOYMENT = "credit_scoring_deployment"
3838

3939

4040
class Artifacts(StrEnum):
@@ -58,6 +58,7 @@ class Artifacts(StrEnum):
5858
EVALUATION_RESULTS = "evaluation_results"
5959
EVAL_VISUALIZATION = "evaluation_visualization"
6060
RISK_SCORES = "risk_scores"
61+
RISK_VISUALIZATION = "risk_visualization"
6162
FAIRNESS_REPORT = "fairness_report"
6263
RISK_REGISTER = "risk_register"
6364

@@ -69,6 +70,8 @@ class Artifacts(StrEnum):
6970
INCIDENT_REPORT = "incident_report"
7071
COMPLIANCE_RECORD = "compliance_record"
7172
SBOM_ARTIFACT = "sbom_artifact"
73+
SBOM_HTML = "sbom_html"
7274
ANNEX_IV_PATH = "annex_iv_path"
75+
ANNEX_IV_HTML = "annex_iv_html"
7376
RUN_RELEASE_DIR = "run_release_dir"
7477
COMPLIANCE_DASHBOARD_HTML = "compliance_dashboard_html"

credit-scorer/src/pipelines/deployment.py

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,7 @@ def deployment(
9292
)
9393

9494
# Generate Software Bill of Materials for Article 15 (Accuracy & Robustness)
95-
generate_sbom(
95+
sbom_data, sbom_html = generate_sbom(
9696
deployment_info=deployment_info,
9797
)
9898

@@ -103,10 +103,12 @@ def deployment(
103103
)
104104

105105
# Generate comprehensive technical documentation (Article 11)
106-
documentation_path, run_release_dir = generate_annex_iv_documentation(
107-
evaluation_results=evaluation_results,
108-
risk_scores=risk_scores,
109-
deployment_info=deployment_info,
106+
documentation_path, documentation_html, run_release_dir = (
107+
generate_annex_iv_documentation(
108+
evaluation_results=evaluation_results,
109+
risk_scores=risk_scores,
110+
deployment_info=deployment_info,
111+
)
110112
)
111113

112114
# Generate compliance dashboard HTML visualization
@@ -118,5 +120,8 @@ def deployment(
118120
deployment_info,
119121
monitoring_plan,
120122
documentation_path,
123+
documentation_html,
124+
sbom_data,
125+
sbom_html,
121126
compliance_dashboard,
122127
)

credit-scorer/src/pipelines/training.py

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -91,11 +91,17 @@ def training(
9191
)
9292

9393
# Perform risk assessment based on evaluation results
94-
risk_scores = risk_assessment(
94+
risk_scores, risk_visualization = risk_assessment(
9595
evaluation_results=eval_results,
9696
risk_register_path=risk_register_path,
9797
approval_thresholds=approval_thresholds,
9898
)
9999

100100
# Return artifacts to be used by deployment pipeline
101-
return model, eval_results, eval_visualization, risk_scores
101+
return (
102+
model,
103+
eval_results,
104+
eval_visualization,
105+
risk_scores,
106+
risk_visualization,
107+
)

credit-scorer/src/steps/deployment/approve.py

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
# you may not use this file except in compliance with the License.
55
# You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
66

7+
import os
78
import time
89
from datetime import datetime
910
from typing import Annotated, Any, Dict, Tuple
@@ -285,8 +286,17 @@ def send_slack_message(message, blocks, ask_question=False):
285286
print("💡 Fix: Use a Bot User OAuth Token (starts with xoxb-)")
286287
return None
287288

289+
# Check for auto-approve from environment variables
290+
auto_approve = os.environ.get("DEPLOY_APPROVAL", "").lower() == "y"
291+
env_approver = os.environ.get("APPROVER", "")
292+
env_rationale = os.environ.get("APPROVAL_RATIONALE", "")
293+
288294
# Send initial notification
289-
header = "MODEL AUTO-APPROVED" if all_ok else "HUMAN REVIEW REQUIRED"
295+
header = (
296+
"MODEL AUTO-APPROVED"
297+
if all_ok or auto_approve
298+
else "HUMAN REVIEW REQUIRED"
299+
)
290300
send_slack_message(header, create_blocks("Model Approval"))
291301

292302
# Determine approval
@@ -296,6 +306,12 @@ def send_slack_message(message, blocks, ask_question=False):
296306
"automated_system",
297307
"All criteria met",
298308
)
309+
elif auto_approve:
310+
approved, approver, rationale = (
311+
True,
312+
env_approver or "automated_ci",
313+
env_rationale or "Auto-approved via environment variable",
314+
)
299315
else:
300316
response = send_slack_message(
301317
f"Override deployment for pipeline '{pipeline_name}'?",

0 commit comments

Comments
 (0)