Airflow 3.0 deprecations

[dhruvmanila]

Tracking issue for the AIR3xx rules that implements Airflow 3.0 deprecations.

For details, refer to the Airflow tracking issue: apache/airflow#44556

Help wanted

The following is a "help wanted" task list:

• Add infrastructure to provide an auto-fix for the migration rules. Look at NumPy 2.0 implementation for reference
• Add a guard specifically for Airflow imports and attributes where a try .. except block could include both the new and deprecated symbols for backwards compatibility. Look at NumPy 2.0 implementation for reference

Ref: apache/airflow#41641

[Lee-W]

There's already a short proposal #14616 (comment). Putting it here might make it easier to find 🙂

---

I'm thinking of separating "removal" and "rename" at least. It would be much easier for the users to know whether there's anything they could do to fix it.

[dhruvmanila]

Additionally, the NumPy 2.0 migration rule is a good reference whose implementation is at: https://github.com/astral-sh/ruff/blob/6f1cf5b686e67fb74383641fcbbe98e33481a271/crates/ruff_linter/src/rules/numpy/rules/numpy_2_0_deprecation.rs

[Lee-W]

Hi, we have grouped our deprecations into several categories and would appreciate it if the team could take a look and let us know what you think about it 🙂

• already merged
  • AIR301: Avoid implicit DAG schedule [airflow] Avoid implicit DAG schedule (AIR301) #14581
• newly proposed error number
  • AIR302: removal or no direct mapping
    • following [airflow] Avoid deprecated values (AIR302) #14582
  • AIR303: rename
    • It would be much easier for the users to know whether they could do anything to fix it.
  • AIR304: moved to airflow providers
    • for these, they're similar to "rename" but require the user to install an additional package (e.g., apache-airflow-providers-fab)

Here's the categorised list we have now
apache/airflow#41641 (comment)
(there's another group we temporarily mark as AIR305, but we'll need some time to figure out what we can do for those. Will likely to be merged into 302 or 303)

[MichaReiser]

I like what's proposed here. Thanks for going through all the deprecations and grouping them by possible rules. We can also do some fine-tuning as we go.

[Lee-W]

Later discuss with @uranusjr, we think it would be better for us to merge rename and removal into 302. So the current idea would be

• already merged
  • AIR301: Avoid implicit DAG schedule ([airflow] Avoid implicit DAG schedule (AIR301) #14581)
  • AIR302: removal ([airflow] Avoid deprecated values (AIR302) #14582)
• new rule number
  • AIR303: moved to airflow providers
    • for these, they're similar to "rename" but require the user to install an additional package (e.g., apache-airflow-providers-fab)

We now use apache/airflow#44556 to track the detail of this task instead of the original one as some of the migration might not be able to be done in ruff

[MichaReiser]

Sorry for pinging you here, but could someone more familiar with Airflow volunteer looking at the changes in #14631? We just want to make sure that the change is correct.

[Lee-W]

Sorry for pinging you here, but could someone more familiar with Airflow volunteer looking at the changes in #14631? We just want to make sure that the change is correct.

Looks like a valid one to me

[MichaReiser]

Thanks for taking a look!

[dhruvmanila]

I think it'll also be really useful to provide an auto-fix for these rules. Refer to how NumPy 2.0 deprecation rule does it:

ruff/crates/ruff_linter/src/rules/numpy/rules/numpy_2_0_deprecation.rs

Lines 683 to 708 in 1bd8fbb

	match replacement.details {
	Details::AutoImport {
	path,
	name,
	compatibility,
	} => {
	diagnostic.try_set_fix(|| {
	let (import_edit, binding) = checker.importer().get_or_import_symbol(
	&ImportRequest::import_from(path, name),
	expr.start(),
	checker.semantic(),
	)?;
	let replacement_edit = Edit::range_replacement(binding, expr.range());
	Ok(match compatibility {
	Compatibility::BackwardsCompatible => {
	Fix::safe_edits(import_edit, [replacement_edit])
	}
	Compatibility::Breaking => Fix::unsafe_edits(import_edit, [replacement_edit]),
	})
	});
	}
	Details::AutoPurePython { python_expr } => diagnostic.set_fix(Fix::safe_edit(
	Edit::range_replacement(python_expr.to_string(), expr.range()),
	)),
	Details::Manual { guideline: _ } => {}
	};

And, I think we'll also need to add this guard check specific to Airflow imports and attributes which allows for having both the new and deprecated symbols to exists in a try .. except .. block:

ruff/crates/ruff_linter/src/rules/numpy/rules/numpy_2_0_deprecation.rs

Lines 671 to 673 in 1bd8fbb

	if is_guarded_by_try_except(expr, &replacement, semantic) {
	return;
	}

[uranusjr]

I agree. We’re on a tight schedule and would like to prioritise adding all the rules first, but contribution is always welcomed on these.

[dhruvmanila]

We’re on a tight schedule and would like to prioritise adding all the rules first, but contribution is always welcomed on these.

Yeah, I think that's fine. Do you have a rough timeline on when v3 is going to be released? I could spend some time in adding the infrastructure for an auto-fix. I think that shouldn't take much time as we already have a reference implementation. And, I'll mark this as a "help wanted" task in the issue description if someone else is interested.

[tirkarthi]

It seems ruff also has interface to add fixes which seems straightforward. I tested below where users can opt for unsafe fixes using --fix and also see --diff before applying them. This fixes some of the common keyword argument related issues. I can raise a PR. This will help users to do migrations using ruff with in place changes for the dag files once they confirm the changes through diffs in large deployments. Sample output as below

Diff : main...tirkarthi:ruff:air302-fixes

cargo run -p ruff -- check --preview --select=AIR302 --unsafe-fixes --diff example_latest_only.py --no-cache
   Compiling ruff v0.8.2 (/home/karthikeyan/stuff/python/ruff/crates/ruff)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 11.86s
     Running `target/debug/ruff check --preview --select=AIR302 --unsafe-fixes --diff example_latest_only.py --no-cache`
--- example_latest_only.py
+++ example_latest_only.py
@@ -27,7 +27,7 @@
 
 with DAG(
     dag_id="latest_only",
-    schedule_interval="daily",
+    schedule="daily",
     start_date=datetime.datetime(2021, 1, 1),
     catchup=False,
     tags=["example2", "example3"],

Would fix 1 error.

[dhruvmanila]

@tirkarthi Thank you for working on that. I think it will be useful to consider the kind of deprecations and provide the auto-fixes accordingly. Your implementation looks simple enough, I'm happy to look at the PR. One suggestion for the None case would be to use the remove_argument function as there's no replacement for that. We could even rename Replacement::None to Replacement::Remove to make the intentions clear.

[tirkarthi]

Thanks @dhruvmanila, created PR #14887 . I agree that the fixes need to be categorized like numpy with Replacement enum updated with more details but I guess it will be a bigger effort and will see if I can tackled it in a separate PR.

[Lee-W]

Just created a new one for AIR303 #14764

[tirkarthi]

Is there a helper method to change the import path while keeping the symbol name? I looked at numpy replacements and they seem to do something like below where the import path remains the same and new symbol is introduced. In terms of Airflow deprecations the symbol remains the same but import path changes. Just changing the import path of the replacement through checker.importer().get_or_import_symbol causes below type of code to be generated I guess and ConflictingName is raised.

from airflow.operators.empty import EmptyOperator
from airflow.providers.standard.operators.empty import EmptyOperator

Deprecated

from numpy import all

Fix

from numpy import all, alltrue

Deprecated and removed that will cause ImportError

from airflow.operators.empty import EmptyOperator

Fix

from airflow.providers.standard.operators.empty import EmptyOperator

[MichaReiser]

@Lee-W and @uranusjr What's the status of the airflow rules? Or, we're working on the Ruff 0.10 minor and we noticed that AIR301 would meet our stabilization criteria but I don't think it makes sense to only stabilize some of them. What do you think?

[Lee-W]

I don't think we'll add new things to AIR301, but there're still some for AIR302 and AIR303 that we're currently out of bandwidth to work on

[Lee-W]

so probably keep all of them as preview for now would be better I think

[MichaReiser]

Sounds reasonable. Thanks for the update.

[Lee-W]

Hey @dhruvmanila @MichaReiser,

Recently, we've been testing these AIR rules with some existing dags to see how they work. However, we discovered there are too many warnings for even simple cases. Some might not be necessary now (backward compat layers have been added.) In addition to adding an auto fix, we'd also like to propose a reorg of these error codes. It would make airflow users migrate from Airflow 2 to Airflow 3 without so much pressure.

Original

• AIR001
  • Name: airflow-variable-name-task-id-mismatch
  • Message: Task variable name should match the task_id: "{task_id}"
  • Status:
• AIR301
  • Name: airflow-dag-no-schedule-argument
  • Message: DAG should have an explicit schedule argument
  • Status: 🧪
• AIR302
  • Name: airflow3-removal
  • Message: {deprecated} is removed in Airflow 3.0
  • Status: 🧪 🛠️
• AIR303
  • Name: airflow3-moved-to-provider
  • Message: {deprecated} is moved into {provider} provider in Airflow 3.0;
  • Status: 🧪

Proposal

• AIR001 (not changed)
  • Name: airflow-variable-name-task-id-mismatch
  • Message: Task variable name should match the task_id: "{task_id}"
  • Status:
• AIR002 (original AIR301)
  • Name: airflow-dag-no-schedule-argument
  • Message: DAG should have an explicit schedule argument
  • Note: [airflow] Move AIR301 to AIR002 #16978
• AIR302
  • Name: airflow3-removal
  • Import or usage removed without fallback in Airflow 3.0
  • Note:
    • e.g., airflow.operators.subdag
    • Some existing rules should be moved to AIR304
• AIR303
  • Name: airflow3-moved-to-provider
  • Message: Import or usage moved from airflow to airflow.providers in Airflow 3.0 with no fallback
  • Note:
    • e.g., airflow.hooks.zendesk_hook.ZendeskHook → airflow.providers.zendesk.hooks.zendesk.ZendeskHook
    • Some existing rules should be moved to AIR305
• AIR304
  • Name: airflow3-0-deprecated
  • Message: The feature is deprecated in 3.0 and will be removed in a future version (e.g., 3.1 or 4.0).
  • Note:
    • e.g., airflow.datasets.Dataset → airflow.sdk.Asset (it's expected to be moved in 3.2)
• AIR305
  • Name: airflow3-0-deprecated-and-moved-to-provider
  • Message: Feature is deprecated in 3.0 and will move from airflow to airflow.providers in a future version
  • Notes:
    • e.g., airflow.triggers.temporal → airflow.providers.standard.triggers.temporal

---

For those who want to migrate to Airflow 3.0, --select AIR302,AIR303 will be required, and we intend to make this set much smaller. For aggressive people, they can turn on --select AIR3. AIR305 is one thing I'm not that sure but I will discuss it with the community and get back to you. Also, would like to know if we move AIR301, do we need to reindex other AIR3xx as well?

[Lee-W]

Or is the intention that all AIR3XX rules are related to airflow 3?

Yes, I think it would lead to less confusion.

If so, maybe consider AIR31X for the deprecations

This is also one thing I think might work.

Maybe regroup it to something like the following?

• AIR301 (original 302)
  • Name: airflow3-removal
• AIR302
  • Name: airflow3-moved-to-provider
• AIR311
  • Name: airflow3-0-deprecated
• AIR312
  • Name: Name: airflow3-0-deprecated-and-moved-to-provider

[Lee-W]

Or is the intention that all AIR3XX rules are related to airflow 3?

Yes, I think it would lead to less confusion.

If so, maybe consider AIR31X for the deprecations

This is also one thing I think might work.

Maybe regroup it to something like the following?

* AIR301 (original 302)
  
  * Name: airflow3-removal

* AIR302
  
  * Name: airflow3-moved-to-provider

* AIR311
  
  * Name: airflow3-0-deprecated

* AIR312
  
  * Name: Name: airflow3-0-deprecated-and-moved-to-provider

Apart from this discussion, if you still have some bandwidth. It would be much appreciated if we can have these PRs reviewed 🙂 .

1. [airflow] Add autofix infrastructure to AIR302 name checks #16965 (This the PR i need more help to ensure I'm not doing thing wrong)
   • [airflow] add autofix to AIR302 ProviderName #17038 bases on [airflow] Add autofix infrastructure to AIR302 name checks #16965
2. [airflow] Add autofix for AIR302 method checks #16976, [airflow] Add autofix for AIR302 attribute checks #16977
3. [airflow] Move AIR301 to AIR002 #16978 (We'll need to make this change before we can move error ocdes)
4. [airflow] Extend AIR302 with additional symbols #17085 (new rules, probably not as urgent as otheres)

[dhruvmanila]

• I think it's fine to reuse the code, considering that the rule is still in preview.

Yes, I also think the same.

Maybe regroup it to something like the following?

• AIR301 (original 302)

  • Name: airflow3-removal

• AIR302

  • Name: airflow3-moved-to-provider

• AIR311

  • Name: airflow3-0-deprecated

• AIR312

  • Name: Name: airflow3-0-deprecated-and-moved-to-provider

Right, so is my understanding correct (?) that there are now two groups:

• AIR30x are hard deprecations, they have been removed
• AIR31x are soft deprecations, they will be removed in future release or have been moved elsewhere

Can you tell me the difference between AIR302 and AIR312 in the above proposal? They both suggests that the symbols have been moved to a provider module but I'm not sure what's the difference.

I think the overall grouping makes sense. I'll start reviewing the linked PRs.

[dhruvmanila]

@Lee-W Can you help me in what order should I be reviewing the PRs? Do you want to regroup the rules first? Does that require changing some of the symbols that are currently being detected as hard deprecation but need to be moved to soft deprecations?

Regardless, I'm going to review the PRs that you've opened as they seem to only implement the autofix for the rules.

[Lee-W]

Right, so is my understanding correct

Yes, this understanding is correct!

Can you tell me the difference between AIR302 and AIR312 in the above proposal?

AIR302 (original 303) is something like airflow.api.auth.backend.basic_auth → airflow.providers.fab.auth_manager.api.auth.backend.basic_auth. AIR312 is for something like airflow.sensors.filesystem.FileSensor → airflow.providers.standard.sensors.filesystem.FileSensor. The only difference is the provider needed in AIR312 is pre-installed with Airflow. So i'm also thinking of just merging it to AIR301 🤔 Before this decision is made. I can still work on other changes

[Lee-W]

Can you help me in what order should I be reviewing the PRs?

The regrouped has not yet been created (except for 301 to 002). Others are auto fix PRs. But after addressing all the comments on current PRs, I'll list them down here again. Thanks!

Do you want to regroup the rules first?

I'll create PRs today or tomorrow. We can look into the autofix first.

Does that require changing some of the symbols that are currently being detected as hard deprecation but need to be moved to soft deprecations?

Yes, it will.

[Lee-W]

@dhruvmanila

Would be great if I can get some feedback on #16965 first

The following four items should be easier to review, as some of them have already been approved.

#16976
#16977
#16978
#17085

Thanks!

[Lee-W]

Hey @dhruvmanila , would like to update the PR orders a bit. Many thanks!

We still need some more feedback on #16965 first.

The following 4 are easier to review.

1. [airflow] Move AIR301 to AIR002 #16978 (this blocks the error code reorg thing)
2. [airflow] Add autofix for AIR302 method checks #16976
3. [airflow] Add autofix for AIR302 attribute checks #16977
4. [airflow] Extend AIR302 with additional symbols #17085

One missing piece found in #17115 and then #17123 which reflects the latest Airflow 3.0 changes and depends on the previous PR.

---

the reorg has not yet been started. all the PR above still follows the current error code design as #16978 is still under review.

---

[dhruvmanila]

Thank you, just acknowledging this. I've added this to my todo list for today, I'll start looking into it in a couple of hours.

[Lee-W]

Much appreciated! 🙏

[Lee-W]

Hey @dhruvmanila , just a heads up. @sunank200 will help with the reorg and apply the autofix to other rules for the following few days. I'll be back next Monday.

I'll send a sample reorg PR later today (branched out from #16978) so it would be easier for @sunank200 to pick up.

Thanks for helping out!

[Lee-W]

@dhruvmanila Thanks so so so much for helping out and merging so many PRs!

---

@sunank200 As I'm out till next Monday, here are things we can do as the next steps.

Fix and adding rules

1. Get [airflow] Update oudated AIR301, AIR302 rules #17123 merged (This should be ready now. But please address the comments if there's any. Thanks!)
2. Address the comments on [airflow] Add missing AIR302 attribute check #17115 (comment) by expanding Replacement::ImportPathMoved to Replacement::ProviderName and get [airflow] Add missing AIR302 attribute check #17115 merged. This would also ease how AIR302 is fixed.

Autofix

• Apply the Replacement::AutoImport thing to Replacement::Name in crates/ruff_linter/src/rules/airflow/rules/removal_in_3.rs whenever it makes sense (take a look at [airflow] Add autofix infrastructure to AIR302 name checks #16965) ([airflow] Add autofix infrastructure to AIR302 name checks #16965 (review))
• Fix [airflow] add autofix to AIR302 ProviderName #17038 and get it merged (I just found out some of the fix is not working, but I don't have enough time to get it fixed)
• add try catch guard to method and class attribute as [airflow] Add autofix infrastructure to AIR302 name checks #16965 (comment) suggested

Reorg

• Reorganized AIR301, AIR302 into (AIR301, AIR302, AIR311, AIR312 as listed down on Organize ruff rules as required changes for Airflow 3.0 and suggested changes apache/airflow#48560 (comment) and Organize ruff rules as required changes for Airflow 3.0 and suggested changes apache/airflow#48560 (comment). I have a super early draft PR [airflow] Extract AIR312 from AIR302 rules (AIR302, AIR312) #17152. Not sure whether it helps, but you use it or create a new one.

[dhruvmanila]

Just a heads up, @ntBre is going to be taking over looking at the remaining parts of the Airflow work. Feel free to tag me if there's anything specific that requires my input / context.

[Lee-W]

Hey @ntBre , just try to list down what we have now so it would be easier for you to track 🙂

1. [airflow] add autofix to AIR302 ProviderName #17038 (comment): suggestion would be much appreciated regarding autofix
2. [airflow] Extract AIR312 from AIR302 rules (AIR302, AIR312) #17152: the first reorganizing PR. It would be great if we could get some initial feedback.
3. [airflow] Refactor AIR301 logic and fix typos (AIR301) #17293: refactoring and bu fix
4. [airflow] Add missing AIR302 attribute check #17115: bug fix

[ntBre]

Thanks for the list! I approved 1-3 with very minor comments, and I'll try to tackle 4 this evening or tomorrow morning.

[Lee-W]

Hey @ntBre , I'll try to use this issue to communicate if there's more than one PR that needs your help, so that you won't need to switch between PRs. 🙂

---

• [airflow] Extract AIR311 from AIR301 rules (AIR301, AIR311) #17310 (this is approved, addressed all the comments, and rebased from the latest main)
  • [airflow] Extend AIR311 rules #17422 is based on the PR above. It adds a few crucial checks to AIR311 without other major changes. @sunank200 will help rebase once [airflow] Extract AIR311 from AIR301 rules (AIR301, AIR311) #17310 is merged

It would be really nice if we could have them in 0.11.6 🙂

---

I'll other autofixes things once I'm back next week.

[ntBre]

I'll review these today!

[Lee-W]

Hey @ntBre ,

Just a quick update on the PRs I have today

1. [airflow] Apply auto fixes to cases where the names have changed in Airflow 3 (AIR301) #17355
2. [airflow] update existing AIR302 rules with better suggestions #17542
   1. [airflow] Apply auto fixes to cases where the names have changed in Airflow 3 (AIR302) #17553
   2. [airflow] apply Replacement::AutoImport to AIR302 #17570
3. [airflow] Apply auto fix to cases where name has been changed in Airflow 3 (AIR311) #17571

[Lee-W]

Hey @ntBre ,

Another quick update on the PRs we need your help 🙂
(The indentation indicates PR dependencies)

1. [airflow] update existing AIR302 rules with better suggestions #17542
   1. [airflow] Apply auto fixes to cases where the names have changed in Airflow 3 (AIR302) #17553
      1. [airflow] apply Replacement::AutoImport to AIR302 #17570
2. [airflow] Apply auto fix to cases where name has been changed in Airflow 3 (AIR311) #17571
   1. [airflow] Extend AIR301 rule #17598

[ntBre]

Thanks for these lists! I reviewed the top-level PRs and can move on to the next layer soon.

[Lee-W]

Hey @ntBre ,

Another quick up on the PRs we have now 🙂
(The indentation indicates PR dependencies)

1. [airflow] update existing AIR302 rules with better suggestions #17542
   1. [airflow] Apply auto fixes to cases where the names have changed in Airflow 3 (AIR302) #17553
      1. [airflow] apply Replacement::AutoImport to AIR302 #17570
2. [airflow] Extend AIR301 rule #17598