Windows actions

[kingbri1]

Hi all,

I've been maintaining a fork with Windows wheels for the past ~9 months. Thanks to @jllllll for the initial action that I worked off of. It's been known that Windows is supported by Flash-attention 2 due to the existence of these wheels and various commits that have fixed Windows builds in the past.

As such, I'd like to get these actions working in the official repo so windows wheels can be available when installing via pip and actions will trigger per-release.

I've currently published the actions that do build wheels, but they'll need editing to support being triggered on release and publishing to pypi via twine. I'm not familiar with twine, so I'll need some help there if possible.

cc: @tridao

[umarbutler]

Is this ready to merge? If so, @tridao can we get this merged?

[Ratinod]

I give up... I can't get flash-attention to compile... Can we get official support for Windows wheels? @tridao Please...

[kingbri1]

I have windows releases located here in my fork. These should work. If you still want to build yourself, consider referencing the actions YML here

However, since building with backward/dropout maxes out the time for a Github action (6 hours), the build does not include these features and requires a self-compile if you want to train with FA2 on Windows.

It seems like this repo is all-in on FA3 now since there hasn't been a release in some time.

[sdbds]

I have windows releases located here in my fork. These should work. If you still want to build yourself, consider referencing the actions YML here

However, since building with backward/dropout maxes out the time for a Github action (6 hours), the build does not include these features and requires a self-compile if you want to train with FA2 on Windows.

It seems like this repo is all-in on FA3 now since there hasn't been a release in some time.

Thank you for your contribution.
Do i need to set additional parameters for the backward compilation, or can it be compiled directly using the original repository?

[kingbri1]

@sdbds bwd is enabled by default in this repo. My fork has it disabled to avoid running out of time on GitHub actions.

[sdbds]

@sdbds bwd is enabled by default in this repo. My fork has it disabled to avoid running out of time on GitHub actions.

Thank you for your response. I have reviewed the code and manually compiled it.

[FurkanGozukara]

we really need Windows builds

it is not hard i compiled for cu128 and works on RTX 5090 or 3090 tested for flash attention 2

now trying flash attention 3 i hope works