Is the CheckMigration option of dirac-dms-create-moving-request compatible with EOS-CTA storages?

[nifuki]

We are testing data transfers from disk to tape at JINR using dirac-dms-create-moving-request and encountered an issue when using the --CheckMigration (-C) option in the following command:

dirac-dms-create-moving-request -N Move_file_batch_00_from_JINR-EOS_to_JINR-TAPE -L file_list.txt -S JINR-EOS -T JINR-TAPE -C -X

Without -C option requests complete successfully, but with it requests get stuck at the CheckMigration stage after successful file transfers:

Request name='Move_Raw_01-04-2025_to_JINR-TAPE_0' ID=5624517 Status='Waiting' ('Canceled' in DB)
Created 2025-04-08 09:38:54, Updated 2025-04-08 17:12:50, NotBefore 2025-04-08 17:16:31
Owner: '/C=RU/O=RDIG/OU=users/OU=jinr.ru/CN=Nikita Balashov', Group: juno_production
  [0] Operation Type='RegisterReplica' ID=10008428 Order=0 Status='Done'
      TargetSE: JINR-TAPE - Created 2025-04-08 11:00:52, Updated 2025-04-08 11:07:47
    [01] ID=52776327 LFN='/juno/raw/2025/0104/RUN.1678.JUNODAQ.WaterFilling.ds-2.20250104185414.002.dat' Status='Done' Checksum='a513282c'
    [02] ID=52776328 LFN='/juno/raw/2025/0104/RUN.1678.JUNODAQ.WaterFilling.ds-2.20250104185449.009.dat' Status='Done' Checksum='4c94c426'
...
    [100] ID=52776497 LFN='/juno/raw/2025/0104/RUN.1680.JUNODAQ.WaterFilling.ds-2.20250104203043.512.dat' Status='Done' Checksum='e584d87e'
  [1] Operation Type='ReplicateAndRegister' ID=10004423 Order=1 Status='Done'
      SourceSE: JINR-EOS - TargetSE: JINR-TAPE - Created 2025-04-08 09:38:54, Updated 2025-04-08 11:00:52
    [01] ID=52768365 LFN='/juno/raw/2025/0104/RUN.1679.JUNODAQ.WaterFilling.ds-2.20250104194347.142.dat' Status='Done' Checksum='7a143aae' Attempts=2
    [02] ID=52768366 LFN='/juno/raw/2025/0104/RUN.1679.JUNODAQ.WaterFilling.ds-2.20250104194707.185.dat' Status='Done' Checksum='9dfcc312' Attempts=2
...
    [100] ID=52768464 LFN='/juno/raw/2025/0104/RUN.1679.JUNODAQ.WaterFilling.ds-2.20250104194133.112.dat' Status='Done' Checksum='371055dd' Attempts=2
  [2] Operation Type='CheckMigration' ID=10004424 Order=2 Status='Waiting'
      TargetSE: JINR-TAPE - Created 2025-04-08 09:38:54, Updated 2025-04-08 09:38:54
    [01] ID=52768465 LFN='/juno/raw/2025/0104/RUN.1679.JUNODAQ.WaterFilling.ds-2.20250104194347.142.dat' Status='Waiting' Checksum='7a143aae' Attempts=12
    [02] ID=52768466 LFN='/juno/raw/2025/0104/RUN.1679.JUNODAQ.WaterFilling.ds-2.20250104194707.185.dat' Status='Waiting' Checksum='9dfcc312' Attempts=12
...
    [100] ID=52768564 LFN='/juno/raw/2025/0104/RUN.1679.JUNODAQ.WaterFilling.ds-2.20250104194133.112.dat' Status='Waiting' Checksum='371055dd' Attempts=12
  [3] Operation Type='RemoveReplica' ID=10004425 Order=3 Status='Queued'
      TargetSE: JINR-EOS - Created 2025-04-08 09:38:54, Updated 2025-04-08 09:38:54
    [01] ID=52768565 LFN='/juno/raw/2025/0104/RUN.1679.JUNODAQ.WaterFilling.ds-2.20250104194347.142.dat' Status='Waiting' Checksum='7a143aae'
    [02] ID=52768566 LFN='/juno/raw/2025/0104/RUN.1679.JUNODAQ.WaterFilling.ds-2.20250104194707.185.dat' Status='Waiting' Checksum='9dfcc312'
...
    [100] ID=52768664 LFN='/juno/raw/2025/0104/RUN.1679.JUNODAQ.WaterFilling.ds-2.20250104194133.112.dat' Status='Waiting' Checksum='371055dd'

Is the CheckMigration option compatible with EOS-CTA storages?

And also it is not quite clear how this operation works, according to docs: Ensure the LFNs are migrated to tape before removing any replicas. Can someone clarify how this verification works - does it compare checksums from replicas metadata or does something else?

[andresailer]

Can someone clarify how this verification works

If I recall correctly it waits for the StorageElement to confirm the "Migrated to tape" status, so that the file does not get lost because the disk server had issues. Checksum checking is supposed to be done by FTS / EOS / CTA itself.

EDIT: The migrated bit is set by the storage element, maybe EOS/CTA does not set this.

Run the requests in DEBUG mode to see the metadata

DIRAC/src/DIRAC/DataManagementSystem/Agent/RequestOperations/CheckMigration.py

Line 48 in d74f819

self.log.debug(f"MetaData: {pformat(metaData)}")