cannot execute initial blueprint in simulated environment #8222

davepacheco · 2025-05-27T17:59:44Z

Under cargo xtask omicron-dev run-all, I tried to execute the sole built-in blueprint, but it failed to execute:

$ omdb -w nexus blueprints target enable 184d8b69-bfdf-4b68-b8a7-f79412b40003
note: Nexus URL not specified.  Will pick one from DNS.
note: using Nexus URL http://[::1]:12221
set target blueprint 184d8b69-bfdf-4b68-b8a7-f79412b40003 to enabled

$ omdb nexus background-tasks show blueprint_executor
note: Nexus URL not specified.  Will pick one from DNS.
note: using Nexus URL http://[::1]:12221
task: "blueprint_executor"
  configured period: every 1m
  currently executing: no
  last completed activation: iter 24, triggered by a dependent task completing
    started at 2025-05-27T17:54:53.806Z (3s ago) and ran for 1603ms
    target blueprint: 184d8b69-bfdf-4b68-b8a7-f79412b40003                                                                                                                                                                                                                                                                                                                                                                                  
    execution:        enabled                                                                                                                                                                                                                                                                                                                                                                                                               
    status:           completed (14 steps)                                                                                                                                                                                                                                                                                                                                                                                                  
    warning:          at: Deploy single-node clickhouse cluster: failed to initialize single-node clickhouse database: Communication Error: error sending request for url (http://[::1]:8888/init)                                                                                                                                                                                                                                          
    warning:          at: Deploy sled configs: Failed to put OmicronSledConfig {                                                                                                                                                                                                                                                                                                                                                            
...                                                                                                                                                                                                                                                                                                                                                                  
                      } to sled b6d65341-167c-41df-9b5c-41cded99c229: Error Response: status: 400 Bad Request; headers: {"content-type": "application/json", "x-request-id": "d57fb89a-2c91-4cee-a7bd-c62511d50a0e", "content-length": "124", "date": "Tue, 27 May 2025 17:54:53 GMT"}; value: Error { error_code: None, message: "Generation number unchanged but data is different", request_id: "d57fb89a-2c91-4cee-a7bd-c62511d50a0e" } 
    error:            (none)

I'll put the full error in a comment because it's too big for a GitHub issue description.

The text was updated successfully, but these errors were encountered:

davepacheco · 2025-05-27T17:59:54Z

$ omdb nexus background-tasks show blueprint_executor
note: Nexus URL not specified.  Will pick one from DNS.
note: using Nexus URL http://[::1]:12221
task: "blueprint_executor"
  configured period: every 1m
  currently executing: no
  last completed activation: iter 24, triggered by a dependent task completing
    started at 2025-05-27T17:54:53.806Z (3s ago) and ran for 1603ms
    target blueprint: 184d8b69-bfdf-4b68-b8a7-f79412b40003                                                                                                                                                                                                                                                                                                                                                                                  
    execution:        enabled                                                                                                                                                                                                                                                                                                                                                                                                               
    status:           completed (14 steps)                                                                                                                                                                                                                                                                                                                                                                                                  
    warning:          at: Deploy single-node clickhouse cluster: failed to initialize single-node clickhouse database: Communication Error: error sending request for url (http://[::1]:8888/init)                                                                                                                                                                                                                                          
    warning:          at: Deploy sled configs: Failed to put OmicronSledConfig {                                                                                                                                                                                                                                                                                                                                                            
                          generation: Generation(                                                                                                                                                                                                                                                                                                                                                                                           
                              2,                                                                                                                                                                                                                                                                                                                                                                                                            
                          ),                                                                                                                                                                                                                                                                                                                                                                                                                
                          disks: IdMap {                                                                                                                                                                                                                                                                                                                                                                                                    
                              inner: {                                                                                                                                                                                                                                                                                                                                                                                                      
                                  08e74f4d-d625-4236-a96c-1f41d2930081 (physical_disk): OmicronPhysicalDiskConfig {                                                                                                                                                                                                                                                                                                                         
                                      identity: DiskIdentity {                                                                                                                                                                                                                                                                                                                                                                              
                                          vendor: "nexus-tests",                                                                                                                                                                                                                                                                                                                                                                            
                                          model: "nexus-test-model",                                                                                                                                                                                                                                                                                                                                                                        
                                          serial: "nexus-test-disk-5",                                                                                                                                                                                                                                                                                                                                                                      
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                      id: 08e74f4d-d625-4236-a96c-1f41d2930081 (physical_disk),                                                                                                                                                                                                                                                                                                                                             
                                      pool_id: 93f85437-824e-468f-a109-b50bf781d105 (zpool),                                                                                                                                                                                                                                                                                                                                                
                                  },                                                                                                                                                                                                                                                                                                                                                                                                        
                                  09b5f0b7-00f4-45a5-b11f-437e8bdbd583 (physical_disk): OmicronPhysicalDiskConfig {                                                                                                                                                                                                                                                                                                                         
                                      identity: DiskIdentity {                                                                                                                                                                                                                                                                                                                                                                              
                                          vendor: "nexus-tests",                                                                                                                                                                                                                                                                                                                                                                            
                                          model: "nexus-test-model",                                                                                                                                                                                                                                                                                                                                                                        
                                          serial: "nexus-test-disk-6",                                                                                                                                                                                                                                                                                                                                                                      
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                      id: 09b5f0b7-00f4-45a5-b11f-437e8bdbd583 (physical_disk),                                                                                                                                                                                                                                                                                                                                             
                                      pool_id: ab1530ef-fce1-43e9-bed5-564bc5631537 (zpool),                                                                                                                                                                                                                                                                                                                                                
                                  },                                                                                                                                                                                                                                                                                                                                                                                                        
                                  1c3b3784-bb0c-462b-a77f-9c641920625d (physical_disk): OmicronPhysicalDiskConfig {                                                                                                                                                                                                                                                                                                                         
                                      identity: DiskIdentity {                                                                                                                                                                                                                                                                                                                                                                              
                                          vendor: "nexus-tests",                                                                                                                                                                                                                                                                                                                                                                            
                                          model: "nexus-test-model",                                                                                                                                                                                                                                                                                                                                                                        
                                          serial: "nexus-test-disk-1",                                                                                                                                                                                                                                                                                                                                                                      
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                      id: 1c3b3784-bb0c-462b-a77f-9c641920625d (physical_disk),                                                                                                                                                                                                                                                                                                                                             
                                      pool_id: f5ea120f-56cb-4448-ac5c-d59803dd7e20 (zpool),                                                                                                                                                                                                                                                                                                                                                
                                  },                                                                                                                                                                                                                                                                                                                                                                                                        
                                  800b8498-396c-4510-814d-f51733079c3d (physical_disk): OmicronPhysicalDiskConfig {                                                                                                                                                                                                                                                                                                                         
                                      identity: DiskIdentity {                                                                                                                                                                                                                                                                                                                                                                              
                                          vendor: "nexus-tests",                                                                                                                                                                                                                                                                                                                                                                            
                                          model: "nexus-test-model",                                                                                                                                                                                                                                                                                                                                                                        
                                          serial: "nexus-test-disk-4",                                                                                                                                                                                                                                                                                                                                                                      
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                      id: 800b8498-396c-4510-814d-f51733079c3d (physical_disk),                                                                                                                                                                                                                                                                                                                                             
                                      pool_id: 3b585a4f-e564-4164-a9e9-f5c21959eaa3 (zpool),                                                                                                                                                                                                                                                                                                                                                
                                  },                                                                                                                                                                                                                                                                                                                                                                                                        
                                  8d51f23e-b079-4518-9c16-d3281a4cb404 (physical_disk): OmicronPhysicalDiskConfig {                                                                                                                                                                                                                                                                                                                         
                                      identity: DiskIdentity {                                                                                                                                                                                                                                                                                                                                                                              
                                          vendor: "nexus-tests",                                                                                                                                                                                                                                                                                                                                                                            
                                          model: "nexus-test-model",                                                                                                                                                                                                                                                                                                                                                                        
                                          serial: "nexus-test-disk-8",                                                                                                                                                                                                                                                                                                                                                                      
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                      id: 8d51f23e-b079-4518-9c16-d3281a4cb404 (physical_disk),                                                                                                                                                                                                                                                                                                                                             
                                      pool_id: 192bdf8e-3e59-4fa5-979a-e7141055b02b (zpool),                                                                                                                                                                                                                                                                                                                                                
                                  },                                                                                                                                                                                                                                                                                                                                                                                                        
                                  ceaad35b-3b07-4035-a3c4-5f4432f725b5 (physical_disk): OmicronPhysicalDiskConfig {                                                                                                                                                                                                                                                                                                                         
                                      identity: DiskIdentity {                                                                                                                                                                                                                                                                                                                                                                              
                                          vendor: "nexus-tests",                                                                                                                                                                                                                                                                                                                                                                            
                                          model: "nexus-test-model",                                                                                                                                                                                                                                                                                                                                                                        
                                          serial: "nexus-test-disk-9",                                                                                                                                                                                                                                                                                                                                                                      
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                      id: ceaad35b-3b07-4035-a3c4-5f4432f725b5 (physical_disk),                                                                                                                                                                                                                                                                                                                                             
                                      pool_id: 2622fbc9-cfda-4391-968c-143a6150b057 (zpool),                                                                                                                                                                                                                                                                                                                                                
                                  },                                                                                                                                                                                                                                                                                                                                                                                                        
                                  dac5bc65-6ddd-4e73-adc8-77ed56745e6f (physical_disk): OmicronPhysicalDiskConfig {                                                                                                                                                                                                                                                                                                                         
                                      identity: DiskIdentity {                                                                                                                                                                                                                                                                                                                                                                              
                                          vendor: "nexus-tests",                                                                                                                                                                                                                                                                                                                                                                            
                                          model: "nexus-test-model",                                                                                                                                                                                                                                                                                                                                                                        
                                          serial: "nexus-test-disk-7",                                                                                                                                                                                                                                                                                                                                                                      
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                      id: dac5bc65-6ddd-4e73-adc8-77ed56745e6f (physical_disk),                                                                                                                                                                                                                                                                                                                                             
                                      pool_id: 74d59fcc-24a3-4dbb-bd48-3073a7efdf7d (zpool),                                                                                                                                                                                                                                                                                                                                                
                                  },                                                                                                                                                                                                                                                                                                                                                                                                        
                                  dd60ccd8-6147-4cb2-8820-9e0474266be8 (physical_disk): OmicronPhysicalDiskConfig {                                                                                                                                                                                                                                                                                                                         
                                      identity: DiskIdentity {                                                                                                                                                                                                                                                                                                                                                                              
                                          vendor: "nexus-tests",                                                                                                                                                                                                                                                                                                                                                                            
                                          model: "nexus-test-model",                                                                                                                                                                                                                                                                                                                                                                        
                                          serial: "nexus-test-disk-2",                                                                                                                                                                                                                                                                                                                                                                      
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                      id: dd60ccd8-6147-4cb2-8820-9e0474266be8 (physical_disk),                                                                                                                                                                                                                                                                                                                                             
                                      pool_id: 6a423229-5ce9-4974-af29-1b58cd8d4711 (zpool),                                                                                                                                                                                                                                                                                                                                                
                                  },                                                                                                                                                                                                                                                                                                                                                                                                        
                                  eb45e958-4ad9-484d-9838-ac0c42650d54 (physical_disk): OmicronPhysicalDiskConfig {                                                                                                                                                                                                                                                                                                                         
                                      identity: DiskIdentity {                                                                                                                                                                                                                                                                                                                                                                              
                                          vendor: "nexus-tests",                                                                                                                                                                                                                                                                                                                                                                            
                                          model: "nexus-test-model",                                                                                                                                                                                                                                                                                                                                                                        
                                          serial: "nexus-test-disk-3",                                                                                                                                                                                                                                                                                                                                                                      
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                      id: eb45e958-4ad9-484d-9838-ac0c42650d54 (physical_disk),                                                                                                                                                                                                                                                                                                                                             
                                      pool_id: 68544ad1-aa4f-4c9f-ae42-72c1962bad4e (zpool),                                                                                                                                                                                                                                                                                                                                                
                                  },                                                                                                                                                                                                                                                                                                                                                                                                        
                                  ecd09714-0d0a-4582-8a2a-a595c83e863e (physical_disk): OmicronPhysicalDiskConfig {                                                                                                                                                                                                                                                                                                                         
                                      identity: DiskIdentity {                                                                                                                                                                                                                                                                                                                                                                              
                                          vendor: "nexus-tests",                                                                                                                                                                                                                                                                                                                                                                            
                                          model: "nexus-test-model",                                                                                                                                                                                                                                                                                                                                                                        
                                          serial: "nexus-test-disk-0",                                                                                                                                                                                                                                                                                                                                                                      
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                      id: ecd09714-0d0a-4582-8a2a-a595c83e863e (physical_disk),                                                                                                                                                                                                                                                                                                                                             
                                      pool_id: aa2ac962-41bb-4a0b-8251-09be9024d1f5 (zpool),                                                                                                                                                                                                                                                                                                                                                
                                  },                                                                                                                                                                                                                                                                                                                                                                                                        
                              },                                                                                                                                                                                                                                                                                                                                                                                                            
                          },                                                                                                                                                                                                                                                                                                                                                                                                                
                          datasets: IdMap {                                                                                                                                                                                                                                                                                                                                                                                                 
                              inner: {                                                                                                                                                                                                                                                                                                                                                                                                      
                                  21d05d26-5531-406b-9f8e-b282d6fca024 (dataset): DatasetConfig {                                                                                                                                                                                                                                                                                                                                           
                                      id: 21d05d26-5531-406b-9f8e-b282d6fca024 (dataset),                                                                                                                                                                                                                                                                                                                                                   
                                      name: DatasetName {                                                                                                                                                                                                                                                                                                                                                                                   
                                          pool_name: ZpoolName {                                                                                                                                                                                                                                                                                                                                                                            
                                              id: 93f85437-824e-468f-a109-b50bf781d105 (zpool),                                                                                                                                                                                                                                                                                                                                             
                                              kind: External,                                                                                                                                                                                                                                                                                                                                                                               
                                          },                                                                                                                                                                                                                                                                                                                                                                                                
                                          kind: TransientZone {                                                                                                                                                                                                                                                                                                                                                                             
                                              name: "oxz_crucible_pantry_58b067c2-9c99-4673-80bb-b11bca6b8d0e",                                                                                                                                                                                                                                                                                                                             
                                          },                                                                                                                                                                                                                                                                                                                                                                                                
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                      inner: SharedDatasetConfig {                                                                                                                                                                                                                                                                                                                                                                          
                                          compression: Off,                                                                                                                                                                                                                                                                                                                                                                                 
                                          quota: None,                                                                                                                                                                                                                                                                                                                                                                                      
                                          reservation: None,                                                                                                                                                                                                                                                                                                                                                                                
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                  },                                                                                                                                                                                                                                                                                                                                                                                                        
                                  4f566fc9-f39c-4ed4-91f2-2699fe09865c (dataset): DatasetConfig {                                                                                                                                                                                                                                                                                                                                           
                                      id: 4f566fc9-f39c-4ed4-91f2-2699fe09865c (dataset),                                                                                                                                                                                                                                                                                                                                                   
                                      name: DatasetName {                                                                                                                                                                                                                                                                                                                                                                                   
                                          pool_name: ZpoolName {                                                                                                                                                                                                                                                                                                                                                                            
                                              id: 68544ad1-aa4f-4c9f-ae42-72c1962bad4e (zpool),                                                                                                                                                                                                                                                                                                                                             
                                              kind: External,                                                                                                                                                                                                                                                                                                                                                                               
                                          },                                                                                                                                                                                                                                                                                                                                                                                                
                                          kind: TransientZone {                                                                                                                                                                                                                                                                                                                                                                             
                                              name: "oxz_external_dns_01259a27-b760-4378-a05f-98fb0261a673",                                                                                                                                                                                                                                                                                                                                
                                          },                                                                                                                                                                                                                                                                                                                                                                                                
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                      inner: SharedDatasetConfig {                                                                                                                                                                                                                                                                                                                                                                          
                                          compression: Off,                                                                                                                                                                                                                                                                                                                                                                                 
                                          quota: None,                                                                                                                                                                                                                                                                                                                                                                                      
                                          reservation: None,                                                                                                                                                                                                                                                                                                                                                                                
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                  },                                                                                                                                                                                                                                                                                                                                                                                                        
                                  611f0480-f619-4ddc-a9f7-7c072116155e (dataset): DatasetConfig {                                                                                                                                                                                                                                                                                                                                           
                                      id: 611f0480-f619-4ddc-a9f7-7c072116155e (dataset),                                                                                                                                                                                                                                                                                                                                                   
                                      name: DatasetName {                                                                                                                                                                                                                                                                                                                                                                                   
                                          pool_name: ZpoolName {                                                                                                                                                                                                                                                                                                                                                                            
                                              id: f5ea120f-56cb-4448-ac5c-d59803dd7e20 (zpool),                                                                                                                                                                                                                                                                                                                                             
                                              kind: External,                                                                                                                                                                                                                                                                                                                                                                               
                                          },                                                                                                                                                                                                                                                                                                                                                                                                
                                          kind: TransientZone {                                                                                                                                                                                                                                                                                                                                                                             
                                              name: "oxz_clickhouse_50e2f8de-d86d-492f-ae3c-f692dd020805",                                                                                                                                                                                                                                                                                                                                  
                                          },                                                                                                                                                                                                                                                                                                                                                                                                
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                      inner: SharedDatasetConfig {                                                                                                                                                                                                                                                                                                                                                                          
                                          compression: Off,                                                                                                                                                                                                                                                                                                                                                                                 
                                          quota: None,                                                                                                                                                                                                                                                                                                                                                                                      
                                          reservation: None,                                                                                                                                                                                                                                                                                                                                                                                
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                  },                                                                                                                                                                                                                                                                                                                                                                                                        
                                  70ff3f41-264c-4c4f-a784-78a2949c25ae (dataset): DatasetConfig {                                                                                                                                                                                                                                                                                                                                           
                                      id: 70ff3f41-264c-4c4f-a784-78a2949c25ae (dataset),                                                                                                                                                                                                                                                                                                                                                   
                                      name: DatasetName {                                                                                                                                                                                                                                                                                                                                                                                   
                                          pool_name: ZpoolName {                                                                                                                                                                                                                                                                                                                                                                            
                                              id: aa2ac962-41bb-4a0b-8251-09be9024d1f5 (zpool),                                                                                                                                                                                                                                                                                                                                             
                                              kind: External,                                                                                                                                                                                                                                                                                                                                                                               
                                          },                                                                                                                                                                                                                                                                                                                                                                                                
                                          kind: TransientZone {                                                                                                                                                                                                                                                                                                                                                                             
                                              name: "oxz_cockroachdb_144fc64a-1e4a-4e91-b859-d73f514ec3ec",                                                                                                                                                                                                                                                                                                                                 
                                          },                                                                                                                                                                                                                                                                                                                                                                                                
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                      inner: SharedDatasetConfig {                                                                                                                                                                                                                                                                                                                                                                          
                                          compression: Off,                                                                                                                                                                                                                                                                                                                                                                                 
                                          quota: None,                                                                                                                                                                                                                                                                                                                                                                                      
                                          reservation: None,                                                                                                                                                                                                                                                                                                                                                                                
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                  },                                                                                                                                                                                                                                                                                                                                                                                                        
                                  a8edfd8c-654c-4e13-be3c-2823169d200c (dataset): DatasetConfig {                                                                                                                                                                                                                                                                                                                                           
                                      id: a8edfd8c-654c-4e13-be3c-2823169d200c (dataset),                                                                                                                                                                                                                                                                                                                                                   
                                      name: DatasetName {                                                                                                                                                                                                                                                                                                                                                                                   
                                          pool_name: ZpoolName {                                                                                                                                                                                                                                                                                                                                                                            
                                              id: 3b585a4f-e564-4164-a9e9-f5c21959eaa3 (zpool),                                                                                                                                                                                                                                                                                                                                             
                                              kind: External,                                                                                                                                                                                                                                                                                                                                                                               
                                          },                                                                                                                                                                                                                                                                                                                                                                                                
                                          kind: TransientZone {                                                                                                                                                                                                                                                                                                                                                                             
                                              name: "oxz_nexus_e6bff1ff-24fb-49dc-a54e-c6a350cd4d6c",                                                                                                                                                                                                                                                                                                                                       
                                          },                                                                                                                                                                                                                                                                                                                                                                                                
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                      inner: SharedDatasetConfig {                                                                                                                                                                                                                                                                                                                                                                          
                                          compression: Off,                                                                                                                                                                                                                                                                                                                                                                                 
                                          quota: None,                                                                                                                                                                                                                                                                                                                                                                                      
                                          reservation: None,                                                                                                                                                                                                                                                                                                                                                                                
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                  },                                                                                                                                                                                                                                                                                                                                                                                                        
                                  e472d225-ec25-49c1-b83c-524431b4e63f (dataset): DatasetConfig {                                                                                                                                                                                                                                                                                                                                           
                                      id: e472d225-ec25-49c1-b83c-524431b4e63f (dataset),                                                                                                                                                                                                                                                                                                                                                   
                                      name: DatasetName {                                                                                                                                                                                                                                                                                                                                                                                   
                                          pool_name: ZpoolName {                                                                                                                                                                                                                                                                                                                                                                            
                                              id: 6a423229-5ce9-4974-af29-1b58cd8d4711 (zpool),                                                                                                                                                                                                                                                                                                                                             
                                              kind: External,                                                                                                                                                                                                                                                                                                                                                                               
                                          },                                                                                                                                                                                                                                                                                                                                                                                                
                                          kind: TransientZone {                                                                                                                                                                                                                                                                                                                                                                             
                                              name: "oxz_internal_dns_6fab3464-463d-40b3-a95e-44b94633eff3",                                                                                                                                                                                                                                                                                                                                
                                          },                                                                                                                                                                                                                                                                                                                                                                                                
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                      inner: SharedDatasetConfig {                                                                                                                                                                                                                                                                                                                                                                          
                                          compression: Off,                                                                                                                                                                                                                                                                                                                                                                                 
                                          quota: None,                                                                                                                                                                                                                                                                                                                                                                                      
                                          reservation: None,                                                                                                                                                                                                                                                                                                                                                                                
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                  },                                                                                                                                                                                                                                                                                                                                                                                                        
                              },                                                                                                                                                                                                                                                                                                                                                                                                            
                          },                                                                                                                                                                                                                                                                                                                                                                                                                
                          zones: IdMap {                                                                                                                                                                                                                                                                                                                                                                                                    
                              inner: {                                                                                                                                                                                                                                                                                                                                                                                                      
                                  01259a27-b760-4378-a05f-98fb0261a673 (service): OmicronZoneConfig {                                                                                                                                                                                                                                                                                                                                       
                                      id: 01259a27-b760-4378-a05f-98fb0261a673 (service),                                                                                                                                                                                                                                                                                                                                                   
                                      filesystem_pool: Some(                                                                                                                                                                                                                                                                                                                                                                                
                                          ZpoolName {                                                                                                                                                                                                                                                                                                                                                                                       
                                              id: 68544ad1-aa4f-4c9f-ae42-72c1962bad4e (zpool),                                                                                                                                                                                                                                                                                                                                             
                                              kind: External,                                                                                                                                                                                                                                                                                                                                                                               
                                          },                                                                                                                                                                                                                                                                                                                                                                                                
                                      ),                                                                                                                                                                                                                                                                                                                                                                                                    
                                      zone_type: ExternalDns {                                                                                                                                                                                                                                                                                                                                                                              
                                          dataset: OmicronZoneDataset {                                                                                                                                                                                                                                                                                                                                                                     
                                              pool_name: ZpoolName {                                                                                                                                                                                                                                                                                                                                                                        
                                                  id: 68544ad1-aa4f-4c9f-ae42-72c1962bad4e (zpool),                                                                                                                                                                                                                                                                                                                                         
                                                  kind: External,                                                                                                                                                                                                                                                                                                                                                                           
                                              },                                                                                                                                                                                                                                                                                                                                                                                            
                                          },                                                                                                                                                                                                                                                                                                                                                                                                
                                          http_address: [::1]:35702,                                                                                                                                                                                                                                                                                                                                                                        
                                          dns_address: [::1]:40484,                                                                                                                                                                                                                                                                                                                                                                         
                                          nic: NetworkInterface {                                                                                                                                                                                                                                                                                                                                                                           
                                              id: d4c4335f-02e0-4d74-a9bb-1dc8369fdaf2,                                                                                                                                                                                                                                                                                                                                                     
                                              kind: Service {                                                                                                                                                                                                                                                                                                                                                                               
                                                  id: 01259a27-b760-4378-a05f-98fb0261a673,                                                                                                                                                                                                                                                                                                                                                 
                                              },                                                                                                                                                                                                                                                                                                                                                                                            
                                              name: Name(                                                                                                                                                                                                                                                                                                                                                                                   
                                                  "external-dns-01259a27-b760-4378-a05f-98fb0261a673",                                                                                                                                                                                                                                                                                                                                      
                                              ),                                                                                                                                                                                                                                                                                                                                                                                            
                                              ip: 172.30.1.6,                                                                                                                                                                                                                                                                                                                                                                               
                                              mac: MacAddr(                                                                                                                                                                                                                                                                                                                                                                                 
                                                  MacAddr6(                                                                                                                                                                                                                                                                                                                                                                                 
                                                      [                                                                                                                                                                                                                                                                                                                                                                                     
                                                          168,                                                                                                                                                                                                                                                                                                                                                                              
                                                          64,                                                                                                                                                                                                                                                                                                                                                                               
                                                          37,                                                                                                                                                                                                                                                                                                                                                                               
                                                          255,                                                                                                                                                                                                                                                                                                                                                                              
                                                          128,                                                                                                                                                                                                                                                                                                                                                                              
                                                          0,                                                                                                                                                                                                                                                                                                                                                                                
                                                      ],                                                                                                                                                                                                                                                                                                                                                                                    
                                                  ),                                                                                                                                                                                                                                                                                                                                                                                        
                                              ),                                                                                                                                                                                                                                                                                                                                                                                            
                                              subnet: V4(                                                                                                                                                                                                                                                                                                                                                                                   
                                                  Ipv4Net {                                                                                                                                                                                                                                                                                                                                                                                 
                                                      addr: 172.30.1.0,                                                                                                                                                                                                                                                                                                                                                                     
                                                      width: 24,                                                                                                                                                                                                                                                                                                                                                                            
                                                  },                                                                                                                                                                                                                                                                                                                                                                                        
                                              ),                                                                                                                                                                                                                                                                                                                                                                                            
                                              vni: Vni(                                                                                                                                                                                                                                                                                                                                                                                     
                                                  100,                                                                                                                                                                                                                                                                                                                                                                                      
                                              ),                                                                                                                                                                                                                                                                                                                                                                                            
                                              primary: true,                                                                                                                                                                                                                                                                                                                                                                                
                                              slot: 0,                                                                                                                                                                                                                                                                                                                                                                                      
                                              transit_ips: [],                                                                                                                                                                                                                                                                                                                                                                              
                                          },                                                                                                                                                                                                                                                                                                                                                                                                
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                      image_source: InstallDataset,                                                                                                                                                                                                                                                                                                                                                                         
                                  },                                                                                                                                                                                                                                                                                                                                                                                                        
                                  144fc64a-1e4a-4e91-b859-d73f514ec3ec (service): OmicronZoneConfig {                                                                                                                                                                                                                                                                                                                                       
                                      id: 144fc64a-1e4a-4e91-b859-d73f514ec3ec (service),                                                                                                                                                                                                                                                                                                                                                   
                                      filesystem_pool: Some(                                                                                                                                                                                                                                                                                                                                                                                
                                          ZpoolName {                                                                                                                                                                                                                                                                                                                                                                                       
                                              id: aa2ac962-41bb-4a0b-8251-09be9024d1f5 (zpool),                                                                                                                                                                                                                                                                                                                                             
                                              kind: External,                                                                                                                                                                                                                                                                                                                                                                               
                                          },                                                                                                                                                                                                                                                                                                                                                                                                
                                      ),                                                                                                                                                                                                                                                                                                                                                                                                    
                                      zone_type: CockroachDb {                                                                                                                                                                                                                                                                                                                                                                              
                                          address: [::1]:40316,                                                                                                                                                                                                                                                                                                                                                                             
                                          dataset: OmicronZoneDataset {                                                                                                                                                                                                                                                                                                                                                                     
                                              pool_name: ZpoolName {                                                                                                                                                                                                                                                                                                                                                                        
                                                  id: aa2ac962-41bb-4a0b-8251-09be9024d1f5 (zpool),                                                                                                                                                                                                                                                                                                                                         
                                                  kind: External,                                                                                                                                                                                                                                                                                                                                                                           
                                              },                                                                                                                                                                                                                                                                                                                                                                                            
                                          },                                                                                                                                                                                                                                                                                                                                                                                                
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                      image_source: InstallDataset,                                                                                                                                                                                                                                                                                                                                                                         
                                  },                                                                                                                                                                                                                                                                                                                                                                                                        
                                  50e2f8de-d86d-492f-ae3c-f692dd020805 (service): OmicronZoneConfig {                                                                                                                                                                                                                                                                                                                                       
                                      id: 50e2f8de-d86d-492f-ae3c-f692dd020805 (service),                                                                                                                                                                                                                                                                                                                                                   
                                      filesystem_pool: Some(                                                                                                                                                                                                                                                                                                                                                                                
                                          ZpoolName {                                                                                                                                                                                                                                                                                                                                                                                       
                                              id: f5ea120f-56cb-4448-ac5c-d59803dd7e20 (zpool),                                                                                                                                                                                                                                                                                                                                             
                                              kind: External,                                                                                                                                                                                                                                                                                                                                                                               
                                          },                                                                                                                                                                                                                                                                                                                                                                                                
                                      ),                                                                                                                                                                                                                                                                                                                                                                                                    
                                      zone_type: Clickhouse {                                                                                                                                                                                                                                                                                                                                                                               
                                          address: [::1]:33876,                                                                                                                                                                                                                                                                                                                                                                             
                                          dataset: OmicronZoneDataset {                                                                                                                                                                                                                                                                                                                                                                     
                                              pool_name: ZpoolName {                                                                                                                                                                                                                                                                                                                                                                        
                                                  id: f5ea120f-56cb-4448-ac5c-d59803dd7e20 (zpool),                                                                                                                                                                                                                                                                                                                                         
                                                  kind: External,                                                                                                                                                                                                                                                                                                                                                                           
                                              },                                                                                                                                                                                                                                                                                                                                                                                            
                                          },                                                                                                                                                                                                                                                                                                                                                                                                
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                      image_source: InstallDataset,                                                                                                                                                                                                                                                                                                                                                                         
                                  },                                                                                                                                                                                                                                                                                                                                                                                                        
                                  58b067c2-9c99-4673-80bb-b11bca6b8d0e (service): OmicronZoneConfig {                                                                                                                                                                                                                                                                                                                                       
                                      id: 58b067c2-9c99-4673-80bb-b11bca6b8d0e (service),                                                                                                                                                                                                                                                                                                                                                   
                                      filesystem_pool: Some(                                                                                                                                                                                                                                                                                                                                                                                
                                          ZpoolName {                                                                                                                                                                                                                                                                                                                                                                                       
                                              id: 93f85437-824e-468f-a109-b50bf781d105 (zpool),                                                                                                                                                                                                                                                                                                                                             
                                              kind: External,                                                                                                                                                                                                                                                                                                                                                                               
                                          },                                                                                                                                                                                                                                                                                                                                                                                                
                                      ),                                                                                                                                                                                                                                                                                                                                                                                                    
                                      zone_type: CruciblePantry {                                                                                                                                                                                                                                                                                                                                                                           
                                          address: [::1]:56517,                                                                                                                                                                                                                                                                                                                                                                             
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                      image_source: InstallDataset,                                                                                                                                                                                                                                                                                                                                                                         
                                  },                                                                                                                                                                                                                                                                                                                                                                                                        
                                  6fab3464-463d-40b3-a95e-44b94633eff3 (service): OmicronZoneConfig {                                                                                                                                                                                                                                                                                                                                       
                                      id: 6fab3464-463d-40b3-a95e-44b94633eff3 (service),                                                                                                                                                                                                                                                                                                                                                   
                                      filesystem_pool: Some(                                                                                                                                                                                                                                                                                                                                                                                
                                          ZpoolName {                                                                                                                                                                                                                                                                                                                                                                                       
                                              id: 6a423229-5ce9-4974-af29-1b58cd8d4711 (zpool),                                                                                                                                                                                                                                                                                                                                             
                                              kind: External,                                                                                                                                                                                                                                                                                                                                                                               
                                          },                                                                                                                                                                                                                                                                                                                                                                                                
                                      ),                                                                                                                                                                                                                                                                                                                                                                                                    
                                      zone_type: InternalDns {                                                                                                                                                                                                                                                                                                                                                                              
                                          dataset: OmicronZoneDataset {                                                                                                                                                                                                                                                                                                                                                                     
                                              pool_name: ZpoolName {                                                                                                                                                                                                                                                                                                                                                                        
                                                  id: 6a423229-5ce9-4974-af29-1b58cd8d4711 (zpool),                                                                                                                                                                                                                                                                                                                                         
                                                  kind: External,                                                                                                                                                                                                                                                                                                                                                                           
                                              },                                                                                                                                                                                                                                                                                                                                                                                            
                                          },                                                                                                                                                                                                                                                                                                                                                                                                
                                          http_address: [::1]:61787,                                                                                                                                                                                                                                                                                                                                                                        
                                          dns_address: [::1]:41524,                                                                                                                                                                                                                                                                                                                                                                         
                                          gz_address: ::1,                                                                                                                                                                                                                                                                                                                                                                                  
                                          gz_address_index: 0,                                                                                                                                                                                                                                                                                                                                                                              
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                      image_source: InstallDataset,                                                                                                                                                                                                                                                                                                                                                                         
                                  },                                                                                                                                                                                                                                                                                                                                                                                                        
                                  e6bff1ff-24fb-49dc-a54e-c6a350cd4d6c (service): OmicronZoneConfig {                                                                                                                                                                                                                                                                                                                                       
                                      id: e6bff1ff-24fb-49dc-a54e-c6a350cd4d6c (service),                                                                                                                                                                                                                                                                                                                                                   
                                      filesystem_pool: Some(                                                                                                                                                                                                                                                                                                                                                                                
                                          ZpoolName {                                                                                                                                                                                                                                                                                                                                                                                       
                                              id: 3b585a4f-e564-4164-a9e9-f5c21959eaa3 (zpool),                                                                                                                                                                                                                                                                                                                                             
                                              kind: External,                                                                                                                                                                                                                                                                                                                                                                               
                                          },                                                                                                                                                                                                                                                                                                                                                                                                
                                      ),                                                                                                                                                                                                                                                                                                                                                                                                    
                                      zone_type: Nexus {                                                                                                                                                                                                                                                                                                                                                                                    
                                          internal_address: [::1]:12221,                                                                                                                                                                                                                                                                                                                                                                    
                                          external_ip: 127.0.0.1,                                                                                                                                                                                                                                                                                                                                                                           
                                          nic: NetworkInterface {                                                                                                                                                                                                                                                                                                                                                                           
                                              id: c4542346-6ff7-4f4a-8671-615c163bd045,                                                                                                                                                                                                                                                                                                                                                     
                                              kind: Service {                                                                                                                                                                                                                                                                                                                                                                               
                                                  id: e6bff1ff-24fb-49dc-a54e-c6a350cd4d6c,                                                                                                                                                                                                                                                                                                                                                 
                                              },                                                                                                                                                                                                                                                                                                                                                                                            
                                              name: Name(                                                                                                                                                                                                                                                                                                                                                                                   
                                                  "nexus-e6bff1ff-24fb-49dc-a54e-c6a350cd4d6c",                                                                                                                                                                                                                                                                                                                                             
                                              ),                                                                                                                                                                                                                                                                                                                                                                                            
                                              ip: 172.30.2.6,                                                                                                                                                                                                                                                                                                                                                                               
                                              mac: MacAddr(                                                                                                                                                                                                                                                                                                                                                                                 
                                                  MacAddr6(                                                                                                                                                                                                                                                                                                                                                                                 
                                                      [                                                                                                                                                                                                                                                                                                                                                                                     
                                                          168,                                                                                                                                                                                                                                                                                                                                                                              
                                                          64,                                                                                                                                                                                                                                                                                                                                                                               
                                                          37,                                                                                                                                                                                                                                                                                                                                                                               
                                                          255,                                                                                                                                                                                                                                                                                                                                                                              
                                                          128,                                                                                                                                                                                                                                                                                                                                                                              
                                                          1,                                                                                                                                                                                                                                                                                                                                                                                
                                                      ],                                                                                                                                                                                                                                                                                                                                                                                    
                                                  ),                                                                                                                                                                                                                                                                                                                                                                                        
                                              ),                                                                                                                                                                                                                                                                                                                                                                                            
                                              subnet: V4(                                                                                                                                                                                                                                                                                                                                                                                   
                                                  Ipv4Net {                                                                                                                                                                                                                                                                                                                                                                                 
                                                      addr: 172.30.2.0,                                                                                                                                                                                                                                                                                                                                                                     
                                                      width: 24,                                                                                                                                                                                                                                                                                                                                                                            
                                                  },                                                                                                                                                                                                                                                                                                                                                                                        
                                              ),                                                                                                                                                                                                                                                                                                                                                                                            
                                              vni: Vni(                                                                                                                                                                                                                                                                                                                                                                                     
                                                  100,                                                                                                                                                                                                                                                                                                                                                                                      
                                              ),                                                                                                                                                                                                                                                                                                                                                                                            
                                              primary: true,                                                                                                                                                                                                                                                                                                                                                                                
                                              slot: 0,                                                                                                                                                                                                                                                                                                                                                                                      
                                              transit_ips: [],                                                                                                                                                                                                                                                                                                                                                                              
                                          },                                                                                                                                                                                                                                                                                                                                                                                                
                                          external_tls: false,                                                                                                                                                                                                                                                                                                                                                                              
                                          external_dns_servers: [                                                                                                                                                                                                                                                                                                                                                                           
                                              1.1.1.1,                                                                                                                                                                                                                                                                                                                                                                                      
                                              9.9.9.9,                                                                                                                                                                                                                                                                                                                                                                                      
                                          ],                                                                                                                                                                                                                                                                                                                                                                                                
                                      },                                                                                                                                                                                                                                                                                                                                                                                                    
                                      image_source: InstallDataset,                                                                                                                                                                                                                                                                                                                                                                         
                                  },                                                                                                                                                                                                                                                                                                                                                                                                        
                              },                                                                                                                                                                                                                                                                                                                                                                                                            
                          },                                                                                                                                                                                                                                                                                                                                                                                                                
                          remove_mupdate_override: None,                                                                                                                                                                                                                                                                                                                                                                                    
                      } to sled b6d65341-167c-41df-9b5c-41cded99c229: Error Response: status: 400 Bad Request; headers: {"content-type": "application/json", "x-request-id": "d57fb89a-2c91-4cee-a7bd-c62511d50a0e", "content-length": "124", "date": "Tue, 27 May 2025 17:54:53 GMT"}; value: Error { error_code: None, message: "Generation number unchanged but data is different", request_id: "d57fb89a-2c91-4cee-a7bd-c62511d50a0e" } 
    error:            (none)

jgallagher · 2025-05-27T18:11:11Z

Does omicron-dev run-all go through RSS based on the simulated sled-agents? Or does it piggy back on nexus-test-utils? I think it's the latter, which means this is probably related to (or caused by) #7081

davepacheco · 2025-06-11T17:02:21Z

It uses nexus-test-utils (that is basically its purpose -- to provide an interactive version of the same environment that tests use).

davepacheco mentioned this issue Jun 11, 2025

cannot run planner in simulated environments #8221

Open

plotnick mentioned this issue Jun 11, 2025

blueprint_planner background task #8287

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

cannot execute initial blueprint in simulated environment #8222

cannot execute initial blueprint in simulated environment #8222

davepacheco commented May 27, 2025

davepacheco commented May 27, 2025

Uh oh!

jgallagher commented May 27, 2025

Uh oh!

davepacheco commented Jun 11, 2025

Uh oh!

cannot execute initial blueprint in simulated environment #8222

cannot execute initial blueprint in simulated environment #8222

Comments

davepacheco commented May 27, 2025

davepacheco commented May 27, 2025

Uh oh!

jgallagher commented May 27, 2025

Uh oh!

davepacheco commented Jun 11, 2025

Uh oh!