JavaScript heap out of memory while trying to start json-schema-fhir

[virtualit786]

Issue workflow progress

From latest code on master branch, running the example graphql-mesh/examples/json-schema-fhir gives the JavaScript heap out of memory exception.

To Reproduce
Steps to reproduce the behavior:

1. git clone https://github.com/Urigo/graphql-mesh.git
2. cd graphql-mesh/examples/json-schema-fhir
3. yarn install
4. yarn start
   FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Environment:

• OS: MacOS Bug Sur
• @graphql-mesh/...:
• NodeJS: v14.19.1

Additional context
Following is the complete JS stack trace

192 :: graphql-mesh/examples/json-schema-fhir ‹master*› % yarn start                                                                                        
yarn run v1.22.18
$ yarn download-fhir-schema
$ node scripts/download-fhir-schema.js
$ mesh dev
💡 🕸️  Mesh - Server Generating the unified schema...
💡 🕸️  Mesh - FHIR Dereferencing the bundle

<--- Last few GCs --->

[30024:0x104c00000]    50313 ms: Mark-sweep (reduce) 4093.7 (4100.8) -> 4093.3 (4103.8) MB, 5078.8 / 0.0 ms  (+ 2.2 ms in 1685 steps since start of marking, biggest step 0.7 ms, walltime since start of marking 6218 ms) (average mu = 0.261, current mu = 0.[30024:0x104c00000]    55368 ms: Mark-sweep (reduce) 4094.3 (4104.8) -> 4093.9 (4105.0) MB, 5053.5 / 0.0 ms  (average mu = 0.150, current mu = 0.000) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x1013221f5 node::Abort() (.cold.1) [/Users/farooqhameed/.nvm/versions/node/v14.19.1/bin/node]
 2: 0x1000b6399 node::Abort() [/Users/farooqhameed/.nvm/versions/node/v14.19.1/bin/node]
 3: 0x1000b650f node::OnFatalError(char const*, char const*) [/Users/farooqhameed/.nvm/versions/node/v14.19.1/bin/node]
 4: 0x100201497 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/Users/farooqhameed/.nvm/versions/node/v14.19.1/bin/node]
 5: 0x100201433 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/farooqhameed/.nvm/versions/node/v14.19.1/bin/node]
 6: 0x1003af545 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/farooqhameed/.nvm/versions/node/v14.19.1/bin/node]
 7: 0x1003b0fea v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/Users/farooqhameed/.nvm/versions/node/v14.19.1/bin/node]
 8: 0x1003ac715 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/farooqhameed/.nvm/versions/node/v14.19.1/bin/node]
 9: 0x1003aa040 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/farooqhameed/.nvm/versions/node/v14.19.1/bin/node]
10: 0x1003b876a v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/farooqhameed/.nvm/versions/node/v14.19.1/bin/node]
11: 0x1003b87f1 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/farooqhameed/.nvm/versions/node/v14.19.1/bin/node]
12: 0x1003865c2 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/Users/farooqhameed/.nvm/versions/node/v14.19.1/bin/node]
13: 0x100706628 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/Users/farooqhameed/.nvm/versions/node/v14.19.1/bin/node]
14: 0x100a8e7d9 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/Users/farooqhameed/.nvm/versions/node/v14.19.1/bin/node]
error Command failed with signal "SIGABRT".
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
192 :: graphql-mesh/examples/json-schema-fhir ‹master*› %                                                                                                                                                                                                                   ```

[virtualit786]

Any update on this issue?

[sesmi123]

I too am facing this issue. Any solutions for this?

[farooqhameed-fistix]

Hi, there, I am also facing this and have tried different ticks to increase JS heap size etc but it always crashes even at 16gb of ram machine. Unfortunately team is not very responsive on replying to issues here on GitHub and neither on their discord channel.

[dr3]

I think i'm having the same issue as this when migrating to the new-openapi handler

Will report back here if i'm able to find the issue, or raise a new issue if unrelated :)

[dr3]

This should now be resolved 🎉 memory issue was resolved in #4237 and surrounding changes

[ardatan]

Closing this issue then! :) Today we have tests for complex and large schemas like FHIR, Jira and so on.

[sesmi123]

Hi, when I clone the repo and run the json-schema-fhir example, I am getting error like this. The json file is present in the given location but fails to load. Any ideas why it is failing?

 🕸️  Mesh - FHIR  Failed to generate the schema Error: Unable to load ./fhir.schema.json from C:\Users\Sesmi\Downloads\json-schema-fhir
    at C:\Users\Sesmi\Downloads\json-schema-fhir\node_modules\json-machete\index.js:157:35
    at async dereferenceObject (C:\Users\Sesmi\Downloads\json-schema-fhir\node_modules\json-machete\index.js:150:40)
    at async dereferenceObject (C:\Users\Sesmi\Downloads\json-schema-fhir\node_modules\json-machete\index.js:265:36)
    at async dereferenceObject (C:\Users\Sesmi\Downloads\json-schema-fhir\node_modules\json-machete\index.js:265:36)
    at async dereferenceObject (C:\Users\Sesmi\Downloads\json-schema-fhir\node_modules\json-machete\index.js:265:36)
    at async Object.dereferenceObject (C:\Users\Sesmi\Downloads\json-schema-fhir\node_modules\json-machete\index.js:265:36)
    at async getDereferencedJSONSchemaFromOperations (C:\Users\Sesmi\Downloads\json-schema-fhir\node_modules\@omnigraph\json-schema\index.js:274:35)    
    at async Object.createBundle (C:\Users\Sesmi\Downloads\json-schema-fhir\node_modules\@omnigraph\json-schema\index.js:1919:32)
    at async Object.getWithSet (C:\Users\Sesmi\Downloads\json-schema-fhir\node_modules\@graphql-mesh\store\index.js:160:38)
    at async JsonSchemaHandler.getMeshSource (C:\Users\Sesmi\Downloads\json-schema-fhir\node_modules\@graphql-mesh\json-schema\index.js:50:24)
💥 🕸️  Mesh  An error occured while writing the schema file: Error: Schemas couldn't be generated successfully. Check for the logs by running Mesh with DEB
UG=1 environmental variable to get more verbose output.
    at Object.getMesh (C:\Users\Sesmi\Downloads\json-schema-fhir\node_modules\@graphql-mesh\runtime\index.js:311:15)
💥 🕸️  Mesh  An error occurred while building the artifacts: Error: Schemas couldn't be generated successfully. Check for the logs by running Mesh with DEB
UG=1 environmental variable to get more verbose output.
    at Object.getMesh (C:\Users\Sesmi\Downloads\json-schema-fhir\node_modules\@graphql-mesh\runtime\index.js:311:15)