Status of smartOS support and what future holds

[anonrig]

Hi everybody,

I'm unfortunately opening this issue to TSC repository and not to the nodejs/build repository, in order to get more visibility and share the urgency/state with the rest of the @nodejs/tsc members. Unfortunately, even though I tagged @nodejs/platform-smartos team multiple times, I haven't seen any progress (meaning updating the GCC version) which concerned me enough to create this issue. My goal is not to offend or blame anyone but just discuss our path forward for operating systems that are blocking future improvements of node.js due to the lack of support or personnel working on them.

Current smartOS machines uses old GCC versions despite the "Supported toolchains" show higher versions. For example, our documentation says we support GCC 12 or higher, but our smartOS machine is using GCC 10.

There is an active pull-request of mine from 3 months ago where it is still failing to build. nodejs/node#54990. This is concerning me the mostly because 3 months is a long enough time to expect a compiler version update from a Tier 1 supported OS.

Today, I opened another pull-request to test if updating Ada to the main branch of Ada breaks anything and it seems smartOS (as well as macOS) have failed to build due to old compiler. (Ref: nodejs/node#56218)

As a result of this fact, I recommend lowering SmartOS to "Tier 3/Experimental" and unblock existing pull-requests. I'm open to all suggestions, even though I have a recommendation.

Happy holidays!

cc @nodejs/build @nodejs/platform-smartos @nodejs/tsc

[bahamat]

I'm on the smartos team. I did receive the notification for nodejs/node#54990. You received an answer from @richardlau that the migration was in progress. This migration was being handled by Ryan Aslett. As of August 22, to my knowledge, Ryan was able to make progress without further assistance from me. I can say that after that date no further assistance was requested. I don't have access to the Jenkins build infra, so aside from assisting in designating which image versions should be used, and provisioning in MNX (which we provide for free), there's not anything I can do. And since I don't have access to the build system, I am unable to even verify whether the task was completed.

As for the compiler version, I discussed this with you today on Slack.

To recap what we discussed on Slack (and to expand a bit):

As of August, I had arranged with Ryan Aslett (via Slack) to have SmartOS build agents for v21.4.1 and v23.4.0.

As far as a newer version of gcc goes:

• 21.4.1 will be unsupported as of next month when 24.4.0 is released
• 22.4.0 has gcc12, which is newer than the requested gcc11
• 23.4.0 has gcc13, which is newer than the requested gcc11
• The upcoming 24.4.0 will have gcc13, which is newer than the requested gcc11

Given this, the best thing today to right now is to replace the 21.4.1 image with 22.4.0, which will be supported for another 13 months. Then, once 24.4.0 is released, the 23.4.0 image can be replaces with 24.4.0, or kept as-is. To the extent that I am able to assist with this, I am ready, able, and willing to provide that assistance.

I think it's a mischaracterization that we have been unresponsive. We've addressed every issue that's been asked of us (to the extent that we have permission to contribute) and in a timely manor. We have provided fixes for bugs in node and v8, and also helped uncover bugs in other operating systems before those bugs could land in a release (i.e., bugs were discovered in a failed build on SmartOS, but further analysis showed that other operating systems were affected by the same bug, even though they did not exhibit a build failure).

As I stated earlier, the one issue you mentioned that we didn't directly respond, you had already received an answer from someone more directly involved and responsible for the task than we are so it didn't seem necessary to have to make any specific reply.

[mcollina]

cc @ryanaslett what's the status of the migration @bahamat is referring to?

[ryanaslett]

Apologies for the delay, and lack of understanding of the urgency and impact.

I have provisioned a Smartos23 instance, and got as far as evaluating whether or not the instances were ready to put into the testing matrix.

When running the tests, they were failing intermittently, or, when they did succeed, the tests were taking over 7 hours to complete, even on subsequent runs.

The next step is to investigate whether they're misconfigured, or underprovisioned, or what is causing them to intermittently fail, as well as underperform.

I had intended to begin that investigation myself, so I had better questions to ask, but other priorities got my attention, and I hadn't yet returned to this initiative to finish it off.

Jenkins had automatically cleaned up the old builds I was testing with in the meantime, so wI'll have to kick off another build to start troubleshooting this. (running here: https://ci.nodejs.org/job/node-test-commit-smartos-test-ryan/nodes=smartos23-64/9/consoleFull)

I should have communicated the status of this much sooner and tried to involve more assistance, as I'm neither fully versed in the intricacies of SmartOS, nor do I have deep experience with building node, specifically. As such I was struggling with where to begin investigating to get these runners into a state that resembles the existing node18/20 performance (30-40 minutes).

[mcollina]

@ryanaslett, could @bahamat help you understand what the problems could be, and help solve them?

My understanding is that we do not have enough volunteers to to maintain a Tier 2 status for SmartOS, so either more volunteers provide help in a timely manner, or I think we would need to move SmartOS to Tier 3 for Node v24.

I propose a deadline of 2025/02/14 so there is enough time for the work to be completed before we stabilize Node.js v24.

[mhdawson]

We discussed in the TSC meeting, and there seemed to be agreement that setting the deadline of 2025/02/14 to get the machines updated makes sense, otherwise testing on SmartOS would be temporarily disabled (and documented as Tier 3) until they are. The assumption would be that it would go back to Tier2 once the machines were re-enabled and testing passing.

As an FYI, the deadline might have to be moved up if we need to do a Jenkins security upgrade which forces the need for a newer Java.

[bahamat]

@mcollina What is the minimum number of volunteers? Is that published anywhere?

Are there any cases where we did not respond in a timely manner (aside from the one already mentioned, where the question was answered, just not by us)?

@ryanaslett As for performance issues in the build, this is probably a resource constraint. I can help you diagnose and identify the issue(s), as well as recommend changes necessary to address it. I'm not familiar with the build process (especially the way it's done with the Jenkins agents) but @jperkin from our team is very good at that kind of thing. If we could get someone who is more familiar with the the node build pipeline, we can get this taken care of pretty quickly.

[bahamat]

@mhdawson Moving to v22.4.0 as the minimum SmartOS build version will also address any Java issues. We had to do the same thing for our Jenkins environment.

[mhdawson]

@bahamat in terms of your question:

What is the minimum number of volunteers? Is that published anywhere?

It is enough that a platform is not blocking progress of the project. The best way to make sure of that is for there to be people active and involved in the build working group who will prioritize fixing problems with machines, do the required updgrades etc. Otherwise you are hoping other volunteers prioritize problems on smartOS over other work which may not align with their interests or priorities. We do have some cycles from Ryan from the Linux foundation but prioritization by the larger project also applies to what he will work on as well.

It's not just about responding to questions, but instead being proactive and leading resolution of problems that occur on smartOS machines.

This is my experience as a person from another company which has a vested interest in operatating systems and architectures which are not MacOs, Windows, Linux on x86 and ARM.

[anonrig]

I think it's a mischaracterization that we have been unresponsive. We've addressed every issue that's been asked of us (to the extent that we have permission to contribute) and in a timely manor. We have provided fixes for bugs in node and v8, and also helped uncover bugs in other operating systems before those bugs could land in a release (i.e., bugs were discovered in a failed build on SmartOS, but further analysis showed that other operating systems were affected by the same bug, even though they did not exhibit a build failure).

@bahamat I've tried to be clear in the issue description:

Unfortunately, even though I tagged @nodejs/platform-smartos team multiple times, I haven't seen any progress (meaning updating the GCC version) which concerned me enough to create this issue. My goal is not to offend or blame anyone but just discuss our path forward for operating systems that are blocking future improvements of node.js due to the lack of support or personnel working on them.

I apologize for this misunderstanding, as it was not my intention. Rather than focusing on the issue, which is an extremely old GCC version that doesn't comply/contradicts with our BUILDING.md documentation, I see that the topic has converted into a different place, a place I'm not comfortable with. I've specifically mentioned "my goal is not to offend or blame anyone but just discuss our path forward".

I'm not familiar with issues around smartOS development, but IMHO:

Even before the communication have come up to this point, we should have made smartOS experimental, and once it complied with the supported toolchains, we should made it Tier 2 again.

I propose a deadline of 2025/02/14 so there is enough time for the work to be completed before we stabilize Node.js v24.

We discussed in the TSC meeting, and there seemed to be agreement that setting the deadline of 2025/02/14 to get the machines updated makes sense, otherwise testing on SmartOS would be temporarily disabled (and documented as Tier 3) until they are.

@mcollina @mhdawson We originally changed the supported compiler versions to GCC 12.2 on August 5. (nodejs/node@046343e). If we go with February 14, 2025, we'll make the deadline 6.5 months (197 days).

I think 6.5 months to update a Tier 2 platform is too much, which delays lots of improvements to the project. This is going to stall 3+ pull-requests, updating of Ada, and the addition of URLPattern to the project, which I'm not comfortable with.

[joyeecheung]

IIUC the situation is:

1. Our CI infrastructure is using EOL/soon-EOL SmartOS versions, where the GCC versions are also EOL/soon-EOL
2. The newer SmartOS versions have higher versions of GCC that could make the problem go away and an upgrade in our CI infra has been overdue
3. Those who have access to the SmartOS machines in the CI don't have the expertise/priority to finish the migration ASAP
4. Those who have the expertise/priority to finish the migration ASAP don't have access to the machines in our CI infra

It seems the problem primarily comes from 4. IMO resolving 4 and setting a deadline makes sense. It seems SmartOS is not the only one blocking PRs using new C++ features and there's also macOS. Unless we are considering demoting macOS at the same time (which shares 1-3 it seems, except s/gcc/clang/), it doesn't seem urgent to demote SmartOS alone. We can revisit demoting smartOS when it becomes the sole blocker.

[anonrig]

It seems SmartOS is not the only one blocking PRs using new C++ features and there's also macOS.

This is not true. Here's a pull-request that's blocked by smartOS and not by macOS: nodejs/node#54990

[anonrig]

I think this is also a good opportunity to mention and highlight that the macOS infrastructure update is on a similar position. cc @mhdawson @targos nodejs/build#3686