Recently, ONAP Dublin has been failing due to SSL certificate expiration. This blog discusses the issue, looks at how the community is addressing it, and then offers some possible alternative solutions.
SSL certificates, generated at the time of building the Docker image, are generally bundled together by ONAP components. These build time SSL certificates (with private & public keys) have default one (1) year validity from the date and time that they are generated. And it is this validity that breaks any and all running ONAP deployments past the expiration date. Past this date, most components in Dublin throw the following exception.
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Few instances, demonstrating this today are:
Overwriting these SSL certificates sounds like an easy fix. However, while we can overwrite these files in case of a simple docker container, we can not do the same in case of a K8s cluster. This is primarily because of how ConfigMap/Secrets and Volume mount works in the K8s cluster infrastructure. Simply put, K8s does not support binary file mounting or overwriting of the file using ConfiMap.
Below are the 2 file formats adapted by ONAP components which indicates the key reason for bundling of these files within Docker image.
Adapted by DCAE components, ONAP Frankfurt generates keystore files at the time of deployment with 1 year certificate validity. Consequently, one runs into issues especially when one wants to run a component for more than a year. In which case, the only way out is to redeploy the affected components.
At present there is no direct solution, since all SSL certificates use ONAP domain name
as a key alias and this requires additional work for each customer.
K8s persistent volumes
We can consider leveraging K8s persistent volumes over NFS share. For each ONAP application, we can pre-create the NFS share path and appropriately mount it with the local application path.
For example, ONAP policy PODs store the SSL keystore under a local folder /opt/app/policy/etc/ssl,. We should mount it to an NFS share /onap-ssl-cert-store/policy/ssl to avoid the certificate expiry issue.
The flip side of this approach is that it opens up security holes because if someone were to gain unauthorized access to this NFS share, it would compromise the entire ONAP system.
Base64 encoded ConfigMaps/Secrets
In this approach, we can base64 encode the SSL keystore files and mount it as a K8s volume. The base64 data can be decoded to generate the keystore file at the time of deployment:
Build docker container images and host them under separate tagsIn this approach, we need to set up a CI/CD pipe-line to build the target K8s service Docker images to inject the SSL keystore with custom or customer provided keys.
The only caveat with this approach is that the Docker image maintainability becomes very challenging when we want to roll out upstream fixes.
Addressing ONAP SSL certificates is one part of the puzzle and having an enterprise ready ONAP deployment to get early adopters to put ONAP into production. We at Aarna Network are working actively on this topic to greatly simplify ONAP enterprise adoption in addition to topics such as backup, restore and upgrade capability.
