- Added support for configuring capacity threshold limit for a cStor Pool. The default threshold limit is set at 85%. The threshold setting has been introduced to avoid a scenario where pool capacity is fully utilized, resulting in failure of all kinds of operations - including pool expansion.
- Validated that OpenEBS cStor can be used with K3OS(k3os-v0.9.0).
- Fixes an issue where Jiva volumes could cause data loss when a node restarts during an ongoing space reclamation at its replica.
- Fixes an issue where cStor restore from scheduled backup fails, if the first scheduled backup was aborted.
- Fixes an issue where upgrade scripts were failing on Mac.
- Fixes documentation references to deprecated
- Fixes documentation to include a troubleshooting section to work with api server ports blocked due to advanced network configuration.
- Support for generating automated ARM builds for Jiva.
- Support for generating automated ppc64le builds for Node Disk Manager.
- Support for volume expansion of ZFS Local PV and add automated e2e tests.
- Support for declarative scale up and down of cstor volume replicas, increasing the e2e coverage and fixing the issue uncovered.
- Incorporate the feedback on the cStor Custom Resource Schema and work towards v1 schema.
|- Fixes an issue where Jiva Replicas could get stuck in WO or NA state, when the size of the replica data grows beyond 300GB.|
- Fixes an issue where unused custom resources from older versions are left in the etcd, even after openebs is upgraded.
- Fixes an issue where cleanup of Jiva volumes on OpenShift 4.2 environment was failing.- Fixes an issue where custom resources used by cStor Volumes fail to get deleted when the underlying pool was removed prior to deleting the volumes.
- Fixes an issue where a cStor Volume Replica would be incorrectly marked as invalid due to a race condition caused between a terminating and its corresponding newly launched pool pods.
- Support for generating automated ARM builds for NDM.
- Support for managing snapshot and clones of ZFS Local PV.
- Support for setting up PDB and PriorityClasses on cStor Pool Pods. Increasing the e2e coverage and fixing the issue uncovered.
- Support for resizing Jiva Volume via CSI driver and other bug fixes.
|- Add support for building cStor on ARM builds by moving cstor-pool-mgmt, cstor-volume-mgmt, maya-exporter and cspi-mgmt build scripts to their specific folders and add arm build scripts.|
- Add support for provisioning Local PV hostpath when the nodes are tainted. It also handles the implementation of Local PV helper pods where the cleanup pod will be launched with the tolerations.
- Enhance the logging mechanism for cStor pool pods during pool import time. These changes of logs will help to identify the existence of bad disk on the node.
- Support for enabling core dump by adding an ENV variable ENABLE_COREDUMP= “1” for cStor pool pod to control whether cores need to be dumped in case of process crashes. By default, dumping cores will be disabled. Make sure this environment variable is not enabled if mountPoint of
- Enhance upgrade logs by providing pool and volume status information during the upgrade and also helps in estimating the time taken for deployment pods to come up.
- Improves Jiva rebuilding process by checkpointing the io numbers. Now only those snapshots will be synced which has less no of io’s.
- Fixes an issue with Jiva controller by removing WO replica if new replica with greater revision count get added to controller.
- Disable core dump in NDM daemon by default. This can be enabled by setting an ENV variable
- Fixes issues in default core dumping location for NDM. System core pattern which is common for all processes on the node will not be modified. NDM will dump the cores in a location under openebs base directory. NDM process will be launched from the openebs directory, so core files will get automatically written to the $PWD, without requiring to change the core pattern.
- Fixes an issue in NDM which caused cleanup pods not being scheduled on nodes with taints, causing BD to be stuck in release state. The fix will add tolerations for the node taints to the cleanup pod.
- Fixes an issue in
- Fixes an issue while cloning an filesystem based cStor volume, get the FStype info from the CAS config instead of using the default ext4 FSType.
- Fixes an issue during upgrade where image tag contains multiple “:”. This usually happens where the proxy image URL is used which can contain multiple “:”.
- Adding support to have the ZFS Local PV Provisioner in HA mode. Default replica count is set 1 and by changing the replica count will enable a new Provisioner pod with anti-affinity so that no two pods get scheduled on the same node.
- Adding volume metric information for ZFS Local PV. These metrics can be collected using Prometheus.
- Add support for configuring volume policies such as tolerations, nodeSelector , priorityClass, resourceLimits for main and sidecar target pod containers of cStor CSI volume using CStorVolumePolicy resource.
- Add metrics support for cStor CSI volume which can be pulled by Prometheus to show the metrics in k8s cluster. Available metrics are
- Add raw block volume support for cStor CSI volume to be attached as Raw Block Volume to pods.
- Add support for performing CSPC general validation in admission server. Some of the checks are included based on scenarios like use of duplicate block devices, duplicate nodes, block device should not be claimed by other CSPC/third party, the capacity validations of replacing block devices etc.
- Add support for new setting requests and limits to cStor pool pod sidecar containers via poolConfig. It will take default auxResource values if it is not specified in poolConfig for a particular pool. The default can be specified in
- Configure Jiva CSI Driver to listen to custom ports for metrics.
- Add metrics support for Jiva CSI volume which can be pulled by Prometheus to show the metrics in k8s cluster. Available metrics are
|- Support BlockVolumeMode for OpenEBS Local PV backed by devices|
- Support ZFS as a filesystem type for OpenEBS ZFS Local PV.
- Support for Block Device Replacement via the cStor YAML file (using new schema)
- Support resizing and remounting of Volumes when using cStor CSI Driver
- Support for generating of ARM builds for cStor Data Engine.
- Introduce block device hierarchy to NDM. 4 fields
- Add support for applications to provision a "zfs" filesystem directly in the ZFS POOL storage which will get the optimal performance.
- Enhanced the cStor pools to handle auto scale down scenarios to avoid shutting down the node where cStor pool is running. This is achieved by adding cluster-autoscaler.kubernetes.io/safe-to-evict": "false" to the annotation of each cStor pool pod.
- Fixes an issue with liveness probe on
- Fixes an issue in cStor CSI volume unit size conversion while transitioning from PVC to CVC storage capacity the way kubernetes handles, by converting to Gi.
- Fixes a bug where OpenEBS Local PV with hostpaths in OpenShift 4.1 environments was failing.
- Fixes a vulnerability issue with default helper pod image by using the latest tag for helper pods so new version of OpenEBS will automatically get updated with new images.
|- Support for scale down of cStor volume replicas|
- Support an alpha feature in NDM to add Prometheus exporter for exposing disk-level metrics. This Cluster level exporter gives details of blockdevice such as state, claim state etc.
- Support of setup arm builds for apiserver and local provisioner. The images will be built on top of arm64v8/ubuntu:18.04.
- Supporting automated creation of TLS certificate signed by Kubernetes cluster root Certificate Authority for external admission-webhook-server. This will establish a trust to secure the communication between admission-webhook-server and kube-apiserver.
- Support the snapshot and clone feature in cStor volume provisioned via CSI provisioner. This feature will be available in alpha.
- Support of encryption feature in Local PV created on a ZFS based storage pool.
- Support of adding topology information for Local PV on ZFS based storage pool. The details of appropriate ZFS storage pool can be mentioned in corresponding StorageClass via topology so that the scheduler will take care of scheduling the application on the appropriate node.
- Support for scheduling Local PV on appropriate ZFS storage pool on a node which has less number of Local PV volume provisioned in the given pool.
- Support of XFS file system for Local PV creation on ZFS based storage pools.
- Enhanced cStor volume description by fixing output of
- Enhanced cStor volume replica status with additional 2 phases based on different scenarios. The additional phases are
- Enhanced admission webhook server by adding missing labels in config,secret and service and will fatal out when a missing ENV's error and configs happen.
- Fixes a bug in Jiva where Jiva replica pods are stuck in
- Fixes a bug in cStor target while rebuilding process in a single replica quorum case.
- Fixes a bug in NDM for device detection in KVM-based virtual machines.. A new disk model
- Fixes a bug in NDM, where the os-disk filter was not able to exclude the blockdevices if the OS was installed on an NVMe device.
|- Add support to scaleup replicas, replica movement across pools and replica replacement scenarios. This feature is in alpha state. This feature will work for cStor volumes which are created with existing SPC configuration.|
- Availability of NDM on different platforms like amd64 and arm64. NDM can now also be compiled in ARM architecture using manual steps.
- Added support for provisioning CSI based volume using lease leader election API.
- Support of running OpenEBS in Kubernetes 1.16 version. The k8s v1.16 release will stop serving the deprecated API versions in favour of newer and more stable API versions.
- Support the addition of resource limit to cStor pool pod using with CSPC configuration.
- Enhanced backup capability of openebs-velero plugin by checking the status of
- Enhanced CVC(cStor Volume Claim) CR by adding provisioning failure events while provisioning cStor volume using CSI provisioner.
- Fixed a bug where cStor volume becoming read-only due to restart of cstor-volume-mgmt container alone in the target pod.
- Fixed wrong status on CVR from Duplicate to Online. Duplicate state on CVR was blocking reconcile on the volume creation in ephemeral case.
- Fixed a bug in cStor storage pool YAML. The livenessProbe command
- Fixed a bug where Jiva volume running on CentOS 7 / RHEL in OpenShift cluster is going to read-only when the controller pod is restarted either due to node restart or upgrades or any other reason. This is due to iSCSI default timeout is replaced with 5 sec if multipath support is enabled on the node.
|- CSI Driver for cStor Volumes (currently in Alpha) has added support for resizing and volume expansion feature.|
- The new version of cStor Schema has been introduced to address the user feedback in terms of ease of use for cStor provisioning as well as to make a way in the schema to perform Day 2 Operations using GitOps.
- Enhanced error logging of cStor storage pool with a new format for automatic alert generation.
- Enhanced Jiva internal snapshot deletion when a number of internal snapshots are more than 10. The deletion happens automatically one by one.
- Enhanced velero-plugin to support backup/restore for OpenEBS installed in a different namespace other than
- Enhanced NDM to include NodeAttributes in BD and BDC. This will support storing of node name along with the hostname on the BD and BDC CRs.
- Fixes BlockDevice CRD by adding node name to the printer column. This feature will get the name of the node to which the BD is attached while performing
- Fixes a bug in Jiva when patching and clean up operation of Jiva deployments are failing on Nodes where
- Support of provisioning Local PV in clusters where
- Support customization of default hostpath for Jiva and Local PV. With the current implementation, customization will not persisted when a restart happened on the Node where maya-apiserver pod is running or when maya-apiserver pod is restarted.
- Fixes a bug in NDM where all devices on a node were getting excluded when os-disk-exclude-filter is failed to find the device where OS is installed.
- Fixes a bug in snapshot controller where snapshot operation is not throwing any error for invalid
- Fixes the bug where more than required BlockDevicesClaims are created for requested SPC in auto pool method.
- Fixes an issue in maya-api installer to skip re-apply of default SPC and SC resources if they were installed previously by older version(s) of maya or prior to mayaapi-server restart(s)
- Fixes a bug in cStor pool when cStor Storage Pool management creates pool if pool import failed when a disk is not accessible momentarily just at the time of import. cStor storage pool will be in the pending state when this scenario occurs. This PR will fix cStor pool creation by looking on
- Fixes a bug during a cleanup operation performed on BlockDevice and clean up job is not getting canceled when the state of BlockDevice is changed from
- Fixes a bug in NDM where cleanup jobs remain in pending state in Openshift cluster. The fix will add service account to cleanup jobs, so that clean-up job pods acquire privileged access to perform the action.
|- Support for an alpha version of CSI driver with limited functionality for provisioning and de-provisioning of cStor volumes.|
- Support for the upgrade of OpenEBS storage pools and volumes through Kubernetes Job. As a user, you no longer have to download scripts to upgrade from 1.0 to 1.1, like in earlier releases.
- Enhanced Prometheus metrics exported by Jiva for identifying whether an iSCSI Initiator is connected to Jiva target.
- Enhanced NDM operator capabilities for handling NDM CRDs installation and upgrade. Earlier this process was handled through maya-apiserver.
- Enhanced velero-plugin to take backup based on the
- Enhanced velero-plugin to allow users to specify a
- Added an ENV flag which can be used to disable default config creation. The default storage configuration can be modified after installation, but it is going to be overwritten by the OpenEBS API Server.The recommended approach for customizing is to create their own storage configuration using the default options as examples/guidance.
- Fixes an issue where rebuilding cStor volume replica failed if the cStor volume capacity was changed after the initial provisioning of the cStor volume.
- Fixes an issue with cStor snapshot taken during transition of replica's rebuild status.
- Fixes an issue where application file system was breaking due to the deletion of Jiva auto-generated snapshots.
- Fixes an issue where NDM pod was getting restarted while probing for details from the devices that had write cache supported.
- Fixes an issue in NDM where Seachest probe was holding open file descriptors to LVM devices and LVM devices were unable to detach from the Node due to NDM hold on device.
- Fixes a bug where backup was failing where
|- Introduced a cluster level component called NDM operator to manages the access to block devices, selecting & binding BD to BDC, cleaning up the data from the released BD. |
- Support for using Block Devices for OpenEBS Local PV.
- Enhanced cStor Data Engine to allow interoperability of cStor Replicas across different versions.
- Enhanced the cStor Data Engine containers to contain troubleshooting utilities.
- Enhanced the metrics exported by cStor Pools to include details of the provisioning errors.
- Fixes an issue where cStor replica snapshots created for the rebuild were not deleted, causing space reclamation to fail.
- Fixes an issue where cStor volume used space was showing a very low value than actually used.
- Fixes an issue where Jiva replicas failed to register with its target if there was an error during initial registration.
- Fixes an issue where NDM would create a partitioned OS device as a block device.
- Fixes an issue where Jiva replica data was not clean up if the PVC and its namespace were deleted prior to scrub job completion.
- Fixes an issue where Velero Backup/Restore was not working with hostpath Local PVs.
- Upgraded the base ubuntu images for the containers to fix the security vulnerabilities reported in Ubuntu Xenial.
- Custom resource (Disk) used in earlier releases has been changed to Block Device.
|Enhanced the cStor Data Engine containers to contain troubleshooting utilities.-Enhanced cStor Data Engine to allow interoperability of cStor Replicas across different versions. -Support for using Block Devices for OpenEBS Local PV.|
- Support for Dynamic Provisioning of Local PV
- Enhanced the cStor Volumes to support Backup/Restore to S3 compatible storage using the incremental snapshots supported by cStor Volumes.
- Enhanced the cStor Volume Replica to support an anti-affinity feature that works across PVs.
- Enhanced the cStor Volume to support scheduling the cStor Volume Targets along side the application pods that interacts with the cStor Volume.
- Enhances the Jiva Volume provisioning to provide an option called DeployInOpenEBSNamespace.
- Enhanced the cStor Volume Provisioning to be customized for varying workload or platform type during the volume provisioning.
- Enhanced the cStor Pools to export usage statistics as prometheus metrics.
- Enhanced the Jiva Volume replica rebuild process by eliminating the need to do a rebuild if the Replica already has all the required data to serve the IO.
- Enhanced the Jiva Volume - replica provisioning to pin the Replica’s to the nodes where they are initially scheduled using Kubernetes nodeAffinity.
- Fixes an issue where NDM pods failed to start on nodes with selinux=on.
- Fixes an issue where cStor Volume with single replicas were shown to be in Degraded, rebuilding state.
- Fixes an issue where user was able to delete a PVC, even if there were clones created from it, resulting in data loss for the cloned volumes.
- Fixes an issue where user was able to delete a PVC, even if there were clones created from it, resulting in data loss for the cloned volumes.
- Fixes an issue where cStor Volumes failed to provision if the
- Fixes an issue where Jiva Volume - Target can mark a replica as offline if the replica takes longer than 30s to complete the sync/unmap IO.
- Fixes an issue with Jiva volume - space reclaim thread, that was erroring out with an exception if the replica is disconnected from the target.
|Release Notes||Enhanced the metrics exported by cStor Pools to include details of the provisioning errors.-Enhanced the cStor Data Engine containers to contain troubleshooting utilities.-Enhanced cStor Data Engine to allow interoperability of cStor Replicas across different versions. |
- Fixed an issue causing cStor Volume Replica CRs to be stuck, when the OpenEBS
namespace was being deleted.
- Fixed an issue where a newly added cStor Volume Replica may not be successfully
registered with the cStor target, if the cStor tries to connect to Replica before the replica
is completely initialised.
- Fixed an issue with Jiva Volumes where target can mark the Replica as Timed out on IO,
even when the Replica might actually be processing the Sync IO.
- Fixed an issue with Jiva Volumes that would not allow for Replicas to re-connect with the
Target, if the initial Registration failed to successfully process the hand-shake request.
- Fixed an issue with Jiva Volumes that would cause Target to restart when a send
diagnostic command was received from the client
- Fixed an issue causing PVC to be stuck in pending state, when there were more than
one PVCs associated with an Application Pod
- Toleration policy support for cStorStoragePool.
|0.8.1||Release Blog |
|Fixes an issue where cStor replica snapshots created for the rebuild were not deleted, causing space reclamation to fail.-Enhanced the metrics exported by cStor Pools to include details of the provisioning errors.-Enhanced the cStor Data Engine containers to contain troubleshooting utilities. |
- Ephemeral Disk Support
- Enhanced the placement of cStor volume replica in a distributed randomly between the available pools.
- Enhanced the NDM to fetch additional details about the underlying disks via SeaChest.
- Enhanced the NDM to add additional information to the DiskCRs like if the disks is partitioned or has a filesystem on it.
- Enhanced the OpenEBS CRDs to include custom columns to be displayed using
- Fixed an issue where cStor volume causes timeout for iSCSI discovery command and can potentially trigger a K8s vulnerability that can bring down a node with high RAM usage.
|0.8.0||Release Blog |
|Fixes an issue where cStor volume used space was showing a very low value than actually used.-Fixes an issue where cStor replica snapshots created for the rebuild were not deleted, causing space reclamation to fail.-Enhanced the metrics exported by cStor Pools to include details of the provisioning errors.|
- cStor Snapshot & Clone
- cStor volume & Pool runtime status
- Target Affinity for both Jiva & cStor
- Target namespace for cStor
- Enhance the volume metrics exporter
- Enhance Jiva to clear up internal snapshot taken during Replica rebuild
- Enhance Jiva to support sync and unmap IOs
- Enhance cStor for recreating pool by automatically selecting the disks.
|0.7.2||Release Notes||Fixes an issue where jiva replicas failed to register with its target if there was an error during initial registration.-Fixes an issue where cStor volume used space was showing a very low value than actually used.-Fixes an issue where cStor replica snapshots created for the rebuild were not deleted, causing space reclamation to fail.|
- Support for clearing space used by Jiva replica after the volume is deleted using Cron Job.
- Support for a storage policy that can disable the Jiva Volume Space reclaim.
- Support Target Affinity fort Jiva target Pod on the same node as the Application Pod.
- Enhanced Jiva related to internal snapshots for rebuilding Jiva.
- Enhanced exporting cStor volume metrics to prometheus
|Fixes an issue where NDM would create a partitioned OS device as a block device.-Fixes an issue where jiva replicas failed to register with its target if there was an error during initial registration.-Fixes an issue where cStor volume used space was showing a very low value than actually used.|
- Enhanced NDM to discover block devices attached to Nodes .
- Alpha support for cStor Engine
- Naming convention of Jiva Storage pool as 'default' and StorageClass as 'openebs-jiva-default'
- Naming convention of cStor Storage pool as 'cstor-sparse-pool' and StorageClass as 'openebs-cstor-sparse'
- Support for specifying replica count,CPU/Memory Limits per PV,Choice of Storage Engine, Nodes on which data copies should be copied.
|Fixes an issue where jiva replica data was not clean up if the PVC and its namespace were deleted prior to scrub job completion.-Fixes an issue where NDM would create a partitioned OS device as a block device.-Fixes an issue where jiva replicas failed to register with its target if there was an error during initial registration.|
- Integrate the Volume Snapshot capabilities with Kubernetes Snapshot controller.
- Enhance maya-apiserver to use CAS Templates for orchestrating new Storage Engines.
- Enhance mayactl to show details about replica and Node details where replicas are running.
- Enhance maya-apiserver to schedule Replica Pods on specific nodes using nodeSelector.
- Enhance e2e tests to simulate chaos at different layers such as - CPU, RAM, Disk, Network, and Node.
- Support for deploying OpenEBS via Kubernetes stable Helm Charts.
- Enhanced Jiva volume to handle more read only volume scenarios
|0.5.4||Release Notes||Fixes an issue where Velero Backup/Restore was not working with hostpath Local PVs.-Fixes an issue where jiva replica data was not clean up if the PVC and its namespace were deleted prior to scrub job completion.-Fixes an issue where NDM would create a partitioned OS device as a block device.|
- Provision to specify filesystems other than ext4 (default).
- Support for XFS filesystem format for mongodb StatefulSet using OpenEBS Persistent Volume.
- Increased integration test & e2e coverage in the CI
- OpenEBS is now available as a stable chart from Kubernetes
|0.5.3||Release Notes||Upgraded the base ubuntu images for the containers to fix the security vulnerabilities reported in Ubuntu Xenial.-Fixes an issue where Velero Backup/Restore was not working with hostpath Local PVs.-Fixes an issue where jiva replica data was not clean up if the PVC and its namespace were deleted prior to scrub job completion.|
- Fixed usage of StoragePool issue when RBAC settings are applied
- Enhanced memory consumption usage for Jiva Volume
|0.5.2||Release Notes||Changed the custom resource (Disk) used in earlier releases has been changed to Block Device.-Upgraded the base ubuntu images for the containers to fix the security vulnerabilities reported in Ubuntu Xenial.-Fixes an issue where Velero Backup/Restore was not working with hostpath Local PVs.|
- Support to set non-SSL Kubernetes endpoints to use by specifying the ENV variables on maya-apiserver and openebs-provisioner.
|0.5.1||Release Notes||-Changed the custom resource (Disk) used in earlier releases has been changed to Block Device.-Upgraded the base ubuntu images for the containers to fix the security vulnerabilities reported in Ubuntu Xenial.|
- Support to use Jiva volume from CentOS iSCSI Initiator
- Support openebs-k8s-provisioner to be launched in non-default namespace
|0.5.0||Release Blog |
|-Changed the custom resource (Disk) used in earlier releases has been changed to Block Device.|
- Enhanced Storage Policy Enforcement Framework for Jiva.
- Extend OpenEBS API Server to expose volume snapshot API.
- Support for deploying OpenEBS via helm charts.
- Sample Prometheus configuration for collecting OpenEBS Volume Metrics.
- Sample Grafana OpenEBS Volume Dashboard - using the prometheus Metrics
|- Enhanced MAYA cli support for managing snapshots,usage statistics.|
- Support OpenEBS Maya API Server uses the Kubernetes scheduler logic to place OpenEBS Volume Replicas on different nodes
- Support Extended deployment of OpenEBS in AWS.
- Support OpenEBS can be deployed in a minikube setup.
- Enhanced openebs-k8s-provisioner from crashloopbackoff state
|- Support OpenEBS hyper-converged with Kubernetes Minion Nodes.|
- Enable OpenEBS via the openebs-operator.yaml
- Supports creation of OpenEBS volumes using Dynamic Provisioner.
- Storage functionality and Orchestration/Management functionality is delivered as container images on DockerHub.
|- Integrated OpenEBS FlexVolume Driver and Dynamically Provision OpenEBS Volumes into Kubernetes.|
- Support Maya api server to provides new AWS EBS-like API for provisioning Block Storage.
- Enhanced Maya api server to Hyper Converged with Nomad Scheduler.
- Backup/Restore Data from Amazon S3.
- Node Failure Resiliency Fixes