I currently have a ticket open for this with VMware support and they keep giving me the runaround on this so I am hoping to enlist the members of this community to determine whether this is a bug or not.
We were on a Windows install of vCenter Server 5.5 U2e with ESXi 5.5U2 hosts and this was working fine. We upgraded our vCenter first to 6.0.0a, then 6.0.0b, then 6.0U1, where we are at now. The vCenter versions are the relevant versions here, not the ESXi host installs. This broke somewhere in this vCenter upgrade path and I'm not sure which version broke it.
There are four intervals (in seconds) of stat performance data that are collected and rolled over from table to table: 300, 1800, 7200, 86400. I won't go into how these are rolled into each other or the statics logging levels as we believe the data within the tables is actually irrelevant to the bug at hand. From within PowerCLI, you can see each of the StatTypes for each interval by executing the following commands:
Get-StatType -Interval 300 -Entity <name of VM> | sort
Get-StatType -Interval 1800 -Entity <name of VM> | sort
Get-StatType -Interval 7200 -Entity <name of VM> | sort
Get-StatType -Interval 86400 -Entity <name of VM> | sort
where <name of VM> is of course the name of any VM within your vCenter. This will output something like:
cpu.ready.summation
cpu.usage.average
cpu.usagemhz.average
disk.maxTotalLatency.latest
disk.provisioned.latest
disk.unshared.latest
disk.usage.average
disk.used.latest
...
Each one of these is a stat type for which data is collected. If you compare the output of all 4 commands above, you should notice that the stat type "disk.used.latest" is not listed in the output of the 7200 command. It's just gone.
You can then see the actual data for each one of these stat types by executing the following command:
Get-Stat -Entity <name of VM> -Stat <StatType> -IntervalSecs <Interval in seconds>
The issue, and what we believe to be a bug, is that "disk.used.latest" is now missing from 7200 as mentioned above. If I execute the following commands:
Get-Stat -Entity <name of VM> -Stat disk.used.latest -IntervalSecs 300
Get-Stat -Entity <name of VM> -Stat disk.used.latest -IntervalSecs 1800
Get-Stat -Entity <name of VM> -Stat disk.used.latest -IntervalSecs 7200
Get-Stat -Entity <name of VM> -Stat disk.used.latest -IntervalSecs 86400
the 300, 1800, and 86400 intervals output the data for "disk.used.latest" just fine. However, the 7200 interval kicks out the following error:
"Get-Stat : 6/8/2016 10:20:24 AM Get-Stat The metric counter "disk.used.latest" doesn't exist for entity"
The issue is that since this metric is missing in 7200, the stats for it fail to roll up into the 86400 interval. Currently the newest data I have for disk.used.latest in 86400 is 7/11/2015 which is when we upgraded to 6.0.0a. The oldest data for 86400 is 6/9/15. This is because 7200 has failed to roll up this data into 86400 since we upgraded but yet the data is still being purged at one year old daily. On 7/11/2016, all my data for "disk.used.latest" in 86400 will be gone.
VMware support keeps going off on a tangent related to how it rolls up data, completely glossing over the fact that the 7200 interval isn't collecting this disk.used.latest metric because it simply doesn't exist and therefore there is nothing to roll up into 86400.
So... I could use some help proving me right or wrong on this. I only have access to a 6.0U1 vCenter, where this is broke. What I would like is for someone, or multiple people, to execute the below commands in a PowerCLI session on the following vCenter versions, 6.0 GA, 6.0.0a, 6.0.0b, 6.0U1, 6.0U1b, 6.0U2
Get-StatType -Interval 300 -Entity <name of VM> | sort
Get-StatType -Interval 1800 -Entity <name of VM> | sort
Get-StatType -Interval 7200 -Entity <name of VM> | sort
Get-StatType -Interval 86400 -Entity <name of VM> | sort
And simply check to see if "disk.used.latest" is in the list of stat types for all intervals. If I am right and this is a bug, it should be missing for the 7200 interval, depending on the vCenter version. Checking this on each 6.0 vCenter version should tell us on what version it broke and if future versions have fixed it. It would probably help if the vCenter was upgraded from a previous 5.x version, not sure if this issue will be present in new installs. And I'm not sure if VCSA would show this problem or not.
Report back here your findings, if you too are missing the "disk.used.latest" stat type in any of the intervals or if you have it present in all intervals, and what version/build number of vCenter you checked against.
Thanks in advance!
