How to configure Archive file location in OpenHistorian?

GPA Products openHistorian
1

Hi,

I want to configure a remote server shared location to save the OpenHistorian Archive files (.d2i files).

Please explain the configuration settings for this.

2

The best documentation we have for this is here:

Thanks,
Ritchie

3

Thanks for your information.

4

Hi Ritchie,

I configured archive file locations below manner using PPA Connection string parameter
E.g.
ArchiveDirectories={C:\Program Files\openHistorian\Archive};WorkingDirectory=E:\Logu\Archive
Output:

Archive files are writing to “E:\Logu\Archive”, once space is getting low, I got below error in log file, it’s not switching to another archive location “C:\Program Files\openHistorian\Archive” automatically.
Do I need to add any other settings?

Please clarify and if anything needs to be done, please let me know.

[7/31/2020 1:39:35 PM] Date and Time:         7/31/2020 1:39:35 PM
Machine Name:          NOJA-180
Machine IP:            fe80::7970:c144:d14d:84c7%8
Machine OS:            Microsoft Windows NT 6.2.9200.0

Application Domain:    openHistorian.exe
Assembly Codebase:     C:/Program Files/openHistorian/openHistorian.exe
Assembly Full Name:    openHistorian, Version=2.6.30.2, Culture=neutral, PublicKeyToken=null
Assembly Version:      2.6.30.2
Assembly Build Date:   3/31/2020 5:29:18 PM
.Net Runtime Version:  4.0.30319.42000

Exception Source:      GSF.SortedTreeStore
Exception Type:        System.Exception
Exception Message:     Out of free space
Exception Target Site: GetPathWithEnoughSpace

---- Stack Trace ----
   GSF.Snap.Services.Writer.SimplifiedArchiveInitializer`2.GetPathWithEnoughSpace(estimatedSize As Int64)
       SimplifiedArchiveInitializer'2.cs: Ln 0168, Col 09, IL 0133
   GSF.Snap.Services.Writer.SimplifiedArchiveInitializer`2.CreateArchiveFile(startKey As TKey, endKey As TKey, estimatedSize As Int64, data As TreeStream`2, archiveIdCallback As Action`1)
       SimplifiedArchiveInitializer'2.cs: Ln 0094, Col 13, IL 0008
   GSF.Snap.Services.Writer.FirstStageWriter`2.RolloverTask_Running(sender As Object, e As EventArgs`1)
       FirstStageWriter'2.cs: Ln 0311, Col 21, IL 0976
   GSF.Threading.ScheduledTask.TryCallback(args As ScheduledTaskRunningReason)
       ScheduledTask.cs: Ln 0277, Col 21, IL 0049
5

How much space is available on each drive?

6

Hi Ritchie,

Thanks for your response,

I used below locations
E:, its a USB drive, it was with 1 GB free space and archive files were stored and once reached 350 MB free space, I got above mentioned error message in error log and not moved to C: drive.

C: drive, its with 200 GB free space.

Do we need to set any other configuration parameters to move automatically?

Thanks & Regards
Logu

7

Hi Logu,

It could be a permissions issue on the files in the archive directory. The openHistorian will try to attach all *.d2 and *.d2i files in that directory at startup. If it encounters an error in that process, it removes the directory from the collection of “write directories” that define where it can put archive files. With no write directories, openHistorian would correctly assess that there is zero drive space for writing the file and throw a misleading error about being out of drive space.

Thanks,
Stephen

8

Hi Stephen,
Thanks for your response.
I have configured below settings and uses 2 directories.

ArchiveDirectories={C:\Program Files\openHistorian\Archive}; WorkingDirectory=E:\Logu\Archive

 1.	C:\Program Files\openHistorian\Archive as secondary storage
 2.	E:\Logu\Archive as main storage

Both locations are having writing permission, to confirm that, if we use each directory as part of “WorkingDirectory” parameter, archive files are storing properly.

Only concern is “ArchiveDirectories” parameter is not working, it means, once WorkingDirectory is filled then not moving to Archive directory automatically.

Thanks & Regards
Logu

9

Now that you mention it, the error is thrown from the FirstStageWriter, indicating that that the E: drive doesn’t have enough free space during a stage 1 rollover. Likely, this means that your file size settings and desired remaining space settings are not set appropriately. To understand why, I’ll need to clarify what you have set up.

Rather than a “main storage” location and a “secondary storage” location, what you have configured is a working directory for stage 1 and stage 2 rollovers as well as a “final write path” for stage 3 rollovers. Files aren’t moved when WorkingDirectory is filled. Instead, in-memory structures get dumped to a stage 1 file, then stage 1 files are combined into a stage 2 file, and finally stage 2 files are combined into stage 3 files. Only the stage 3 files will be written to ArchiveDirectories.

If you can only fit 1 GB of data on your working directory’s drive, you need to target a stage 3 file size that is less than 1 GB so it rolls over before filling up the E: drive. The TargetFileSize setting allows you to select an appropriate size. The default target file size is 2 GB so that is clearly not going to work. The minimum is 0.1 GB. Note that the target file size is more of a suggestion than an absolute maximum, so I would recommend selecting a conservative number.

If you want your working directory’s folder to fill up beyond 650 MB, you need to decrease the desired remaining space for the stage 1 and stage 2 rollovers. Both stages’ settings are hardcoded to 1 GB so you would need to modify the code to reduce it. I suspect this may be something you’ve already done, considering your openHistorian version number and the fact that it fills up to 350 MB remaining.

Thanks,
Stephen

10

Also note there is an additional setting DesiredRemainingSpace, in Gigabyte units, that comes into play here - each drive must have at least this much space left in order to be usable.

Note that the default value is set to 500 GB,

Ritchie

11

FYI, it looks like the DesiredRemainingSpace setting was added in v2.7.36.0 so it should not available in v2.6.30.2. Even if it were, the setting only affects stage 3 rollovers. Stage 1 and stage 2 are still hardcoded at 1 GB and would therefore need to be tweaked via code modifications.

12

Hi Stephen/Ritchie,

Thanks for your response.

Please clarify my queries and confirm my statements/understandings.

  1. Is the first priority “WorkingDirectory”?

  2. “DesiredRemainingSpace” applicable only for “WorkingDirectory” or each ArchiveDirectories including WorkingDirectory?

  3. Looks like “DesiredRemainingSpace” parameter is not available with my current openHistorian version 2.6.x.
    By default, the desired DesiredRemainingSpace is 500 GB. If my working directory and archive directory contains 500GB free space then rollover will happen automatically, without any code changes?

Please clarify.

Thanks & Regards
Logu

13

Hi Logu,

  1. There is no priority. WorkingDirectory is where stage 1 and stage 2 files go. ArchiveDirectories is where stage 3 files go. They are mutually exclusive. You can expect your most recent data to exist in the WorkingDirectory, and data typically won’t be written into the ArchiveDirectories until the amount of data you’ve built up reaches the target file size. For that reason, you can also expect many more writes in the WorkingDirectory, and relatively few writes in the ArchiveDirectories.

  2. The DesiredRemainingSpace connection string parameter applies only to the destination folder of the stage 3 rollover. In other words, it only applies to ArchiveDirectories and does not apply to WorkingDirectory. However, each rollover does apply the same logic to determine if a drive has enough space left to write an archive file into the destination folder. The stage 1 and stage 2 rollover processes have a value of 1 GB for DesiredRemainingSpace that can only be modified by changing the openHistorian source code.

  3. In v2.6.30.0 of openHistorian, the DesiredRemainingSpace is 1 GB for stage 1 and stage 2 rollovers and 5 GB for stage 3 rollovers. When performing a rollover, openHistorian estimates what the size of the file it will be producing is going to be. It then determines the total free space on the drive(s) that contain the destination folder for the archive file to be written and subtracts the estimated size from it. If that amount is greater than or equal to the DesiredRemainingSpace for that stage, then it will write the file to the destination folder. Otherwise, it will throw the “Out of free space” error.

Thanks,
Stephen

14

Hi Stephen/Ritchie,

thanks for your help.

Regards
Logu

15

Hi Stephen,

I have an additional queries on this topic

  1. Once the Archive Directories are reached max size, if we need to delete those old archive files automatically, what is the process?

I found “ArchiveOffloadMaxAge” property under group “statArchiveFile” . Can we use this property?
Please give more details about this property and how to use ? if its suitable for my requirement.

Thanks & Regards
Logu

16

Hi Logu,

No, the ArchiveOffloadMaxAge you referenced will only affect the STAT historian, which is using an openHistorian 1.0 archive to save system performance statistics. Instead, the openHistorian 2.0 archive has two relevant connection string settings: ArchiveCurtailmentInterval and MaximumArchiveDays.

The ArchiveCurtailmentInterval indicates how often you’d like openHistorian to clean up the archive, in seconds. The default interval is 86400 seconds, or once per day. The cleanup will remove all archive files that only contain data older than MaximumArchiveDays. The default MaximumArchiveDays is 0, which means that the curtailment is disabled. Increase the value of MaximumArchiveDays to enable the system to clean up after itself automatically.

There is no setting to clean up files based on available disk space. You will need to determine what should be an appropriate setting for MaximumArchiveDays that will prevent your drives from running out of space.

Thanks,
Stephen

17

Hi Stephen,

Thanks for your support. Its working fine.

Regards
Logu