Creation: 2009-08-29 This version overrides any eventually older definitions. HOW TO MANAGE THE W2P2OFF DISK SPACE ------------------------------------ (0) Scope This file describes which files on w2p2off need to be deleted periodically. Apart from FITS data, there are QC products from the FEROS DRS. There are certainly reasons why these QC products exist and switching off their creation completely is neither reasonable nor feasible in a simple way. Therefore, the disk space on w2p2off needs to be checked periodically each 2-3 months. Not doing this will cause the disk to fill up completely! Obviously the system is not LS2010+ compliant, but this has been well known. If you need to delete something due to a disk space emergency, then please note that (1.1) and (4) are the huge directories. The dataSubscriber should never be switched on for reduced data, as currently there is no way that w2p2off could store the reduced WFI data with 260 MB per file. (1) Incoming directories with FITS data (1.1) Managed by CRON scripts: /data/raw/ /data/reduced/FEROS/ These are managed by CRON script /home/archeso/scripts/CRON/cleanupSpace. (1.2) Managed by FEROS instrument scientist or DHA up to now (Aug 2009): /data/reduced/ Leave at least 1 month of data (in case a VA DVD needs to be re-burned). (2) Pipeline QC products in the astro account You can either clean the following directories with the script 'ferosNewVA', which resets the FEROS pipeline for each new VA and deletes all old files if the command is run with the option --delete. Make sure the FEROS DRS is switched off and you have read and understood how this script works, see: http://www.eso.org/sci/facilities/lasilla/sciops/team_only/feros/ Procedures/FEROS-Procedure-NewFEROSVA.html Alternatively, you can manage these directories manually each few months, as these QC data can indeed be helpful for quality checks and for other VAs as well. It is this latter approach that was applied in the last years. /data/reduced/FEROS/html/ Leave like about 2 months, just for convenience if you want to compare FEROS setups of the previous months, but it can be less as well. /data/reduced/FEROS/saved_session/ You could leave 3 guess sessions per month and the recent 1-2 months completely, just for convenience, but it could even be removed completely. But NEVER delete the 2 guess sessions which are needed to startup the pipeline (guess2005-05-*tar.gz). Remember: - You are also free to delete everything periodically with the script ferosNewVA - ferosNewVA can also be used to completely reset the user copy of the FEROS pipeline, in case a VA has seriously tempered with some key files. This is rarely needed (about once in 2 years). - Run ferosNewVA with the --delete option, otherwise it will just move the data elsewhere. (3) Visitor directories /data/reduced/Visitor VAs are asked to leave their own temporary files only in the above directory and asked to clean up when they leave. It is worth to delete old directories from time to time, as VAs may tend to leave GBs of obsolete data there. (4) DHA managed directories /data/msg When several copies of the same dataSubscriber session are running, several 100 GBs may accumulate in /data/msg. To be checked frequently. /data/bad Due to errors in retrieving FITS files between two machines, some GB of obsolete data may also accumulate in /data/bad. (5) Pipeline QC products in the e2p2ops account /data/E2P2OPS/FEROS-Archive/raw If the new (but not working) FEROS CPL pipeline overrides results of the old pipeline, then many small temporary files may accumulate in this directory. Avoid this by never clicking "reduced data" in the dataSubscriber. This is also justified by the size of 260 MB per each reduced WFI file. /data/E2P2OPS/FEROS-Archive/raw Files older than 10 days are removed by ferosCleanUpLocalArchive running as a CRON job. Check that this CRON job is always active (there have never been problems so far). ferosCleanUpLocalArchive is also cmm-ed. /data/E2P2OPS/FEROS-Archive/DRS Keep optionally 1-2 years of reduced VM data. Most data accumulate in the calibration accounts. For these, leave the last 3 months and up to 3 days per previous months. /data/E2P2OPS/FEROS-QC/saved_sessions Keep 3 sessions per month and all for the recent 3 months. NOTE: ALL THE ABOVE IS JUST A KIND SUGGESTION OF PROCEDURES THAT WORKED WELL IN THE RECENT YEARS. ANYBODY WHO TAKES THE FULL RESPONSIBILITY OF W2P2OFF CAN DELETE THE DATA INSTEAD AS HE/SHE SEES IT.